Yellow just means you're fatigued and are only earning a portion of what you would normally get from an activity. The numbers are still accurate and you won't lose anything from your accumulated totals. When the "fatigue week" resets, the number will no longer be yellow and you'll just go back to earning normal amounts of exp or skill.
For example:
Say I'm earning a solid 1000 exp from each synth I do.
When I hit physical fatigue, that number goes yellow and I'll start seeing something like this from the exact same synth:
You gain 900 experience points.
You gain bonus experience points.
I'm still getting exp, just at a reduced rate (-10% for this example), and the more I continue to grind away at it without taking a break, the less I will be getting.
I think it's generally all understandable from this video:
http://www.bluegartr.com/threads/985...e-System-VIDEO
When it comes to your numbers:
Green = enhanced skill/exp gains (Guardian's Aspect)
White = normal skill/exp gains
Yellow = reduced skill/exp gains (fatigue/surplus/bonus/etc)