I hope someone can help me out as this is driving me insane

Exceptions just aren't work for me. It seems to be making up it's own rules.
This will be a long post but just want to explain what and how I've set things up.
Under Setting in Admin I have these options selected:-
Enable Inventory tracking
Exception Style: Hide Exceptions
I have created two Global Options:-
Colour
Inventory is ticked
Variants
Brown & Green
Sizes
Inventory is ticked
Variants
Small,Medium,Large, X-Large
In product page
I have tried Track With Options and Track Without Options
I have applied the two global options to the product and have built my option combinations fine.
The problem is when I try to apply exceptions to this product. In this case I have all sizes in brown but only medium in green.
When I access Exceptions screen I have created the following exceptions:-
Green + Small
Green + Large
Green + X-Large
In my mind this should mean that when I view the product in the store front and select Green only Medium should be shown
If I select Brown then all sizes should be displayed.
When I do view this product in the store front end Green isn't even available as an option.
Have I got this wrong somewhere or have I picked up the purpose of exceptions wrong?
I have tried deleting the Global Options, Combinations, Exceptions, Clearing Caches and recreating everything but end up with the same problem.
Hope someone can help. I need to know if what I have set up should work or if I've got it wrong.
Cheers
John