Simple test for you to double check with your cats: Pet cat. Nice kitty. Rub eyes. If eyes get itchy and swell up, youre allergic to cats.
It's not foolproof but it's pretty common among pet allergies for the eyes to be the most sensitive. I can be around cats, though if I'm around them too long I start getting a more annoying reaction, but if I pet a cat and don't wash my hands before touching my face, my eyes inform me of my error.
XI Wiki




