Yeah, unfortunately it's virtually impossible to not pass it around the house. Any milk from you will help him fight it off, even just a little.

Ok. I do nurse but not totally since my supply gas never been good so he gets formula (and now solids) too

Breast feeding is supposed to be good for passing on immunities