Only thing I've heard is that it can cause problems if you let them walk across your lap. I'm pretty sure it is just their poop. The doctor does recommend washing your hands after you pet a cat.
as long as you wash your hands you should be fine. I wouldn't do it anymore though because you never know what diseases they are carrying and you don't want to put yourself at unnecessary risk.