Is it rude to think that if people want to meet our child, they should come to us?

Mil keeps hounding for us to visit "faaaaamily". My husband doesn't even talk to most of them.

We have our own place, people are welcome to visit. What we do not have is money and time to drive around for people to honestly overstep and give us unsolicited advice. People love to wake baby and he ends up overtired from the constant stimulation.