my son hates me....

I feel like my son hates me. He's 3 and about 3 weeks ago he started telling me he wants his daddy. Whom has visitation on the first, third and fifth weekend of every month. I was happy he wanted to see his daddy and that they're building a bond. But now he hates me. He tell me he doesn't like me and that he likes daddy, and that he doesn't like Mommy's house. All he talks about is his dad(which I don't mind) but I do mind him telling me how much he dislikes me. I love my son more than the world itself and I try to be the best I can be for him, but for some reason he just doesn't like me anymore. For example, we went to the zoo today and walked the entire zoo. And once we got in the car I asked if he had fun with Mommy and he said no. Then after he woke up from a nap on the ride home the first thing out of his mouth is that,"he can't see mommy and that he doesn't want mommy house" . I don't know what to Do. It's breaking my heart. No matter what I try and do for him(within reason) to make him like me, he just doesn't..... Is it just me or is it because hes a toddler? I read online that toddlers have a hard time being attached to more than one person. But my son LOVES his nanny, and grandma, and daddy. Everyone but me.... Sorry if the post is everywhere but I'm just at such a loss..