I need advice about something ladies..

My boyfriends mother wants to baby sit out baby girl. Which I'm okay with. However she's not very reliable and doesn't stick to her word often. My boyfriend and I go into an argument because she keeps going behind my back about my daughter. I get it's his daughter as well, but am I wrong for being pissed that they both keep going behind my back about her? That and I don't know if I can trust her to even show up on time. I expressed how I feel about it with him and he gets pissed off. Just a stable trust isn't there. Am I wrong for not trusting her? He says it's not fair for my mom to always watch her yet when we agreed that my mom would watch her unless she has a Drs appointment because my mom lives closer to us. His mom live way out of the way of all three of my jobs. Really need advice on how to handle this situation. TIA.