Need some advice ladies: Long post (sorry)

my boyfriend and I have recently broken up. We have a 15month old daughter and I am currently 22 weeks pregnant. I moved out a little over a month ago and am now living with my parents. To this day since our daughter has been born he has yet to buy any type of clothing, shoes, etc. He has bought one pack of burp cloths, diapers, and formula/milk once and a blue moon. He gave me $70 over a month ago to “put towards” expenses. Nothing since. He has never been anywhere with our daughter by himself nor has he tried to take her anywhere. Since his mother moved back in town that’s when our problems began. Long story short about his mom. His mom has previously given his son nighttime medicine for him to sleep at night and during the daytime before a car ride so he’ll sleep. She spanks his son, and yells at him. My ex does nothing to correct her. I am not comfortable with my daughter being around her without me around. Even with my ex since he sees nothing wrong with what his mother does. I told him he can come by my parents house to visit Our daughter anytime but as for him taking her I am not comfortable with. Am I wrong for telling him he can come visit but can not take her? Too many things happen to kids nowadays and I would never forgive myself if something happened to my daughter and I could’ve prevented it.