I've ended my 4yr relationship with my baby father I've had enough of the back & front & the…

I've ended my 4yr relationship with my baby father I've had enough of the back & front & the arguments. We live in separated house because he is still married not even divorce plus I feel our daughter shouldn't be spending the night cause he has a younger brother that was accuse of touching his niece and his little cousin & my baby father did Absolutely nothing to reassure me that it wouldn't happen am I wrong?