Baby daddy texted me saying "Where is Josiah, I want to take him back" I responded "No" then he said…

Baby daddy texted me saying "Where is Josiah, I want to take him back" I responded "No" then he said "Fym no I been waiting it's time for me to take my son" & I replied "bye I am not going to argue with you" Am I wrong for saying no ? Remind you he NEVER provides for our son, he goes week without asking about him or seeing him & after I heard yesterday that since we have a open case of DV CPS can get involved & try to take my son away. I won't be taking any chances my bestfriend said he probably just wants to see him so he can say he's been spending time with him I feel like that can be the reason too