He doesn't even see her or anything thing like that and I want to take her to meet my other side of the family that can't come to the U.S

that's not true. regardless of whether or not he's on the birth certificate he DOES have rights, even if he doesn't pay child support he has rights. if you guys don't have court orders or visitation set up then there is really nothing he can do unless he does take you to court so I wouldn't be concerned.

Okay because he's been threatening me about taking me to court but the only one on the birth certificate is me

If he not on the birth certificate he has no rights which means he can't do anything about it. I would take her!