Sperm donor threatens me after I refused sex — I'm raising our daughter alone

My daughter sperm donor became

very angry when I told him I won’t have sex with him anymore. Now he threatens to have his family to hurt me. I don’t know what to do but I refuse to lived in fear. This is causing me a lot of anxiety. He does absolutely nothing.. I take care of our daughter all by myself. I am tired he using our child excuse to come over just to have sex with him. I won’t go thru it anymore..any thoughts/advice ladies?