Overreacting or right?

So my in laws don't ever try to come see my daughter. She's 11 weeks old and they've seen her 5 times (including 2 times at the hospital and at Christmas). They are very controlling and manipulative people. My daughter has started to recognize people and doesn't like strangers now-she cries when someone she doesn't know holds her. Well obviously, she doesn't know my in laws. Yesterday they came over and my fil got mad when I wouldn't give him my daughter. He stormed out of my house mad then came back in getting all in my face yelling at me and pushing me WITH MY DAUGHTER IN MY ARMS mind you. I told him to leave or I was calling the police. My mil just sat there watching not saying anything to him or trying to do anything to get him away from me. My husband ran downstairs when he heard me yell and got all in his dad's face yelling at him and telling him that he wasn't going to treat me like that. Well of course both my mil and fil have fed my husband a bunch of crap and he wants to make up with them. I told him that was fine and he could do that because that was his parents but that they would never see our daughter or myself again! Am I wrong or should I forgive too? A man should never put his hands on a woman and I feel like since that's how he acted towards me, especially while my daughter was in my arms, I don't want to ever give him the chance to hurt my daughter or myself again.