My SO and I are planning to get married in April. I just want it to be us and our kids, courthouse…

My SO and I are planning to get married in April. I just want it to be us and our kids, courthouse style then, head away the next day for our honeymoon. BUT he wants to invite his mom. Which I'm not opposed to at all but here's my issue, if we invite his mom then I can't NOT invite my mom. Then there's no way I'm inviting my mom and not my dad.... Then the next thing you'll know there 20+ ppl invited now it's turning into a big thing which is what I DONT want. So how do I tell him he can't invite his mom lol?????