Should I Give Up Family Vacation Time for Father's Day?

Ok I need some advice, not really baby related either. So my family and I go once a year, every year and we all get together for a beach vacation. My family lives all over the US but we take this one week out of the year to get together. My fiancé has now come with us for the second year and his dad has complained because we leave the weekend of Father's Day. (his dad complains about a lot and is very clingy to my fiancé) So my fiancé asked me if next year we could take one weekend away from my vacation to spend the weekend with his dad because his dad complained. (let me also add that before we went on vacation my fiancé took a full day off work to spend with his dad and do whatever his dad wanted/paying for his dinner to be like their own little Father's Day) But of course that wasn't good enough. I told him I am not willing to give up the ONE TIME a year I get to see a lot of this family because of his dad being selfish and wanting him that one specific day.. It shouldn't matter what day they celebrate Father's Day but his dad just always finds something to complain about. We live down the street from his dad, I literally see him multiple times a week.. Yet I'm being asked to give up some of the time I have with family I never get to see because of this... Am I being selfish? My mom says I'm not because it's not fair of him to ask me and I don't think I am either but my fiancé seems to think other wise. Not to mention this will be the first year my baby will be part of the vacation so it makes next year even more important. I need opinions.