Is it really that harmful to sleep on your back? It's the only way that's comfortable for me

no you can sleep however you want but obviously your stomach. they just say the right side is more comfortable cus the baby isn't pushing against your organs

Yes, it is harmful . It can cut off the supply of blood to baby and you.

Chances are you will become uncomfortable long before you cause yourself or your baby any harm. I found I could sleep quite comfortably on my back early in pregnancy but that option went away in the third trimester because it didn't feel comfortable anymore. Here is a little info about the specifics of back sleeping if you want actual info to make your own judgement call: https://expectingscience.com/2014/08/19/it-is-safe-to-sleep-on-your-back-while-pregnant-or-maybe-not

My doctor said it's fine as long as I'm comfortable. once it gets uncomfortable then to switch to the sides

You can sleep anyway you want your body will make you move before anything bad happens!