You should eat healthy fats like avocado and nuts. I have the same problem it's hard for me to gain weight because I have hyperemesis gravidarum. And I'm a vegan.
I'm in the same boat. My doctor says we use more calories while pregnant then we realize. She told me to eat things like avocado to help gain a little weight and be healthy. But unless you doctor tells they're worried you are perfectly fine.