I ended up bringing him to the hospital after he was refusing to wake up and eat once again. Thank you so much for all your advice.

If he's gaining weight should be ok. Babies like to sleep esp preemies. You could shoot for 3-4 hours tho since he's so little rather than 5 hours. As mentioned before keep check on his temp too