My son eats like crazy I'm not sure how to manage my supply.

I usually only pump one side while he eats but its yielding about 5-6 oz each time and if I don't feed him every 1.5 -2 hrs my boobs hurt really bad and I feel the urge to pump but I don't because I don't want to make my supply worse.

Any advice on what can help? My LO is only 2 weeks old