Sometimes my son will be fine on whichever boob I put him on and then out of nowhere he'll let go…

Sometimes my son will be fine on whichever boob I put him on and then out of nowhere he'll let go and won't latch back on and he'll fuss. I'll burp him and try to put him back on the same boob and he still refuses it so I'll switch him to the other side and he'll be fine again. I always check to see and there is always more milk coming out so Idk why he does this, but he's killing me with it.