I’ve had the same issue. After catching it happening a few times, I figured out that doing a restart of the watch solves the problem every time. Super annoying, but now I just shut off my watch before putting it on the charger and let it restart itself when it notices it’s on the charger. Haven’t had a problem since.
As it appears that restarting the watch fixes the issue temporarily, my guess is that it’s a software issue of some kind. I have tried restoring as a new watch, and the problem persists. I’m quite tempted to take it to the apple store to do a clean install of watch os and see if it keeps happening.