Are you positive your wrist is rolling the crown? Do you have it too tight or too low on the wrist? If you have the S9, S10, or Ultra 2, check to make sure you aren't activating the Smart Stack through the double tap gesture.
On compatible Apple Watch models, tap your index finger and thumb together twice to perform common actions.
support.apple.com
When I bring up the Smart Stack and then lower my wrist, it goes back to the watch face on its own after a few seconds as long as there are no Live Activities going on, such as a timer. (I'm using all default settings and haven't changed anything.)
In a nutshell, the Smart Stack behaves differently if there's a Live Activity going on. In your case, are you just seeing your regular widgets in your Smart Stack, or is there a Live Activity showing at the top of your stack? If there's a Live Activity, that would probably explain why it still shows. You can disable Live Activities altogether if you haven't already...or disable it for the apps that are causing the Smart Stack to show for you.
There are other ways to dismiss the Smart Stack. You can swipe down from the top of the screen or press the Digital Crown. If there are no Live Activities displaying, covering the watch face with your palm will also dismiss the Smart Stack. (If there's a Live Activity going on, then covering the watch face brings you to the top of the stack.)