I'm pretty sure I've found the culprit.
The TV app.
After some checking back and forth with the displays sleep and screen sleep settings in systemsettings/lockedscreen without success, and confirming that my "hot-corners" for screen sleep/auto-off does actually work, I asked myself; what's keeping these functions to not kick in. They obviously work.
Of course, it could have been some device or network activity causing it, but I've been running lots of applications in the background the last few weeks (simply because I now have the luxury of 32GB RAM), and my first suspect was the TV app. Which I almost never use, but I got a month free TV+ in december.
I quit the TV app, and screensaver and screen auto-off works again.
I also noticed while checking TV app settings, some strange behaviour and blank windows and the app hiding for no reason, so it seems to me there is something not right with it. This is TV 1.3.6 on macOS 13.6.1.
Added: I wonder if TV app being fullscreen in its own space has something to do with it, like having a browser streaming fullscreen video stops screensaver and screen-off from kicking in.