EDIT: Oopsie, you were a bit faster
Ok, here's what I found out. I THINK that the issue is the extension 1Blocker (unfortunately this probably isn't the solution for you since you tried to launch Safari without extensions but maybe it helps someone else). Here was my test procedure.
One note beforehand: To monitor the RAM usage I used iStat Menus. iStat Menus always gives me a way higher Safari memory usage compared to the Safari entry in the activity monitor. I think this is because iStat Menus shows multiple Safari entries combined (e. g. there are also entries called "Safari web content (in cache)" in the activity monitor).
1.) Completely quit Safari
2.) Launch Safari, watch heavy 4K/60 clip
3. [See Safari memory usage with iStat Menus]
4.) Go on some websites
5.) [See Safari memory usage]
6.) Quit Safari (but NOT completely, so just click the red X)
7.) [See Safari memory usage]
WITH 1Blocker X, after the video the usage was at around 2 GB, climbed to 2.5 while surfing on some websites and now the very interesting part: When I quit Safari with the red X the usage stayed about the same BUT: In the activity monitor Safari was listed with hundreds of MB (<-- I will come back to this fact later).
WITHOUT 1Blocker X, after the video the usage was at around 1 - 1.5 GB and stayed there while surfing on some websites and when I quit Safari with the red X the Safari entry itself showed just a usage of around 160 MB (+ some "Safari web content (in cache)" entries with a few hundred MB each.
So it seems like with 1Blocker the used RAM doesn't get freed somehow.