I am facing a similar issue right now.
I have a MacBook Pro (M2 Max model) connected to one 8K TV and two 4K displays.
However, I cannot find any options to scale bigger than 4K for 8k TV.
While using BetterDisplay's custom resolution, I can scale it to 5K, but it only offers LoDPI.
Is this related to the statement, "Maximum native smooth scaling resolutions depend on the GPU capabilities and the display's resolution (horizontal width limit is 6144 pixels for M1/M2, 7680 pixels for M1/M2 Pro/Max/Ultra)"?
If it's a hardware limitation, it seems like I can only hope for improvements in future models, possibly with the M3.