You could probably use the 5K display at 4K connected to the HDMI port using a HDMI to USB-C DisplayPort Alt Mode adapter.
There is currently no adapter that can convert HDMI to DisplayPort with a width greater than 4096 pixels. You have to wait for someone to make a HDMI 2.1 to USB-C DisplayPort 1.4 Alt Mode adapter.
https://insights.club-3d.com/thread/hdmi-2-1-to-usb-c-displayport-1-4-alt-mode/
Even if such an adapter exists, the LG UltraFine 5K can only do 5K with a single connection at 39 Hz (because of ≈ 600 MHz max pixel clock). Normally, the LG UltraFine 5K uses two DisplayPort connections over Thunderbolt to achieve 5K 60Hz (900+ MHz total). Also, there's currently no method on M1 Macs to create an EDID Override to add custom timings.
There exist other 5K displays that can do 60Hz with a single DisplayPort 1.4 connection, but the Mac mini uses a DisplayPort 1.4 to HDMI 2.0b converter internally which can't do more than 14.4 Gbps because of HDMI 2.0 limit (Apple should have used a DisplayPort 1.4 to HDMI 2.1 converter). Also, remember that to do 5K width you need the non-existent HDMI 2.1 to DisplayPort 1.4 adapter (unless you can find a 5K display that has HDMI inputs that can accept 5K width). With the adapter (which doesn't exist) and the method to create custom timings (which doesn't exist), you could do 5K using 60Hz 10bpc 4:2:0, 57Hz 8bpc 4:2:2, 51Hz 6bpc RGB (but macOS doesn't allow 6bpc), 46Hz 10bpc 4:2:2, 38Hz 8bpc RGB, 31Hz 10bpc RGB.
It is unknown whether or not the LG or any other 5K display supports chroma sub sampling so you may be stuck with RGB.
The XDR definitely doesn't support chroma subsampling so from the HDMI 2.0 port it could only get 6K using 28Hz 8bpc RGB or 22Hz 10bpc RGB. It is unknown whether or not the XDR can do custom timings below 47.9 Hz.
