I'm having the exact same problem but with an LG 49WL95C display connected to a Mac Pro 2013 6,1 (Trashcan) with D700s running Catalina - the DisplayPort 1.2 (supposedly) cable will not work unless forced to 3840x1080 (Low resolution). The default for the DisplayPort cable forces it to produce the unreadable screen shared by the OP. Using HDMI results in the 3840 resolution being automatically selected - go figure.
Oddly when I put the Mac into 5120x1440 mode and capture a screen shot it's "correct" (see second screen shot) - so whatever the problem is seems to be downstream of what the windowing system is presenting to the graphics driver.
My concern is that I am not 100% certain that I am getting the pixels I paid for - oddly the screen is being detected as a 5120x1440 screen (see attached) but I'm not confident that I'm getting as much screen real-estate as I was expecting.
To verify this I put the monitor in PBP Mode (picture by picture) whereby you can effectively treat the display as two seperate monitors with one getting its signal from DP and the other from HDMI, much as one of the other posters did.
As expected the menu bar, fonts, dock etc were all rescaled and the whole dimply felt "larger" (maybe my imagination.
Having heard that this was a Catalina specific problem I tested the same thing with my Late-2015 MBPR with AMD GPU tuning Mojave and sure enough, the problem presented in the same way as the nMP - HDMI good, DP bad. So Catalina is not the core issue AFAIK.
In desperation I tried both SwitchResX and ResXtreme and neither were able to allow me to use the screen in native 5120x1440 mode leading me to conclude that - umm, it's time for a beer.
The obvious next step is to plug in a PC or MBP with Bootcamp and see if the same DisplayPort cable works at least enabling me to eliminate that as the absolute problem.
Best of luck to everyone, a solution (or explanation) would be helpful to many.
[automerge]1578993462[/automerge]
Forgot to add the system profile for Graphics/Display which shows that the screen is being used at the lower res as you would expect (and oddly 30 bit colour)