If I hook up my two old 27" 4K Dell monitors U2718K (one via HDMI, one via TB), it works fine.
The Dell website does not list a U2718K. Did you mean U2718Q?
The Dell U2718Q is limited to HBR2 x4 connection for 4K60. Two of these cannot exceed the bandwidth limit of Thunderbolt 3 so they usually will work well with a Thunderbolt 3 dock.
The WD19TB dock doesn't have a Thunderbolt port. What ports are you connecting the two Dell U2718Q monitors to?
When I try to hook up my two new 32" 4K Alienware AW3225QF monitors, only one will display.
I can plug in one at a time and set it to 30 Hz, but I can never get both to work. Same exact resolution and cables as the old monitors.
Are you connecting the two 32" displays the same way as the 27" displays - to the same ports of the WD19TB?
The AW3225QF is a HBR3 x4 display supporting 4K up to 240Hz using DSC.
To connect two of these to Thunderbolt 3, you would need to limit link rate to HBR2 x4 which can allow 4K up to 220Hz using DSC@8bpp but maybe the M4 Pro graphics drivers are not smart enough to allow two of these to connect to Thunderbolt 3. You should complain to Apple.
The AW3225QF doesn't appear to have options to select max DisplayPort version or link rate or enable/disable DSC?
One way to limit a display to HBR2 x4 link rate/width is to use a DisplayPort 1.2 MST Hub. It would be interesting to know if a DisplayPort 1.2 MST Hub allows the passthrough of DisplayPort with DSC.
Another way is to connect the display to a USB-C hub that supports USB 3.x data. In this case, the DisplayPort connection will be HBR3 x2 which uses 75% bandwidth of HBR2 x4 which in turn lowers the max resolution using DSC@8bpp to 4K 170Hz.
Some displays have a built-in USB-C hub. The AW3225QF does not (the USB-C port is for data only, not video).
A third way is to use a DisplayPort cable that has one of the lanes disabled. This will force HBR3 x2 link width. I don't think such a
macOS on Intel Macs limits DSC target bits per pixel to 12 but there is a patch to get around that.
I've seen macOS on Apple Silicon Macs use DSC@8bpp for 4K 240Hz mode. I don't know if it will allow DSC@8bpp for other modes or refresh rates.
If macOS were smart enough, it could allow two displays to connect at HBR3 x4 to Thunderbolt 3. Each display would be limited to 20 Gbps which is still enough for 4K240 with DSC@8bpp.
When I plug in my personal MacBook Pro M3 Max, I can have no issues with the driving both AW3225QF monitors.
Is M3 Max more capable than M4 Pro?
Are the two AW3225QF connected directly to the MacBook Pro M3 Max or to the WD19TB?
Did you try connecting the two AW3225QF directly to the Mac Book Pro M4 Pro?