OK, time to report back about some 32GB DIMM progress.
One of my German net friend went through a series of test with some Samsung 32GB RDIMM (Part number M393B4G70EMB-CK0). So, all the following credits go to Tobi Jagalski.
It's confirmed the 140.0.0.0.0 firmware can boot with 8x32GB DIMM installed (with dual X5680). The computer will POST, and able to show boot manager.
However, all 10.11, 10.12, 10.13, 10.14 cannot boot with this particular Samsung RDIMM. Either KP, or hang with the loading bar. No matter 1 DIMM to 8 DIMM installed, always the same.
If we mix these 32GB DIMM with some 16GB RDIMM, then macOS will boot successfully, but only the 16GB DIMM will be recognised, and run at 800MHz.
Since the cMP can POST, and boot manager works. And all issues only happen after select macOS. Therefore, the next test is to boot Windows. Mr Jagalski has no valid Win 10 key on hand at this moment (I believe he will get one soon for 256GB RAM test). So, only tested it with Win 7 Pro, which is known to limit to 192GB of RAM. And the cMP actually boot all the way to Win 7 Pro desktop with 8x32GB RDIMM installed, and 192GB of memory recognised (at 800MHz).
So, the compatibility issue is from the OS, not the firmware, or hardware. I also suggested to Tobi that he may try Ubunto 64bit Live CD. So, we may see some data with Linux later.
This M393B4G70EMB-CK0 has different spec than the DIMM that from the known good 5x32GB config tested in Yosemite (M393B4G70BM0).
Another source suggested that we really need DDR3-10600 for this particular scenario to make it work in macOS. However, due to lack of proper hardware on hand. No way to test this theory. But if anyone temped to try 32GB DIMMs, better to avoid DDR3-12800. Even this DIMM is known good to work on the 6,1 (in macOS), it doesn't work in any macOS on the 5,1.
But at least now we know that the firmware is not limited to 160GB memory, and because the cMP can POST and shows boot manager with 8x32 config. Most likely it means our cMP can actually run 256GB RAM as long as we get the correct DIMM, and if the OS itself support that config.
In terms of speed. If install all 8 DIMM, memory runs at 800MHz. But if only install 6 DIMM, then memory runs at 1066MHz. It's very typical when using "wrong" memory rank DIMM. Since there is no 2Rx4 32GB DIMM available. We can only use 4Rx4 to do the test, and this "downclock issue" is somehow expected.