For anyone that have this issue with the 387 driver.
It's actually OK to downgrade the driver back to 378 to get rid of this bug.
I reported this to Nvidia, but they failed to fix this issue in the latest 387...161 driver.
Rather than wait, I've tested a few methods. Eventually, 2 working options.
1) Safer way. Enable "Reduce transparency" in System preferences -> Accessibility -> Display. I tested this for a few days. Didn't see a single colour bug.
2) Risky way. Disable SIP (I can confirm it's OK to disable SIP with the latest 161 driver installed in 17D102), then run
Code:
bash <(curl -s https://raw.githubusercontent.com/Benjamin-Dobell/nvidia-update/master/nvidia-update.sh)
Mr. Benjarmin Dobell wrote a script to detect the "best" driver for your OS version. This "best" basically means the newest driver minus the reported buggy versions from the forums. The "black list" is on his GitHub. Since all 387 drivers are in the list at this moment. This script will automatically determine the 378.10.10.10.25.106 is the "best" version for 17D102 (regardless your actual hardware), download it, and install it for you (the script will ask your admin password before driver installation. Without that, nothing will happen).
This is risky, because there is no guarantee that any web driver other than the V161 can work in 17D102. However, there are plenty of Hackintosh guys tried it already (because they have lots of issues with the 387 driver, so lots are forced to downgrade to the 378 driver). I also gave it a go, and tested this script and driver. I can confirm that this script and the 378...106 driver both can work on my 5,1 (GTX 1080 Ti, spec as per my signature). I ran few quick tests, no bug / glitches so far. And I disabled reduce transparency, tried lots of Windows movement, scrolling, etc. No colour bug yet.
Apart from that, I also ran few benchmarks, tried some simple video editing / rendering, also no issue. Just like running the 378...104 driver when I was with the 17C205 build.
Despite the info on GitHub says SIP can remain enable. It's not entirely true. You can keep SIP enable to run this script. It can still install the "best" web driver for you, and patch it accordingly. However, the patch will void the NVDAStartupWeb.kext signature. Which means, OS won't load it if SIP is enabled. It's not just theory, but I actually tested it by myself.
I tried to run this script with SIP enable, the cMP will restart into black screen only.
I then update the web driver back to 161, disable SIP, and run the script again. This time the 378 driver can load properly on first restart.
For anyone that require the Mac for working, but face this annoying colour bug, I recommend you simply enable "Reduce Transparency". But if it's just your personal computer, and you know how to recover your computer if the web driver screw up your Mac. You may try this scrip to downgrade the web driver.
At this moment, not sure if this driver downgrade can also fix the possible memory leak in High Sierra (in my case, mainly from Safari and Messages). I will monitor that in the next few days. Hopefully this is a good solution to fix the abnormalities that only exist after upgrade to 10.13.3.
Update 1:
After 2 days of testing, no colour / performance bug. Safari, Messages memory usage back to normal (it seems the memory leak was really web driver's fault, not Apple's fault). However, I realise when the monitor is in standby mode (computer normal idle, not sleeping), if I want to wake up the screen, this force installed 378 driver may need extra 10-15 seconds (never see this with any "correct" web driver). It only happen twice out of more than 10 "wake up" in these 2 days, not too bad, and no other abnormalities after wake up.