Would just like to make a post about how I got it to work.
About six months ago I tried installing scripts and disabling all sorts of stuff (all guides from within this thread). Recently I've had to do some Uni work utilising surround sound so decided to try my best to make this work again, I got it working and it was a lot less painful than I thought it would be.
I have a 2009 Mac Pro (2010 firmware), Zotac GTX 980 Ti AMP! Edition, GT 120 for recovery mode stuff, El Capitan 10.11.5, Nvidia driver 346.03.10f02
Here are the steps I went through...
1) I began by removing the 980 Ti and replacing it with the GT120.
2) Restarted into recovery mode.
3) Opened Terminal and disabled SIP using the command "csrutil disable" and checking it worked with the command "csrutil status" (when I did this WavTap, a piece of software I forgot I had installed started to show up again)
4) Boot into El Capitan.
5) Check SIP is disabled by entering "csrutil status" into terminal
6) Instal HDMI Audio 1.1 using instructions given in the download (
https://mega.nz/#!b5JRnRRZ!5hc7EyGDCx0vHdY5-nCvi1PagOkOyInupvEFHlZUysY)
7) Restart
8) Check HDMI Audio is installed by entering "kextstat" into Terminal
9) Place Graphics card back into Mac Pro (Note: I only have one graphics card in the mac at one time)
9) Check HDMI audio is working. I had to unplug and plug in the HDMI cable before the audio would show up. Works through display port too.
Here's a screenshot to show it's all working, even 7.1 surround!
I hope this helps! Please ask if you have any
Edit: Encoded digital output works through VLC too (DTS, Dolby etc.)