No, I swap cards. I know, that even using a HD5770 with RX480 will not work anymore.MouSSE has over 7,000 users, so it's a reasonably-proven product. I assume "I tried it with DosDude's patcher" means you got the latest version of his patcher, and installed the SSE4.2 patch during installation (which installs MouSSE)? I assume that "and with MouSSE" means you installed MouSSE per the instructions in the first post of this thread (i.e. you linked it into the kernel)? When you get an error message, there is usually a "Report..." button in the window; if you click that, you'll be presented with a lot of information about what happened. Please copy/paste that information and post it here (inside a {code} or {spoiler} tag, if it's long).
If you're trying to boot with both the 680 and 480 installed at the same time, you're unlikely to succeed - AMD and Nvidia cards don't play well together.
The patcher, I (still) use, has an option to directly patch the AMD driver. I used that and it failed. Then I reinstalled OSX and tried MouSSE. It loads, I don't get an error and still that also fails, when I swap in the RX480.
Dosdude's patcher using MouSSE seems to be something new.
So, can anyone of the 7000 users help me?