The Best way to get a clean trouble free install of DosDudes Patched OSX 10.13 and 10.14 on an unsupported Mac
is to format the drive as APFS - GUID from the start with Disk Util before you install OSX
That way you get a clean install, and no up date issues,
Not to mention that APFS format runs noticeably faster and is more secure
than a standard Mac OS Extended (Journaled) installation.
just remember to Back up your personal Data before you do the formatting.
All the GTX cards from GTX 660 to latest will run fine,
Just remember to use the GTX 680 OSX 10.13 drivers for High Sierra,
It is a good idea to used a flashed card though so you can have a boot screen.
If you are running Boot camp Windows, I would suggest installing ReFind on your OSX drive
so you don't have to hold the Option key at startup,
I would also recommend installing ReFind If you are NOT running Boot camp Windows,
as it makes it far easier to access you recovery drive and any bootable media at startup,
such as OS installer CD's-DVD's-Thumb Drives and external USB drives when you want to boot from them.
I used this method when I installed DosDudes Patched High Sierra on My Mac Pro 3,1 and MacBook Pro 4,1
and I have had no hassles at all with either machine.