The GTX 960 won't be any faster than a used GTX 680, which is around $150-$160. The latter can be easily flashed and runs on OS X stock drivers, so it's a quite obvious decision IMO.
The R9 280[X] will run like **** in a 3,1 because of a power management issue. You'd have to remove some kext files form the system to release it's full power, otherwise it won't be much faster than your ancient 2600XT.