(Actually it can sculpt up to a billion polygons in HD mode and around 40 - 60 million comfortably in standard mode...)
But aside from that did you ever find an answer to your Mudbox question? I have a 5870 and find that its not so great painting on denser meshes :/
Nobody really has any valuable input, simply saying
this card works better than
this in maya mudbox.
now, mudbox surprisingly puts a lot on CPU which is freakin weird, for example amount of models in scene = CPU, subdues = RAM. Somewhere there most likely sculpting / paining / rotation in viewport is based on GPU and some of the data on CPU.
So in my theory yes radeon 7970 3gb and geforce 770 2-4gb are best options for mac os x right now for pro CGI work, quadro K5000 is out of my budget reach but probably is best solution out there if you really want smooth performance, since those are the only cards supported by Autodesk officially.
But heres catch, if radeon is 7970 so powerful in pc benchmarks, knowing that 7950 is absolute JUNK in os x 10.8.4 and speed bump over 5870 is 5%-10%, makes me feel like 7970 will not perform any better and is more of an investment for upcoming 10.9 Mavericks, but weird investment into old form factor, so kind of falls short here.
so the choice is 680 4gb or 770 2-4gb. I'm hundred percent sure for painting in mudbox 680 or 770 will be a drastic jump from 5870. Don't know for sculpting thugh.
From personal sources i know for sure that AMD right now collects feedback to simply get rid of performance issue in OS X with pro apps. They were busy with other important unknown task. So probably Mavericks will finally get rid of cards performance issue.
GTX 770 4gb (slightly better) > GTX 680 4gb and since it was kind of official card for macs the drivers seem to be somewhat adequate.
I will risk and go with GTX 770. Hopefully macvidcards will come out with 4gb version soon.
My setup right now i believe most optimal for Rednering/Modeling/Texturing.
MPro 5,1 (2010) 3.33 6 core.
Sonnet tempo pro with samsung 840pro ~256gb. + old 3g 120gb OWC hard drive for scratch drive ( should upgrade )
4 1tb FAEX western digitals in RAID 0 (i don't care i back up locally).
2 x 2tb EARX western digitals in Raid 0 (i don't care i backup to CrashPlan aswell, can't allow time machine being bogged down by RAID 1)
48gb RAM.
the card will be either 680 or 770.
there is lots of room to improve but really can't improve CPU GPU RAM any further without toooooooo much cash.
I think this is as much as you can go without blowing toooo much money, while waiting for new MP's.
But to be honest i'm tired of all this **** with apple targeting photographers and video editors as pros there are far more people in STEM industry who can benefit and have different opinion on mac pro.
P.S. HD mode is weird you get little circle which is not really useful for proper sculptors, especially coming from traditional media for me it's crucial to evaluate whole thing every other minute it has to all hold up together.