Well, let me preface this by saying I am not an expert, nor do I have direct experience with both cards using OpenGL Apps or OpenGL apps on the Mac specifically, either. I just did some basic searching around.
It was actually difficult to find ANY information on this topic, because nearly all of the discussion of graphics cards in the context of OpenGL and CAD/CGI apps invariably is about the professional workstation cards ala ATI FireGL and nVidia Quadro.
After searching around, it becomes obvious that for [current] workstation cards, nVidia's Quadro is faster than the equivalent ATI FireGL in professional OpenGL Apps. However, most likely
due to nVidia crippling their Geforce OpenGL drivers to upsell you to Quadro, the Geforce line (including G92/8800GT) appears to be slower than the newest ATI Radeon 38xx series. I am actually somewhat shocked by this due to the domination by Nvidia's G92 (8800 series) in DirectX gaming performance and the industry-standard '3DMark' benchmark versus ATI's 3850/3870.
Since many of OSX's APIs are built on (i assume) OpenGL, this might explain why there has always been anecdotal reports (and benchmarks somwhere - I'm sure) of ATI cards dramatically outperforming more expensive nVidia cards in applications utilizing Core Image/Animation and even the OSX desktop feeling more fluid and responsive.
Unfortunately, I didn't write down all the references I found, but I found this one in my history:
Benchmark: SPEC viewperf (OpenGL CAD and 3D graphic apps benchmark)
http://forums.anandtech.com/messageview.aspx?catid=31&threadid=2168891&enterthread=y
Results: ATI 3870 2x-2.5x as fast as nVidia 8800GT in OpenGL rendering tasks