The OS X drawing path completely separates window composition from redraw. That's why even non-responding apps do not end up with trails in their window space unlike on Microsoft Windows.
One of the things it does to help achieve this is to keep a pixel copy of the current contents of every window in memory. When the window changes its contents, that buffer is updated. Any time the window needs to be drawn to the screen the contents of that buffer are used.
Since 10.2 we've had Quartz Extreme, which moves the compositing onto the video card. So it is responsible for taking the various window buffers and combining them all to make the desktop. Of course it does that by referencing the various textures one at a time, in turn.
If you have "enough" video memory then all your open apps are able to permanently keep their window contents buffers in VRAM and all desktop redraw tasks are nice and smooth - including Expose and Dashboard tasks.
If you do not have "enough" video memory then the window contents buffers must be swapped in and out of VRAM as and when they are needed. I'm not 100% sure how OS X 10.2 deals with that, but certainly on 10.3 and above VRAM is virtualised in much the same way as main memory is. When apps allocate more memory than you have in physical RAM the system uses the hard disk to swap bits in and out. When everything wants more VRAM than is available, parts have to be swapped from there in and out of RAM.
That costs time and makes the system slower.
That said, even without knowing the exactities of pixel format and how cards handle memory allocation for non-power-of-2 textures, you have to be going some to fill up even most of the low end cards.
Supposing you have a 1440x900 pixel display (the current low end iMac, I have no idea what a likely average would be as I'm not an obsessive follower of models and revisions and so on), even a 32bpp buffer as large as the whole screen should take only around 5 mb. So if you had a 64 mb card then you'd lose about 10 mb for the display, which I believe uses two buffers to ensure smooth drawing, and then be able to simultaneously run around 10 apps that had windows expanded to fill the whole screen hopefully without the OS having to do any significant shuffling back and forth between VRAM and ordinary RAM.
In addition, any running OpenGL app may arbitrarily try to use as many textures as it wants, which will also go into the VRAM virtualisation scheme. Many apps that you wouldn't necessrily think are using OpenGL may do so, especially where they are doing quite a lot of their own geometric drawing rather than being mostly widget based.
Image manipulation programs are another big eater of VRAM space - including Photoshop, anything based on Core Image or Core Video and presumably also Quicktime judging by the comments of radiantm3.
OS X 10.4 includes a new feature, Quartz 2d Extreme (i.e. with an added "2d" in the middle) that on cards that support the OpenGL ARB fragment shader extension - in practice a GeForceFX, Radeon 9500 or better - allows a large proportion of window content drawing to be done on the graphics card. The feature is currently disabled by default, and so is presumably either not complete or not stable but it gives an idea of where things will head in the future. That should allow much greater redraw speeds but again will eat more video memory.
Professionals also like to use multiple screens. That obviously eats further video memory.
Amongst the current GUI providers, Apple are at the forefront of using the GPU to help with ordinary user interaction. Microsoft currently don't use the GPU at all, but Vista seems to do much the same stuff as OS X based on preview columns that I have read. In the UNIX world the next release of X.Org, the newish de facto standard for X-Windows servers in free UNIXs after a rapid fall from grace for XFree86, supports a suitable GL based drawing path extension that is hooked by KDE4 to do much the same thing.
That all said, my Powerbook has 16 mb VRAM and as an undemanding user I rarely see any slow down from Quarz Extreme. I tend to run a browser, maybe a couple of Finder windows and XCode simultaneously and all Expose actions and other window manipulation tasks remain smooth the whole time - with the obvious exception of window resizing which is never smooth. If I plug in a second monitor I seem to lose Quartz Extreme judging by the sudden drop in overall smoothness.