Just found that Adobe limits to 3 GB. But there's a workaround on MacOSHints:
Work around Photoshop's 3GB RAM limit
Fri, Jul 18 2008 at 7:30AM PDT Submitted by Anonymous
With RAM prices dropping so fast, it seems a shame that I can't really use more than 3GB of memory for Photoshop. As a 32 bit application, Photoshop CS3 can only "use" 4GB of real memory -- a 1GB chunk for the application, leaving a potential 3GB of real RAM available for Photoshop to use for my images. For anyone with 8GB or more of memory, here's an old concept that tricks Photoshop into using as much memory as you want.
Remember RAM disks from OS9? The feature is available in the command line in OS X. By creating a RAM disk, and having Photoshop use that as the first scratch disk, you'll speed up Photoshop as long as you have enough real memory to do it. For my test, I allocated 3GB to Photoshop in its prefernces (real memory usage), and then created a 2GB RAM disk.
The Terminal command to create a 2GB drive is:
hdid -nomount ram://4194304
The number is the number of 512 byte blocks in your RAM disk. The example above creates a 2GB disk, so a 4GB drive would be 2x that number, and a 1GB drive would be half that number, and so on. Once the disk is created, format it with Disk Utility. Then configure Photoshop to use the RAM disk as your first scratch disk. As soon as CS3 hits the 3GB memory barrier, it starts using the RAM disk as "memory" first. Remember, it will be "swapping" memory pages in and out of the 3GB memory set, but this is still better than swapping to and from a hard drive.
I created a 4.5GB file in Photoshop, and was pleased to see almost no disk activity on my real drive, but my RAM drive was full during my test.
Note: This really only helps someone with 8GB or more of memory running CS3 a lot. For example, if you had 16GB of memory, you could create an 8GB RAM disk, and then allocate 3GB in CS3, effectively having Photoshop "use" 11GB of real memory before hitting the disk. Also note that RAM disks are not quite as fast as regular memory access native to an application, but they are much faster than having CS3 swap to your hard drive.
While it takes some steps to get this configured, I thought there might be some people out there interested in improving how CS3 uses real memory, since there is no 64-bit version yet.