It depends on a number of factors, but it is really something that only you can judge for yourself. Everyone's tolerance level is different.
For a number of years, my "at work" computer was a PM 4.1 and I spent most of every day editing images in Ps. The images were stored on external hard drives connected to either the PM's FW800 or USB2 interfaces. It was a productive arrangement except when I was dealing with upsampled files with lots of layers and smart objects, where the file sizes would get significantly over 500mb and occasionally getting into the 1-3gb range. Things would be a bit slow when opening and saving those monsters. Working on 20" x 30" 300dpi images in the 100-300mb range never seemed to be a pain.