I use Indesign all the time and this has been my experience as well. Although I've got a hex processor, 1GB VRAM and 32GB RAM, the High Quality display mode is sluggish. For this reason, I suspect the issue is not hardware but rather software related. Perhaps Adobe has yet to optimize the software to take advantage of available horsepower -- this has been the trend for several years.
For this reason, I would avoid swinging for the fence when you purchase your next machine. When I bought my Pro for example, I bought the hex as I considered it to be a good compromise between baseline and 2nd mortgage.