I have a few questions though. I have to admit, I just do not know enough to perfectly understand what you wrote, but I think I get some of it.
If I understand correctly, hexacore = more cores (and I do not need more cores?), but not more RAM or SSDs. RAM and SSDs determine I/O?
Not quite. The Hexcore cost more than a quad core. All the money you spend on those two additional cores cannot be spent on RAM, SSD, or other things that might improve I/O ( which is the primary bottleneck).
The amount of RAM is limited to the number of RAM slots attached to the CPU package. The Hex and Quad offerings both offer a maximum of 4 RAM slots to fill. So what you need is more expensive memory to put into those 4 limited slots (e.g., 8GB DIMMs ).
The dual package models (starts off at 8) have 4 RAM slots per package. With 8 total slots to fill you can stuff more RAM into those boxes.
The reason I did not consider the 2.8 , or even the 3.2, is that they seems to be slower, in even normal applications, than even the MacBook Pros.
I'm not sure how that is relevant since you wish to run an abnormal workload on the box ( server + VMs + normal apps). I'm 99.99% certain that any "normal app benchmarks" you might have seen that placed MBP running faster than a 2.8 or 3.2 were
not running that workload mix.
Any one of those subgroups perhaps. But all three at the same time, I extremely doubt it. Either that or the 2.8/3.2 was running stock RAM (3GB) levels. Pushed up into the 16-32GB range where actually using that much RAM the Mac Pro is better.
For example, some folks get stuck on the first graph on this page (speed test).
http://macperformanceguide.com/Reviews-MacBookProFeb2011-Photoshop.html
However, if you get to the "Medium test" below you'll see that the Mac Pro 2.8 beats the MBP when it comes to doing something that actually uses 16GB worth of memory., Yeah the Hex is a bit faster but you also pay more. Having enough memory to load everything all at once is critical to making the spinning cursor disappear. As soon as apps (a VM, photoshop, reader, web browser, etc.) start competing for memory resources the cursor will pop up.
Another perspective of range of performance if have "too little" or "too much" RAM with respect to the workload demands.
http://macperformanceguide.com/Mac-Upgrade-CaseStudy-MacPro-Memory.html
Hex 3.33GHz doesn't
save you if you only have 3GB of RAM in the box.
I know I do not need six cores, but I read somewhere that extra cores can help even for non-multithreaded apps.
Yes. If you have multiple users or multiple apps running at the same time then that workload can be spread over multiple cores. (an operating system with several daemons running also can spread work over multiple cores. )
Also, is Lion optimized for multi-cores?
Lion requires more memory to run without spinning cursors. If you are going to run it natively ( and in VMs ) you'll need more memory.
If I go with the DIY upgrade, I can do all of the other upgrades you mention; also, I already have 2 X25Es. Do you see a reason, other than warranty, to not do the DIY?
Yeah.
" ... While there are no pins to bend/break on these LGA CPUs, if anything goes wrong the socket is toast. In this case, both the socket and CPU were beyond saving. ... "
http://www.anandtech.com/show/2800/11
hidden behind many of these "void the warranty" discussions are implicit presumptions that warranty wasn't really needed anyway. Upgrades can be done without fraking the board, but Apple doesn't really optimize making the swap easy or safe for the untrained.
If the Hex upgrade is within your risk threshold fine. But I'd rank order that upgrade last among the other mentioned. Something like
i. enough RAM for all foreseeable apps/VMs/etc. combinations. (Photoshop + VM + server software/caching )
ii. enough Disks to spread I/O load. (SSDs for highly independent workload elements except for bulk data. )
iii. back-up infrastructure ( external disks and card to get to disks. )
.........
last. hex upgrade.
IMHO, I'd wait for new Mac Pros and make "hex upgrade" unnecessary. Much of the "get the Hex" chant around here is primarily because the Quads in the single package models are dated. That's because the Xeon 3600 line was jacked up. There was only effectively one model and Apple backfilled the rest of the line up with 3500 series minor speed bumped packages. Once, the E5 1600 series online Apple can leave that cluster-screwed baggage behind.