A PCI-e SSD is a great solution for a boot drive. For the 1,1 and 2,1s, it's the only solution that I'm aware of since all the PCI-e SATA 3 adaptors I've seen don't support booting on these models.
Sure, the CPUs for these models can be found at extremely low prices now. However, you have to consider that these CPUs use a lot of power. I believe TDP is around 120W per CPU. Modern systems are far more efficient.
Yes, there are more multi core aware apps now, but there are still tons that aren't.
Investing in a powerful video card also makes little sense to me because it will be contained by the PCI-e 1.0 spec.
VT-D is an awesome technology and can make for lots of fun and experimenting. I wish OS X supported it. I would love to be able to use it to for some testing without having to reboot.
With prices of the 3,1s dropping to about $300-400 levels on eBay, I fail to see the attractiveness in investing time and money to upgrade 1,1s and 2,1s. The 3,1s eliminated many of the bottlenecks of its predecessors.