Mac Pro 2011 "Sandy Bridge E" everything we know

[robains]

Late March and still no signs of a new MacPro and Apple still have not asked for any Engineering samples ... anyway ... but we do have an iPad3 that sold 3 Million units already -- I'll never understand the iPad craze.

But back on topic, I just finished building a couple of video/audio workstations for myself (Microsoft Windows7 based) with Adobe PP.

I was able to build two workstations for $5700 (for both NOT each). One is based on i7 3820 (Sandy-E) and the other is based on the i7 3960X (Sandy-E).

Workstation 1
CPU: i7 3820 @ 4.5Ghz
Cooling: H100 Corsair (water)
Motherboard: ASUS Rampage IV Gene X79
RAM: Quad Channel 4X4GB (16GB) Corsair 1866Mhz
HD: Intel 520 (240GB) SSD
Video: nVidia GTX570 2.5GB

Workstation 2
CPU: i7 3960X @ 4.8Ghz
Cooling: H100 Corsair (water)
Motherboard: ASUS Rampage IV Gene X79
RAM: Quad Channel 4X8GB (32GB) Corsair 1866Mhz
HD: Intel 520 (240GB) SSD
Video: nVidia GTX570 2.5GB

I can probably go higher on the CPU speeds but I want to keep temps the same as if the system were just using air cooling, but this setup is VERY stable in Prime95 and Sandra 2012 burn-in tests. The motherboards are EFI based with the usual plethora of options. But no Thunderbolt, only USB 3.0.

Anyway down to testing ... I have access to a MacPro 12 core (32GB) and ran some Adobe PP tests using identical project material across the three systems ... in application render times:

MacPro 12 core: 325 seconds
Workstation 1 i7 3820 (4 core): 205 seconds
Workstation 2 i7 3960 (6 core): 172 seconds

Now Apple's Compressor (setup to use all 12 cores)
Adobe's Media Encoder (setup to all 4 or 6 cores) - make sure same codec used and no "single" core exclusive FX

MacPro 12 core: 210 seconds
Workstation 1 i7 3820 (4 core): 180 seconds
Workstation 2 i7 3960 (6 core): 155 seconds

Now onto Cinema 4D (animation rendering a scene)
MacPro 12 core: 295 seconds
Workstation 1 i7 3820 (4 core): 234 seconds
Workstation 1 i7 3960 (6 core): 184 seconds

CUDA was not used in any testing.

Make your own conclusions, as much as I love OSX over Windows7, the current MacPro just can't keep up with a $2000 Windows7 box (i7 3820). You might say "well it's overclocked" but show me how you can overclock a MacPro and still retain reliability or do it without hacking Apple's EFI?

But here is a thought, since Apple seem less and less interested in Desktop computers, why not release an OSX version that will work on any EFI based motherboard? Get's Apple out of the high end hardware market yet get's their OSX out to more of the world and extending their "influence". Just a thought.

Rob

 

[xgman]

Apple still have not asked for any Engineering samples ..

Rob

huh . . . ? And you know this how?

 

[thekev]

huh . . . ? And you know this how?

I was wondering the same thing.

 

[beaker7]

which model 12 core?

 

[wallysb01]

Late March and still no signs of a new MacPro and Apple still have not asked for any Engineering samples ... anyway ... but we do have an iPad3 that sold 3 Million units already -- I'll never understand the iPad craze.

But back on topic, I just finished building a couple of video/audio workstations for myself (Microsoft Windows7 based) with Adobe PP.

I was able to build two workstations for $5700 (for both NOT each). One is based on i7 3820 (Sandy-E) and the other is based on the i7 3960X (Sandy-E).

Workstation 1
CPU: i7 3820 @ 4.5Ghz
Cooling: H100 Corsair (water)
Motherboard: ASUS Rampage IV Gene X79
RAM: Quad Channel 4X4GB (16GB) Corsair 1866Mhz
HD: Intel 520 (240GB) SSD
Video: nVidia GTX570 2.5GB

Workstation 2
CPU: i7 3960X @ 4.8Ghz
Cooling: H100 Corsair (water)
Motherboard: ASUS Rampage IV Gene X79
RAM: Quad Channel 4X8GB (32GB) Corsair 1866Mhz
HD: Intel 520 (240GB) SSD
Video: nVidia GTX570 2.5GB

I can probably go higher on the CPU speeds but I want to keep temps the same as if the system were just using air cooling, but this setup is VERY stable in Prime95 and Sandra 2012 burn-in tests. The motherboards are EFI based with the usual plethora of options. But no Thunderbolt, only USB 3.0.

Anyway down to testing ... I have access to a MacPro 12 core (32GB) and ran some Adobe PP tests using identical project material across the three systems ... in application render times:

MacPro 12 core: 325 seconds
Workstation 1 i7 3820 (4 core): 205 seconds
Workstation 2 i7 3960 (6 core): 172 seconds

Now Apple's Compressor (setup to use all 12 cores)
Adobe's Media Encoder (setup to all 4 or 6 cores) - make sure same codec used and no "single" core exclusive FX

MacPro 12 core: 210 seconds
Workstation 1 i7 3820 (4 core): 180 seconds
Workstation 2 i7 3960 (6 core): 155 seconds

Now onto Cinema 4D (animation rendering a scene)
MacPro 12 core: 295 seconds
Workstation 1 i7 3820 (4 core): 234 seconds
Workstation 1 i7 3960 (6 core): 184 seconds

CUDA was not used in any testing.

Make your own conclusions, as much as I love OSX over Windows7, the current MacPro just can't keep up with a $2000 Windows7 box (i7 3820). You might say "well it's overclocked" but show me how you can overclock a MacPro and still retain reliability or do it without hacking Apple's EFI?

But here is a thought, since Apple seem less and less interested in Desktop computers, why not release an OSX version that will work on any EFI based motherboard? Get's Apple out of the high end hardware market yet get's their OSX out to more of the world and extending their "influence". Just a thought.

Rob

But we're now talking about 2 different CPU generations, plus the over-clocking and water cooled at that, while possibly only weeks away from the Mac Pro catching up to Sandy Bridge.

I'd like to see you spend $5700 on a Sandy Bridge E Mac Pro and then make this comparison again. At $5700, you'll probably be up against 16 cores (a pair of E5-2650s, with room for RAM, SSD/RAID, plus an video card upgrade if available/needed). And while you might still win, the gap is likely to be so small that it not even close to worth dealing with over-clocking and water cooling a professional machine.

If in 3 months we still don't have a Sandy Bridge Mac Pro, I'd be more inclined to agree with you here, but not yet. At this point I think you're criticism of Apple is a little premature.

 

[kylepro88]

Rob, your tests may be accurate, but somehow I bet that if you ran an identical project render out of FCPX, that the MacPro 12 core would either win or be on par with your other machines (with CUDA on in the PC's).

It's a software problem in my opinion. Compressor has always taken far longer then it should. If you ran Adobe Media Encoder on the MacPro doing the same operation as your PC's instead of using Compressor you'd find a very different result. Compressor just isn't written very well, and Adobe Media Encoder brings any machine to its KNEES. It's not even a contest in terms of encoding speed.

Premiere has never been all that wonderful on the Mac. I understand your point, but when properly tapped for performance the MacPro would win or match what you built, I'm sure of it.

 

[deconstruct60]

which model 12 core?

while I doubt this is going to be a productive benchmark comparison, not only the Mac Pro CPU left in vagueness, the storage system is also. Somewhat telling with the "advertising" details on the other systems are quite detailed and the Mac Pro details wishy washy. Shows the rigor of the experimental design here.

 

[robains]

2.93Ghz 12 core

Oh come on, like I'm going to reveal my source given Apple's reputation of plugging leaks ... fortunately it's not within Apple so there's a good hint for ya. Believe or don't, makes no difference.

But as you can see from my testing (I used applications that are available for both platforms) and maybe now I can understand why Apple are EOL for the MacPro line ... unless someone is tied to OSX, from a performance perspective, it would make little sense to remain on a MacPro given current costs to performance differences. Even less sense if one makes a living thru their computing usage.

Apple's price for 32GB is $1550 everywhere else it's $299 ... make that work for me? ECC is NOT a $1300 "option" (worst case it's $200 increase). Apple's current MacPro pricing is exactly why sales have declined -- seems like 2 + 2, Apple have priced themselves OUT of a viable market. It's kinda spooky that Apple feel they "need" this kinda jaw dropping profit margin.

----------

while I doubt this is going to be a productive benchmark comparison, not only the Mac Pro CPU left in vagueness, the storage system is also. Somewhat telling with the "advertising" details on the other systems are quite detailed and the Mac Pro details wishy washy. Shows the rigor of the experimental design here.

Oh geez, ok

12 core 2.93 Ghz
32GB RAM
512GB SSD X 1
4TB RAID internal (2TB X 2)
ATI 5770

Rendering tests are for the most part CPU tests not hard drive tests ... video capture would be hard drive test. Codec's used support threading in all cases, no single thread FXs used.

I posted my testing with projects I typically work with, feel free to post your own test results. There just aren't that many applications that run on both platforms, but I'm interested to see what your results are.

----------

Rob, your tests may be accurate, but somehow I bet that if you ran an identical project render out of FCPX, that the MacPro 12 core would either win or be on par with your other machines (with CUDA on in the PC's).

It's a software problem in my opinion. Compressor has always taken far longer then it should. If you ran Adobe Media Encoder on the MacPro doing the same operation as your PC's instead of using Compressor you'd find a very different result. Compressor just isn't written very well, and Adobe Media Encoder brings any machine to its KNEES. It's not even a contest in terms of encoding speed.

Premiere has never been all that wonderful on the Mac. I understand your point, but when properly tapped for performance the MacPro would win or match what you built, I'm sure of it.

I didn't test FCPX because there is no multi-platform version (aka Windows version) so I kept testing within same or equivalent applications that have versions for both platforms.

I'll see if I can run an FCPX render vs. Adobe PP render on my workstations using similar or same material ... not sure how valid a comparison that would be, but I think I can come up with a test ... but my hunch is that the MacPro will still fall behind because it's hard to beat Ghz advantage.

 

[wallysb01]

2.93Ghz 12 core

Oh come on, like I'm going to reveal my source given Apple's reputation of plugging leaks ... fortunately it's not within Apple so there's a good hint for ya. Believe or don't, makes no difference.

There are rumors, and there are things people just make up. Without at least a hint of a credible source, this seems like something you just made up.

Apple's price for 32GB is $1550 everywhere else it's $299 ... make that work for me? ECC is NOT a $1300 "option" (worst case it's $200 increase). Apple's current MacPro pricing is exactly why sales have declined -- seems like 2 + 2, Apple have priced themselves OUT of a viable market. It's kinda spooky that Apple feel they "need" this kinda jaw dropping profit margin.

Why are you wanting to buy Apple RAM anyway? Order some ECC from newegg or OWC. This is really a non-issue. Other venders have similarly absurd mark ups for BTO options. I take it as a sign that they just don't want to buy a variety of RAM sticks in large quantities or open these things up more than once. Apple does that same thing with all of their products. Going from the 16GB iPad, to the 32 GB version costs you $100. Do you think the difference between the 16 GB SSD and 32 GB SSD is anywhere close to the $100 they charge? At least you can swap out the RAM and everything else on the Mac Pro.

Oh geez, ok

12 core 2.93 Ghz
32GB RAM
512GB SSD X 1
4TB RAID internal (2TB X 2)
ATI 5770

Rendering tests are for the most part CPU tests not hard drive tests ... video capture would be hard drive test. Codec's used support threading in all cases, no single thread FXs used.

I posted my testing with projects I typically work with, feel free to post your own test results. There just aren't that many applications that run on both platforms, but I'm interested to see what your results are.

Well, we don't have the machines you do. In fact, I don't have access to anything SB-E or i7 3xxx yet, so what's the point? However, I can tell you no i7 will ever be able to do the work I need to do and its got nothing to do with the processor speed and everything to do with the amount of RAM it can handle. So it would take infinitely longer on the i7 as it would on any recent-ish Xeon model. Close enough for you?

 

[beaker7]

Ran this just now:

Benchwell

Mac Pro 12 core 2.93 Ghz 2010: 3:35
Intel 3930x machine at stock speed: 6:06
Intel 3930x machine at 4.6 Ghz: 4:15


Cinebench 11.5

Mac Pro 12 core 2.93 Ghz 2010: 15.16
Intel 3930x machine at 4.6 Ghz: 13.13


Nothing surprising here. 12 core machine better with well threaded apps. Newer core architecture w/ higher core speed better for poorly threaded apps.

 

[wallysb01]

I'll see if I can run an FCPX render vs. Adobe PP render on my workstations using similar or same material ... not sure how valid a comparison that would be, but I think I can come up with a test ... but my hunch is that the MacPro will still fall behind because it's hard to beat Ghz advantage.

If you're talking about something that scales perfectly linearly with core count (i.e. 1:1), simple algebra is enough to get a rough guess. You have 6*4.5 (or 4.8, but lets just pick one), plus about a 20% bonus for the newer tech. So that comes to 32.4. Then the mac pro has 12*2.93 which is 35.16. Now most things are not scaling 1:1, but good programs come close. Several I use maybe cut time down by a factor of ~.55-65 for each doubling of the core count (perfect scaling would of course be .5 and diminishing returns are often not seen until >100 cores). So instead of 12 cores, it would really be more like 10 or 11, with the 1 or 2 getting lose in the over head of multi-threading. For estimation purpose, lets say it actually 10.5*2.93, which would be ~30.8.

Just quickly, if you were instead up against E5-2650s, you'd lose the 20% architecture advantage (dropping you to 27), and with 8 cores active they will hit 2.4 GHz. Then with the non-1-to-1 scaling you'd be facing an equivalent of about about 14 cores, adding up to about 33.6.

So, its interesting to see that you're getting results nearly twice as fast with 6 core and about 1.66 times as fast with 4 core. That tells me what you're doing is not scaling very well at all, because that much closer to the raw GHz advantage you have, and not close at all to the total CPU throughput differences of the machines.

So, I'm with kylepro88 here, the software you're using seems to be the limitation not the hardware, but I don't have any real experience with photo/video stuff mentioned here.

 

[robains]

There are rumors, and there are things people just make up. Without at least a hint of a credible source, this seems like something you just made up.



Why are you wanting to buy Apple RAM anyway? Order some ECC from newegg or OWC. This is really a non-issue. Other venders have similarly absurd mark ups for BTO options. I take it as a sign that they just don't want to buy a variety of RAM sticks in large quantities or open these things up more than once. Apple does that same thing with all of their products. Going from the 16GB iPad, to the 32 GB version costs you $100. Do you think the difference between the 16 GB SSD and 32 GB SSD is anywhere close to the $100 they charge? At least you can swap out the RAM and everything else on the Mac Pro.



Well, we don't have the machines you do. In fact, I don't have access to anything SB-E or i7 3xxx yet, so what's the point? However, I can tell you no i7 will ever be able to do the work I need to do and its got nothing to do with the processor speed and everything to do with the amount of RAM it can handle. So it would take infinitely longer on the i7 as it would on any recent-ish Xeon model. Close enough for you?

You lost me there, you say you don't have any SB-E's and haven't tested any, yet you can for sure tell me no i7 will work for you??

I don't know what projects you work on but, I'd love to know what needs more than 32GB of RAM?? The quad channel bandwidth is staggering for these two workstations I built according to Sandra 2012.

Checking the Apple Store, current entry level MacPro 12 core 2.66Ghz is $5000, my (better) workstation 2 was $3400 and equipped with more RAM and an SDD. I do NOT have access to a MacPro 12 core entry level system to do the same testing but it'll certainly be slower than the MacPro 12 core 2.93Ghz system I tested with.

H100 Corsair water cooler is self contained, it's installation is as easy as any standard heat sink or air cooler and more importantly it's very quiet. In fact, I prefer this CPU cooler because it isn't heavy like giant sized heat sinks and/or fan combos that when in an upright position tend to increase pressure on one side of the CPU overtime as the unit goes thru many heat cycles.

 

[beaker7]

You lost me there, you say you don't have any SB-E's and haven't tested any, yet you can for sure tell me no i7 will work for you??

I don't know what projects you work on but, I'd love to know what needs more than 32GB of RAM?? The quad channel bandwidth is staggering for these two workstations I built according to Sandra 2012.

Checking the Apple Store, current entry level MacPro 12 core 2.66Ghz is $5000, my (better) workstation 2 was $3400 and equipped with more RAM and an SDD. I do NOT have access to a MacPro 12 core entry level system to do the same testing but it'll certainly be slower than the MacPro 12 core 2.93Ghz system I tested with.

H100 Corsair water cooler is self contained, it's installation is as easy as any standard heat sink or air cooler and more importantly it's very quiet. In fact, I prefer this CPU cooler because it isn't heavy like giant sized heat sinks and/or fan combos that when in an upright position tend to increase pressure on one side of the CPU overtime as the unit goes thru many heat cycles.

My MacBook pro runs 16GB ram and the Mac Pro runs 64. Standard load for dual processor workstations these days start at 32 GB.

 

[spacedcadet]

Ram

"I don't know what projects you work on but, I'd love to know what needs more than 32GB of RAM??"

Lloyd Chambers suggests 48GB is getting to be almost a minimum requirement for maximum efficiency when doing high end Photoshop work.

http://macperformanceguide.com/blog/2012/20120319_2-MemoryUsage.html

 

[derbothaus]

"I don't know what projects you work on but, I'd love to know what needs more than 32GB of RAM??"

Lloyd Chambers suggests 48GB is getting to be almost a minimum requirement for maximum efficiency when doing high end Photoshop work.

http://macperformanceguide.com/blog/2012/20120319_2-MemoryUsage.html

Lloyd has unlimited cash flow. PS has abysmal memory management. Recipe for 32GB+. When you really only needed 16GB.
Someone needs to define "high-end". Working with 1GB+ file sizes, or what?

 

[beaker7]

Lloyd has unlimited cash flow. PS has abysmal memory management. Recipe for 32GB+. When you really only needed 16GB.
Someone needs to define "high-end". Working with 1GB+ file sizes, or what?

File size on the one i have open right now is 4GB. Think the biggest i ever saw was 24 GB.

 

[Tutor]

Late March and still no signs of a new MacPro and Apple still have not asked for any Engineering samples ... anyway ... but we do have an iPad3 that sold 3 Million units already -- I'll never understand the iPad craze.

Here's the key to understanding the iPad craze. If the iPad could run on Sandy Bridge E5s, then Apple surely would have asked for engineering samples and been an E5 early adopter. But an E5 announcement from Apple will most like take the form of an announcement of the new iPad Pro Thick SS series, sporting twin 2690's, and the iPad Pro Thick WS, sporting twin 2687Ws: "Servers and workstations for the rest of us - Think different[ly]."

 

[wallysb01]

You lost me there, you say you don't have any SB-E's and haven't tested any, yet you can for sure tell me no i7 will work for you??

Yes, when you need >32-64 GB of RAM, which a substantial part of my work does, i7s won't do.

I don't know what projects you work on but, I'd love to know what needs more than 32GB of RAM?? The quad channel bandwidth is staggering for these two workstations I built according to Sandra 2012.

Things like genome assembly require up to 1 TB of RAM. Obviously this isn't going to be done on a Mac. However, other bioinformatic work can easily max out 64 GB of RAM. Especially if you actually want to use 16 cores at once and each thread will want more than 4 GBs. 4 GB per core just isn't very much. This is why I'm hoping OSX will start supporting at least 128 GB of RAM. With quad channel memory, that seems a must for any new Mac Pro, as even 8 GB per core isn't that much for what I do. Several rather simple perl scripts I use can load 30+ GB into memory. If all you have is 32 or 64 GB of RAM, that limits the amount of multi-tasking that can be done, as you have to wait for other things to finish, even if you have available cores.

I'd also be interested to see if any of my bioinformatic stuff works any better with hyperthreading on Sandy Bridge E. On westmere's it really slows things down to use the virtual cores, so I try to avoid it. However, hyperthreading becomes more useful, then I'd like to have the RAM to support up to 32 threads, at which point 256 GB of RAM would be nice. Without a major redesign however, that's probably not going to be possible until both 32 GB sticks become affordable in sufficient quantity and OSX can support that amount of RAM.

 

[jayhawk11]

Yes, when you need >32-64 GB of RAM, which a substantial part of my work does, i7s won't do.



Things like genome assembly require up to 1 TB of RAM. Obviously this isn't going to be done on a Mac. However, other bioinformatic work can easily max out 64 GB of RAM. Especially if you actually want to use 16 cores at once and each thread will want more than 4 GBs. 4 GB per core just isn't very much. This is why I'm hoping OSX will start supporting at least 128 GB of RAM. With quad channel memory, that seems a must for any new Mac Pro, as even 8 GB per core isn't that much for what I do. Several rather simple perl scripts I use can load 30+ GB into memory. If all you have is 32 or 64 GB of RAM, that limits the amount of multi-tasking that can be done, as you have to wait for other things to finish, even if you have available cores.

I'd also be interested to see if any of my bioinformatic stuff works any better with hyperthreading on Sandy Bridge E. On westmere's it really slows things down to use the virtual cores, so I try to avoid it. However, hyperthreading becomes more useful, then I'd like to have the RAM to support up to 32 threads, at which point 256 GB of RAM would be nice. Without a major redesign however, that's probably not going to be possible until both 32 GB sticks become affordable in sufficient quantity and OSX can support that amount of RAM.

Did we ever figure out what the reasoning is behind artificially limiting the Pro's to 96GB? I know that OWC sells those 128GB kits, but OS X only uses 96GB. It reminds me of the old MacBook Pro's that could accept 4GB and list it in System Profiler, but only used 3GB.

 

[wallysb01]

Did we ever figure out what the reasoning is behind artificially limiting the Pro's to 96GB? I know that OWC sells those 128GB kits, but OS X only uses 96GB. It reminds me of the old MacBook Pro's that could accept 4GB and list it in System Profiler, but only used 3GB.

I have no idea, but it is rather annoying to be sure. It makes it quite tempting to just use linux if the extra 32 GB the hardware can support will actually make a difference.

 

[Umbongo]

Did we ever figure out what the reasoning is behind artificially limiting the Pro's to 96GB? I know that OWC sells those 128GB kits, but OS X only uses 96GB. It reminds me of the old MacBook Pro's that could accept 4GB and list it in System Profiler, but only used 3GB.

Probably comes from the last time they updated that part of the OS and also developed the firmware they only had systems that supported 96GB to test it on. The last gen Xserves would have been able to be tested with such an amount as far back as 2008 if Apple had early access to components. They don't sell or support DIMMs over 8GB so no need to change that yet. Maybe it's a maximum of 16GB per memory channel currently. Perhaps someone who understands the relation between OS, firmware and the memory controller can offer more insight.

Just as Windows XP and Vista supported 128GB max and 7 now supports 192GB max and those are artificial limits. Microsoft don't expect it to be used on servers so why make it support amounts only found on them (at that time).

 

[GermanyChris]

Probably comes from the last time they updated that part of the OS and also developed the firmware they only had systems that supported 96GB to test it on. The last gen Xserves would have been able to be tested with such an amount as far back as 2008 if Apple had early access to components. They don't sell or support DIMMs over 8GB so no need to change that yet. Maybe it's a maximum of 16GB per memory channel currently. Perhaps someone who understands the relation between OS, firmware and the memory controller can offer more insight.

Just as Windows XP and Vista supported 128GB max and 7 now supports 192GB max and those are artificial limits. Microsoft don't expect it to be used on servers so why make it support amounts only found on them (at that time).

At what point though do you kick work off the computer and onto a cluster if it's going to require that much RAM. I have a feeling that anything requiring that much RAM isn't going to like only having 12 cores / 24 threads to work with.

Before jumping down my throat though this is a question/suggestion.

 

[Umbongo]

At what point though do you kick work off the computer and onto a cluster if it's going to require that much RAM. I have a feeling that anything requiring that much RAM isn't going to like only having 12 cores / 24 threads to work with.

Before jumping down my throat though this is a question/suggestion.

I think it just really depends on too many variables as people use workstations for such a variety of tasks. Also a lot of people just don't know how to process work off their main workstation and have no infrastructure in place to learn or try it. Just a big time/money issue really.

 

[GermanyChris]

I think it just really depends on too many variables as people use workstations for such a variety of tasks. Also a lot of people just don't know how to process work off their main workstation and have no infrastructure in place to learn or try it. Just a big time/money issue really.

I thought clusters were now available for "rent"

 

[wallysb01]

At what point though do you kick work off the computer and onto a cluster if it's going to require that much RAM. I have a feeling that anything requiring that much RAM isn't going to like only having 12 cores / 24 threads to work with.

Before jumping down my throat though this is a question/suggestion.

A lot of the things I deal with that require huge amounts of RAM are actually just single threaded, or can't deal with more than about 16 cores all that effectively (i.e. I give them 32 to work with, but only use 24 at most and spend substantial amounts of time using just 1 or 8 or 12). Others work great with clusters and can use 100's of GBs to multiple TBs of distributed RAM on 100's of nodes. Its really all over the place, at least for me. But it is nice to not have to deal with a cluster for your huge memory needs, because they just don't have a lot of nodes with huge memory. So wait times are long, even if only 1 or 2 other people are using it. Now, I don't we'll be buying 1 TB of RAM any time soon, so something will have to go to the cluster, but if we can cut into the low 100's or get up towards 200, that would be a huge help.

----------

I thought clusters were now available for "rent"

They are, such as Amazon EC2. But it takes time to learn the job scheduling system and to upload/download data. For those without relatively high speed connections, moving 100's of GBs of data could take days. Then it also takes a fair amount of cash. Our university server charges 2.5 cents per cpu hour. Its pretty easy for a project to burn through several thousand dollars in 6m-1y at that rate. Many people would rather buy another computer at that point. So, it just all depends.