Even After Defeat, My Life Goes On
1) The 8 GPU processor brick wall: My Tyan Server has 8 double wide GPU slots. Until recently, I had hoped that I could load the Tyan with 8 double wide GPU cards and have it recognize all of the GPUs. My goal was to use Titan Z cards to exceed 8 GPU processors and to have 14 GPU processors in one system. Having tried bios mods, Regedit hacks and various card stack configurations, the hardware will not recognize more than 8 GPU processors. So having multiple GPUs on one card does not get around the 8 GPU limit because it's processor based. I can put either a maximum of 8 single GPU processor GTX cards ( such as 780 Tis, 780 6G, original Titans, Titan Blacks, etc.) in it, or 4 dual processor GTX cards (such as 590s, 690s or Titan Zs), or a combination of single and dual processors cards in it so long as I do not exceed the 8 GPU processor limit. All of the other processors above 8 are wasted there. Because of that discovery/realization, I've decided to put 2 Titan Zs (4 GPU processors total), plus 1 Titan Black (1 GPU processor), plus 3 Titans (3 GPU processors total) (4+1+3=8) in my Tyan Server. So, I assess my accomplishment of my goal here as a major failure.
2) Mr. Freezer begging for attention: While so far devoting all of my time for this project to the 8 GPU processor limit, I haven't had the time to find the optimal settings for Mr. Freezer. Disappointingly, I haven't seen temps for coolant exiting Mr. Freezer below 20 degrees centigrade. Usually when I've noticed the coolant temps exiting Mr. Freezer and entering my reservoir, the temps are around 21-23 degrees centigrade. The system was then recognizing only 8 of the 14 GPU processors, but for rendering Octane benchmarks and other render jobs the processors' temperatures (read by GPU-Z ) never exceeded 35 degrees Centigrade and averaged 33 degrees centigrade. At idle the temperatures of the GPUs processors were the within 1-2 degrees centigrade of the coolant temperature as it exited Mr. Freezer (so the GPUs temps ranged from about 22 to 25 degrees centigrade). I was and am shooting for an average of around 5 degrees centigrade at idle and an average of around 10 degrees centigrade under load. So, I assess my accomplishment of my goal here as, currently, trending toward major failure with a tiny semblance of promise.
3) Flopping about: GPU-Z readings of single precision floating point peak performance are (A) slightly in excess of 4,500 GFLOPS for each of my old Titan SCs and (B) slightly in excess of 5,000 GFLOPS for my Titan Black and and for each GPU of my Titan Zs. I've yet to over clock any on them for these tests. So after reconfiguring my GPU cards in my Tyan server as I've indicated above, I fully intend to achieve a single system floating point peak of 38,500 (5x5,000 + 3x4,500) GFLOPS for the Tyan system before I even overclock any of the the GPUs. Moreover, 38,500 GFLOPS equals 38.5 PFLOPS (petaflops).
*/ However, my goal was to achieve a single system single precision floating point peak performance of at least 56 PFLOPS. So, I assess my accomplishment of my goal here as, currently, strongly trending towards complete failure, with the degree of completeness of failure dependent significantly on how no. 2, above, shakes out since my ability to safely over clock the GPUs is affected by their temperatures.
4) Pulling it apart and configuring, but this time not from scratch - so it's reconfiguring: Luckily, I had the foresight to buy and spread quick disconnects throughout this mod, buy two dual pumps (one as a replacement - luckily too I don't need it yet) and buy lots of tubing and non-conductive coolant. So, I'll be reconfiguring the Tyan as set forth above. I'll be installing the other four Titan Zs (40 of the PFLOPS that I had intended to all be in one system) in one of my SilverStone cases, on a 4xPCIe double wide slotted GUP4 motherboard. Both of these systems will be water cooled by the same cooling system. I am glad that I, at least, had a plan in place for potential failure(s). Life goes on, regardless.
*/I. On June 10, 2013, China's Tianhe-2 was ranked the world's fastest single computer with a record of 33.86 petaflops. [
http://en.wikipedia.org/wiki/FLOPS#Single_computer_records ]
