When choices were fewer, decisions were easier, but things ran slower. Now flip 180
They look amazingly efficient on space - very tidy.
How does this tie in the with limits you have found on number of GPUs per system?
It appears that the number of GPUs per system is mainly dictated by the following factors:
1) system IO space size,
2) other system bios features or their equivalent,
3) PCIe implementation of the particular motherboard,
4) the particular GPU and each GPU's IO space requirements,
5) the particular software application, and
6) the OS.
And don't forget that these factors may interact in non-obvious ways with one another.
Suplementation and examples:
Factors 1, 2, 3 and 4) Amfeltec [ "
The motherboard limitation is for all general purpose motherboards. Some vendors like ASUS supports maximum 7 GPUs, some can support 8. All GPUs requesting IO space in the limited low 640K RAM. The motherboard BIOS allocated IO space first for the on motherboard peripheral and then the space that left can be allocated for GPUs. To be able support 7-8 GPUs on the general purpose motherboard sometimes requested disable extra peripherals to free up more IO space for GPUs. The server type motherboards like Super Micro (for example X9DRX+-F) can support 12-13 GPUs in dual CPU configuration. It is possible because Super Micro use on motherboard peripheral that doesn’t request IO space."[Emphasis added]] and other providers of PCIe expansion products caution that exceeding 7 to 8 GPUs per systems is possible only with certain Supermicro systems because only they have the IO space sufficient to recognize GPU counts above 8 or 9. However, Tommes was able to get Octane to recognize only 7 of his 10 GPUs [
https://render.otoy.com/forum/viewtopic.php?f=23&t=44209 ], despite the fact that his Windows OS system running on his Supermicro X9DRX+-F recognized all ten of his GTX 780 TIs. But, GTX 780 TIs might present a special case GPU - see discussion of Factor 4,
below.
Factors 2 and 5) A) OctaneRender's license [ see, e.g.,
https://render.otoy.com/shop/nuke_plugin.php ] states: "
A maximum of 12 GPU's may be used. You will not attempt to circumvent the physical GPU or single machine license limit, including obfuscating or impairment of the direct communication between Octane and the physical GPUs, virtualization, shimming, custom BIOS etc.; [Emphasis added]" B) Redshift3d states: "
Each instance of Redshift can currently use up to 8 GPUs concurrently. To take advantage of more than 8 GPUs on a single machine, you can launch multiple instances of Redshift each rendering a different job on a different subset of available GPUs."
Factor 3) My cMP2,1s can recognize and run stably only 4 GPUs (or 4 GPU processors = 2xGTX 590) which matches the number of PCIe slots in the system.
Factor 4) Running 8 GTX 780 Ti ACX SC OCs in my Tyan steals resources necessary to recognize all of the system ram that I've installed and thus less ram, than all installed, is recognized. However, neither my 8 GTX Titans, nor my 4 Titan Z (4x2 GPU processors), rob other system resources when installed in the same system, i.e., all of my system's ram is recognized. Thus, I believe that certain GPUs consume more IO space than others [see discussion of Tommes's issue in 1, 2, 3 and 4),
above.
Moreover, running 16 GTX GPUs, even with a TDP of 250w, consumes a lot of power, and depending on the application each of them could be consuming 300w or more: 16 * 250w = 4,000w; 16 * 300w = 4,800w. Very few PSU systems can handle that without special precautions/power/electrical/ systems being employed. Thus, only a few GTX cards would be powered sufficiently by a 3,000w PSU.
6) As to the OS, I do not now know what is the max GPU limit, if any, of OSX. I do know that Mavericks V. 9.2 has a 32 CPU core limit. Moreover, all that I've read on the best OS for large numbers of GPUs for system are:
1) Linux,
2) Windows and
3) OSX
in that order.
OctaneRender provides a cure to, at least some, missing GPUs running under Windows:
"
Issue 9. Windows and the Nvidia driver see all available GPU's, but OctaneRender™ does not.
There are occasions when using more than two video cards that Windows and the Nvidia driver properly register all cards, but OctaneRender™ does not see them. This can be addressed by updating the registry. This involves adjusting critical OS files, it is not supported by the OctaneRender™ Team.
1) Start the registry editor (Start button, type "regedit" and launch it.)
2) Navigate to the following key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}
3) You will see keys for each video card starting with "0000" and then "0001", etc.
4) Under each of the keys identified in 3 for each video card, add two dword values:
DisplayLessPolicy
LimitVideoPresentSources
and set each value to 1
5) Once these have been added to each of the video cards, shut down Regedit and then reboot.
6) OctaneRender™ should now see all video cards." [
http://render.otoy.com/manuals/Standalone/?page_id=62 ]
I've had to resort to this technique to get both the system and Octane to recognize my eight GTX 780 TI ACX SC OCs in my Tyan. It worked successfully [
https://render.otoy.com/octanebench/summary_detail_item.php?systemID=8x+GTX+780+Ti ].
TheaRender provides a little more guidance:"
- Configure Windows Watchdog (Windows only). Windows runs a service, called "watchdog" that monitors the graphic driver. If the driver does not respond within 2 seconds, it decides that there is a kind of instability so it terminates and restarts the driver process. The driver is the process responsible for handling all Presto commands to the GPU and - unfortunately - Presto is responsible for keeping the driver super-busy when there is a heavy rendering job.
You can actually configure watchdog service. And this is the recommendation to stay in the safe side, in all cases. In that situation, you can even set your device priorities to Highest (which means fastest Presto - at the expense of a less responsive graphic system).
Read here [ http://msdn.microsoft.com/en-us/library/windows/hardware/ff553890(v=vs.85).aspx ] about watchdog service." [
https://www.thearender.com/site/index.php/features/engines/presto-gpu-cpu.html ]
Did the 8 GPU limit apply to Windows as well or just OSX? I cant remember...
My current impression is that there are a number of factors at play when one tries to install a number of GPUs in a system that exceeds the number of factory based PCIe slots. My current belief is that an 8 GPU limit is much too high (about twice as much too high) for an Apple system (pre-2013) running OSX. I also do not know for certain whether an 8 GPU limit applies to the latest Windows OSes, but I am beginning to doubt that it's based on the OS, but rather has more to with the particular system, application and GPUs. I haven't seen any evidence of a single system running more than 8 GPUs on OctaneRender, FurryBall, TheaRender or Redshift3d. Moreover, I haven't been able to find any GPU maximums for either FurryBall (" Do I need FurryBall licence for each GPU? No, the license is per workstation - number of GPUs in computer is UNLIMITED") or TheaRender. Within the next few weeks, I intend to install Amfeltec (SKU-042-43) GPU oriented PCIe 4-way Splitters in my quad E5-4650 SuperMicro Systems. I'll keep you apprised about what I learn from that process because I hope to test, at a minimum, 16, 12 and 8 GPU processor configurations.
P.S. I opted for the Amfeltec splitters over the chassis because of the transfer speed and price of the splitters and the fact that I have some classic tower motherboard chassis that I can Dremel into GPU expansion service [ see "Waste Not - Want Not" post no. 1223,
above.]
