NVMe with ST-NGFF2013-C; Vega Internal GPU; Mac Pro 2013 (6,1)

[CodeJingle]

I was referring to the "hobby", not your day job.

You are asking me what my hobby work is? At least for this project anyway. The type of job isn't clear so I wouldn't easily categorize it. Somewhere under the umbrella of hardware engineer or electrical engineer. Regarding specifics of this project, well if I were structuring my hobby work as a SOW, I would say the first deliverable is documenting the layers, even just photos, of the three boards CPU Riser Card, Logic Board, I/O Board. There are probably 30 layers in total. I've documented 13 so far. I am still at this stage.

The second deliverable is analyzing the layers to document the pinout of the three logic board flex connectors, not necessarily all ~920 pins but whatever pins aren't passing through the middle-man chipset. There are at least roughly 328 pins, in particular, I am interested in documenting, though the more, the merrier.

The third deliverable is to connect another pair of GPU to the Mac Pro and get them to work in place of the built-in GPU. Probably Vega 64. Though doing this requires several other significant tasks to be completed, such as moving the internal assembly of the Mac Pro out of the trash can into a legacy tower to improve the thermals. Also documentation of the CPU pins, and documentation of which of those CPU pins map to two GPU connected in a typical server setup. This deliverable has perhaps the most risk.

The fourth deliverable would be to define a process with minimal impact for others to replicate. This will involve designing an adapter and possibly an external enclosure so others can connect external GPU at full bandwidth without having to take apart their trash can. Maybe similar to eGPU but connects directly to the logic board instead of the rear pass-thru.

I've completed minimal documentation of one of the three boards so I would call that a success milestone even if I haven't finished the first deliverable. Otherwise completing the deliverables are measures of success. I've never done anything like this before so I would need to be a bit farther along before I start imposing a schedule on myself. Most of the work of finishing the first milestone is figuring out a more efficient way to get to the next layer in the same board. I've learned about several tools and processes I previously didn't know existed. There is also a minimum amount of pacing required to not destroy the board in the process. There are also constraints on when and where this process can take place because it is loud and in some ways dangerous (inhaling microscopic shards of glass is probably bad).

Not sure what the ultimate goal is. Knowledge. Experience. A better Mac. Though if I succeed in all other aspects but the resulting system is still not capable of driving high-end VR then it is still a failure.

 

[CodeJingle]

Here is a 'Numbers for macOS' document for dual GPU slots. What is filled out currently is the generic pinout value not the specific CPU origin pin. The first GPU has PCIe lanes numbered 0-15, the second GPU slot has PCIe lanes numbered 16-31. I'll be adding the CPU origin pins to this document after logic probing a Dell Xeon board that has dual GPU.

overlaid against images of actual PCIe x 16 slots but not easily printable
https://www.dropbox.com/s/d2au1mdz77jqowy/BlackAndBlue.numbers?dl=0

printable version without a graphic overlay
https://www.dropbox.com/s/wt3t8o20ozsb83d/BlackAndBluePrintable.numbers?dl=0

Source material for the generic pinout
https://en.wikipedia.org/wiki/PCI_Express

 

[CodeJingle]

I was referring to the "hobby", not your day job.

Replacing the GPUs is just one option. The Mac Pro 2013 boots normally with only one GPU installed, freeing up 16 PCIe slots. And knowing the pinout, if adding another NVMe is picked up by the OS then I could try booting up 5 NVMe instead of 1, although doing that isn't any fun right now. I'd be more inclined to figure out a way to hook up 4+ NVMe drives if a future version of macOS adds native support for APFS raid0 including making it bootable. Compared to two GPU and hard drive throughput of 1.5 gigabytes per second, I'd be happy with one GPU and hard drive throughput of ~12 gigabytes per second.

 

[AidenShaw]

Replacing the GPUs is just one option. The Mac Pro 2013 boots normally with only one GPU installed, freeing up 16 PCIe slots. And knowing the pinout, if adding another NVMe is picked up by the OS then I could try booting up 5 NVMe instead of 1, although doing that isn't any fun right now. I'd be more inclined to figure out a way to hook up 4+ NVMe drives if a future version of macOS adds native support for APFS raid0 including making it bootable. Compared to two GPU and hard drive throughput of 1.5 gigabytes per second, I'd be happy with one GPU and hard drive throughput of ~12 gigabytes per second.

Unfortunately, increasing sequential drive throughput to 12 GBps might have a very minor improvement in the system performance.

And the MP6,1 is headed for the scrap heap.

 

[CodeJingle]

color-coded processor map for Xeon E5 v2 (open numbers document for full view the thumbnail is just a preview)
https://www.dropbox.com/s/dpzd7eywu13di12/XeonE5v2_no_image_color_coded_2018032001.numbers?dl=0

[doublepost=1521596525][/doublepost]

And the MP6,1 is headed for the scrap heap.

Yeah, I know it is headed for the scrap heap. You or someone else might appreciate it if I duplicate some of this same effort for the new Mac Pro once it comes out. So let's just call what I am doing now practice. I gotta have something to do while I wait.

 

[jclmavg]

And the MP6,1 is headed for the scrap heap.

Your negativity towards the trashcan is well-known, you don't even own one. Why all the whining? Every Mac will end up on the scrap heap eventually.

 

[saulinpa]

Didn't you hear? The newly released iMP1,1 is headed for the scrap heap along with the iPhone X. We just weren't told the day or year.

Also news: Tim Cook is leaving Apple. Sometime in the next 30 years.

 

[itdk92]

Apple never said it is a failed turkey. If they felt that way, why even make the iMac Pro, which is even more closed than the 6,1 ever was.

They *clearly* stated, between the lines, that they f***ed up with the Mac Pro 6.1, both in their technological assumptions, and design-wise.

 

[jclmavg]

They *clearly* stated, between the lines, that they f***ed up with the Mac Pro 6.1, both in their technological assumptions, and design-wise.

They never said that, having to "read between the lines" is just a poor excuse to drive home your own interpretation of things. If design-wise the 6,1 is flawed they just repeated their greatest mistake with the iMac Pro, which is even more closed as a system than the 6,1. The only real f*ckup I would agree on is the graphics part of the 6,1 if eGPU remains disabled.

 

[MarkJames68]

They *clearly* stated, between the lines, that they f***ed up with the Mac Pro 6.1, both in their technological assumptions, and design-wise.

If that’s the case then why are they still selling and supporting it?

I would say more correctly that they designed themselves into a thermal corner and overestimated the reaction people would have with the design.

I agree that if they keep the eGPU disabled it will be a huge negative.

 

[CodeJingle]

Whenever I finish this project the recognition if any will be short lived so I'll just start over with the new Mac Pro once it comes out. I'm not going to worry about WWDC this year. The money and time are better spent on supplies to finish the project and to build up that experience and to learn those reusable skills. I'll be in a better position next year to afford the WWDC trip regardless of reimbursement.

If the new Mac Pro isn't completely open hopefully that is something that can be fixed by the community much sooner than 4.5 years later.

 

[itdk92]

They never said that, having to "read between the lines" is just a poor excuse to drive home your own interpretation of things. If design-wise the 6,1 is flawed they just repeated their greatest mistake with the iMac Pro, which is even more closed as a system than the 6,1. The only real f*ckup I would agree on is the graphics part of the 6,1 if eGPU remains disabled.

From that famous interview at BuzzFeed:

“Being able to put larger single GPUs required a different system architecture and more thermal capacity than that system was designed to accommodate,” the exec is reported as saying. “So it became fairly difficult to adjust.”

“We wanted to do something bold and different. What we didn’t appreciate completely at the time was how we had so tailored that design to a specific vision that in the future we would find ourselves a bit boxed in — into a circular shape.”

And Schiller offered something even rarer: an apology. "The current Mac Pro ... was constrained thermally and it restricted our ability to upgrade it," he said. "And for that, we’re sorry to disappoint customers."

^THIS and the small fact that they ran so hot that the D500 and D700, even being mediocre GPUs, burned out and created render glitches all over the place in many cases (hence Apple unofficial replacement program for D500 and D700, later downclocked to the D300 performance-level via software).

The problem with the 6.1 design is not only that it's closed (that's the part they apologized for) but mostly that it's thermal capacity was not even enough to cool down some of its own standard components (see three lines above).

iMac Pro has "only" the problem of being locked down, but at least the thermals seems to work fine with the new fan system, although top-models stil throttle a little under extreme stress.

Lastly, the newer problems - and here I agree with you - will be the probable lack of eGPU to force people into buying newer Macs.

Sure you want to ask AidenShaw again why he is negative about the 6.1?
[doublepost=1521659884][/doublepost]

If that’s the case then why are they still selling and supporting it?

I would say more correctly that they designed themselves into a thermal corner and overestimated the reaction people would have with the design.

I agree that if they keep the eGPU disabled it will be a huge negative.

Apple already gave you an apology, but there's a limit to how much they are willing to admit, and honestly, why should they totally, unequivocally, shoot themselves on the foot? Much better to give a little discount (done), downclock the D500 and D700 models to reduce overheat cases (done), and wait for the whole matter to phase away after a good new product (arguably iMac Pro) and the 2018-19 Mac Pro.

As another example, see the Maps case. Heads rolled, apologies were made, but they (obviously) still didn't go back to Google Maps.

 

[CodeJingle]

iMac Pro has "only" the problem of being locked down, but at least the thermals seems to work fine with the new fan system, although top-models stil throttle a little under extreme stress.

Part of what Apple did with iMac Pro to reduce the thermal load is to create custom versions of the CPUs that have significantly more cache but are also underclocked. This sounds good in theory if it truly gives the same performance as the non-custom version while creating less heat, but it also adds significant cost to the CPUs. It may have been better for Apple to not create custom versions of those chips and just beefed up the iMac Pro thermal capacity even more and then the cost of ownership would go down. So I'm a little iffy on that part of how Apple solved the thermal problem for the iMac Pro.

The problem with the 6.1 design is not only that it's closed (that's the part they apologized for) but mostly that it's thermal capacity was not even enough to cool down some of its own standard components (see three lines above).

Clearly, Mac Pro 2013 has issues. I've heard plenty of times the answer to go with Mac Pro 2012 or wait and buy Mac Pro 2018+. As an exercise in futility, if you have a Mac Pro 2013 that is already maxed out on RAM, CPU, and hard drive (either one NVMe or up to three drives with the Amfeltec AngelShark), then how to upgrade graphics? Having no thermal capacity left inside the enclosure is actually not that bad based on Apple's marketing for the Mac Pro 2013 that we should settle on certain kinds of upgrades being available only externally through the thunderbolt ports. So the idea that we can't upgrade the GPU internally because it would fry the computer ... and GPUs more powerful than D700s have to exist externally outside of the trash can ... is fine according to Apple.

By uncovering the pinout and using that knowledge to connect other PCIe devices directly to the bus even though those devices reside externally outside the trash can, that mitigates the failures of the Mac Pro 2013's thermal design while also adhering to their "external upgrades are not so bad" marketing. The hard part is figuring out the pinout. Failures are to be worked around through mitigation if mitigation is possible. In this case, mitigations seem not only possible but probable enough to be worthy as a project whose major purpose is unrelated. I'm not concerned with the work being throw-away because I care more about the skills and experience gained throughout which I can apply later to a different architecture and a different board design.

If there are even more failings of the Mac Pro 2013 besides being a closed design and having limited or no thermal overhead (if there are other things Apple hasn't disclosed) then yeah maybe it would be best to give up on it and not look back.

 

[CodeJingle]

I've started mapping actual pinout from the processor to the dual GPU. This is based on the Dell board for T3600 since I am mapping to standard PCIe x 16 slots, but I'm going to bet for now that the mapping is pretty close to the Mac Pro 2013. I've only mapped receive pins so far; there seems to be an issue with testing transmit pins I may need a more advanced type of continuity test for that. Some of the P and N pins labels might be swapped. I generated the original table of CPU pad labels programmatically based on Intel's official Xeon documentation so any number of issues could cause a label mismatch. I'll be correcting documentation bugs as I go but even now swapped labels are ok because my tables are physically accurate regarding layout. The relative physical location is correct even if the labeling is slightly off.

dell board info



http://www.dell.com/support/article...e-precision-t3600-desktop-workstation?lang=en

http://i.dell.com/sites/doccontent/...ments/Dell-Precision-T3600-Spec-Sheet-tab.pdf

dual-GPU standard PCIe x 16 connector pinout with cross-reference (same link as before - I will update as progress continues)
https://www.dropbox.com/s/wt3t8o20ozsb83d/BlackAndBluePrintable.numbers?dl=0

color-coded processor pinout with cross-reference (same link as before - I will update as progress continues)
https://www.dropbox.com/s/dpzd7eywu13di12/XeonE5v2_no_image_color_coded_2018032001.numbers?dl=0

 

[CodeJingle]

The non-workstation Vega 64 16GB card (the frontier version) has not gone down in price at all, still $800, and the reviews not very good. The workstation equivalent, Vega Pro WX 9100, is a $1600 card, coincidentally the same price as a WWDC ticket lol. So I'm not affording that right now. I bought a FirePro S9000 to test if I eventually get the pinout correct or not (which I think is kind of lame).

https://www.amd.com/en-us/products/graphics/workstation/firepro-3d

https://www.newegg.com/Product/Product.aspx?Item=N82E16814105073

https://www.newegg.com/Product/Product.aspx?Item=N82E16814105087

https://www.ebay.com/itm/122998356472

 

[CodeJingle]

Here is a not-so-quick video tutorial on microscopic probing (or nearly microscopic). About 34 minutes duration. With several digressions and tangentials. If it was truly microscopic just add a microscope to the mix. But in this case, I can barely see the pins and if I was 10 years older I'm pretty sure I would need a microscope (and bifocals) to do this kind of work. My phone was out of space so I had to record, upload, and delete to make room for more so it ended up broken into three videos. Also to help establish context, by the time I finished recording it was ~ 2 am. Unedited (not even trimmed).



https://www.dropbox.com/s/m0fq9z7p054uexk/20180324_234939.mp4?dl=0

https://www.dropbox.com/s/dbkmbn39my09o7z/20180325_003214.mp4?dl=0

https://www.dropbox.com/s/16dl6wirb8nie5y/20180325_013957.mp4?dl=0

I had a tiny breakthrough for continuity testing the PCIe transmit pins. See follow up post for updated Numbers documents.

I said I was going the Colin Furze route, but I am close to breaking down and doing probing directly on a Mac Pro 2013 (6,1). I am trying to obtain another set of Mac Pro Logic Board, CPU Riser Card, and I/O Board exclusively for probing. I already have two sets of those boards, but one set I am sanding down and the other set is in my personal Mac Pro which I am reluctant to take apart because I use it as my main 'day job' dev box (my employer lets me use my personal computer). My goal is still to fully sand down and document all the layers in addition to using probing to document pinouts (and the work in each minus some overlap may be complimentary).

I will still probe the PC board to map the same pins to regular PCIe x 16 slots so the work isn't throw away I need to do the same work on both the PC board and the Mac Pro boards.

It looks like the PCIe pins for GPU are NOT going through the platform controller they are direct connections between the CPU and GPUs (I was originally told by RyanXM to worry about the platform controller). Still, this is the PC board, but I expect the same from the Mac Pro boards.

Here are links to supplies for microscopic probing. Disclaimer here that I am not 100% sure these covers fit for these probes. But it seems likely - the outer diameter of the cover is 0.86mm, the inner diameter of the cover is 0.72mm, and the largest diameter of the probe (at the barrel) is 0.68mm, so it seems likely that a 0.68mm plug would fit snugly into a 0.72mm receptacle (not too loose, not too tight). It has been almost a year since I took over the hardware lab at my day job but the guy who left didn't leave enough info for me to reorder when the lab runs out of supplies. So when I wanted to order more of the lab's microscopic probes for personal use I had to scour online and eyeball it.

For continuity testing, I solder the cover to the probe barrel, and I solder the probe barrel to the pogo pin tip. Less false negatives that way. The pogo pin part of the probe is used more for temporary mounting of an IC or board in a jig for stuff like programming and is more of a distraction otherwise. Though it may be up to personal taste.

(mini alligator clips that also attach to the tip of multimeter probes)
https://www.amazon.com/dp/B078W6FFCP


(very well made and pre-tinned 30 gauge wires attached to probe covers which you slide the probe into)
https://www.amazon.com/gp/product/B075FML78J


(probe covers only which you slide the probe into)
https://www.amazon.com/gp/product/B071DNQ3B9


(probes; pogo-pin; 0.45mm diameter spear tip)
https://www.amazon.com/gp/product/B01N95D7BF


(cheap multimeter that I have experience working with, that also does continuity testing ... otherwise for a non-cheap multimeter get a FLUKE)
https://www.amazon.com/dp/B00SU0W2C6

 

[MarkJames68]

I’ll admit I will probably never pursue this GPU hack, but I am thoroughly enjoying the work that is going into this. And I’m forever thankful for the work that went into the NVMe solution.

 

[CodeJingle]

I mapped all of the PCIe RX and TX pairs for the dual GPU (on the PC board not Mac Pro). For the second GPU, I did just a little bit of probing and then extrapolated the results, so P and N may be swapped for some but as pairs, they should all be correct. I'll manually probe all the documented pins next weekend and correct any swapped pairs then. See follow up post for links to updated Numbers documents. I'm already at the point now where I am blocked without another CPU Riser Card and Logic Board to probe the Mac Pro 2013 directly but hopefully, that will be fixed soon.

 

[CodeJingle]

I updated the numbers documents with the cross-references. I've italicized the cross-references to make them easier to spot. The original links point to the same updated document but I'll also include them here again. I've also added some basic color coding to the GPU numbers document so now both are color-coded. I have to double check everything I will manually probe all the connections again this weekend and update afterward. I've mapped all the PCIe pins of the two GPU slots. That makes in total 128 pins mapped between the CPU and the two GPU slots (2 slots x 16 lanes per slot x 4 pins per lane). Though this is still on the PC board, not the Mac Pro.


https://www.dropbox.com/s/mur8i7ne3k6hx9o/XeonE5v2_no_image_color_coded_2018032001.numbers?dl=0


https://www.dropbox.com/s/wt3t8o20ozsb83d/BlackAndBluePrintable.numbers?dl=0

Progress with my limited time and resources means most likely by the time I am done there is no opportunity for others or myself to profit off of my work. The risk is extraordinary so being 'done' most likely means permanently blocked by an unsolvable problem so "doesn't work at all" versus successfully getting it working to connect a pair of external GPUs by plugging directly into PCIe bypassing the external eGPU method and also with the required setup to be under the thermal limits and the NVRam is not limiting those external GPUs into thinking they must be D300/D500/D700.

And even past the risk of it not working there is a chance to get sued by Apple. Maybe they should sue. If you assume they would otherwise sue then the only reason Apple would not sue is as a gesture of upholding the principles the Mac Pro 6,1 was shipped on which is the promise of upgradeable GPUs.

It is better for a company to not contact me. No one is going to sponsor my work. When I explain the scale and complexity of the project and the full scope of the risks they balk. It isn't my job to convince someone to invest in the work I am doing when I am just doing it as a hobby for fun (at least ideally part of a hobby's purpose is for fun). If they want to accelerate the work so I either fail faster or complete successfully faster so there is still time to go to market, then they need to take risk. There is no opportunity without risk.

However, if you have a personal interest in my project and no internal agenda for profit and would still like to help then I would be glad to talk to you. But I'm not going to convince you. If this project ends successfully maybe I'll sell out and do a Patreon page for my next project.

I'm probably going to transplant my Mac Pro into a tower casing in the next few days if I can't secure another Mac Pro for probing. I don't want to probe my Mac Pro because I use it for work and breaking it may inevitably lead to losing my job or worse downgrading to my original official work machine which is a mac mini. But I would still rather do that than wait another month. I'll risk damaging my computer to get this done. The type of person that would do this project as a hobby is going to be a little stubborn.

 

[nigelbb]

Your negativity towards the trashcan is well-known, you don't even own one. Why all the whining? Every Mac will end up on the scrap heap eventually.

As I recall not only does he not own a 2013 Mac Pro he doesn't own any Mac & hasn't owned one for 20 years. Why he continues to whine about Apple in general & the Mac Pro in particular I have no idea perhaps he just enjoys irritating Mac owners.

 

[AidenShaw]

As I recall not only does he not own a 2013 Mac Pro he doesn't own any Mac & hasn't owned one for 20 years. Why he continues to whine about Apple in general & the Mac Pro in particular I have no idea perhaps he just enjoys irritating Mac owners.

Actually, I've bought about ten MacBook Pros so far this year.

 

[CodeJingle]

Why he continues to whine about Apple in general & the Mac Pro in particular I have no idea perhaps he just enjoys irritating Mac owners.

Every group discussion should have a skeptic. If everyone agrees with everything I say I'm not sure how useful that is. I think the best ideas come out of conflict. AidenShaw is fairly mild compared to the actual trolls. He has a purpose to his posts. I had MacRumors moderator OllyW threaten to ban me because I was battling it out in a thread with a troll back in 2013 who wasn't contributing to the conversation. Trolls use phrases like 'Apple fanboys' and 'nazis'. This thread hasn't had a legit troll yet but I'm probably jinxing it here by saying it.
[doublepost=1522167994][/doublepost]

Actually, I've bought about ten MacBook Pros so far this year.

10 MacBook Pros?! You gonna share the love dude? Do you have a Mac Pro 6,1 lying around gathering dust that you bought solely for the purpose of loathing that you would lend me for probing? I swear I'll give it back when I'm done!

 

[AidenShaw]

10 MacBook Pros?! You gonna share the love dude? Do you have a Mac Pro 6,1 lying around gathering dust that you bought solely for the purpose of loathing that you would lend me for probing? I swear I'll give it back when I'm done!

I've bought no MP6,1 systems in the last five years.

But if I had, I'm sure that they'd be in the eWaste queue and I'd be happy to send them to you.

 

[CodeJingle]

I should be getting S9000 GPU tomorrow. I can't disassemble my work machine in the middle of the week so have to wait until at least Friday night. I don't have much time this weekend so realistically the best I can do is attempt to turn my Mac Pro inside-out and mount it in a tower case. I've had the equipment to do that for several months now. Maybe I'll be feeling it this weekend.

On Friday I ordered 400 microscopic probes and fancy wires to mount the probes in. They are coming from China so won't be here for a while (3 to 5 weeks). I'll have a couple weeks of waiting before I can test an external GPU. I can turn the Mac Pro inside-out and probe it in the meantime. I could wing the last part if I get far enough fast enough. It really depends. I did make a lot of progress recently. But I wanted to use the fancy wires instead of a spool of buss wire or a spool of 'wire wrap' wire.

The improvement in GPU memory performance is 10x by integrating the memory controller directly into the core and mounting the memory in the package as close as possible to that controller. This improvement applies to CPUs too not just GPUs. Pretty much every CPU in an Intel Mac is complete crap using an ancient architecture until this starts shipping. I would argue this architecture improvement should have shipped with the iMac Pro for both the CPU and GPU, not just the GPU. But maybe there is a limit to how much memory can fit in the package and still take advantage of the improved architecture. Though at least 16GB will work. I would pick 10x memory performance over memory upgradeability any day of the week.

Integrated GPUs have been a thing and they have mostly sucked but eventually, embedded GPU will be faster than discrete GPU. One might argue that in the latest iPad Pro Apple finally managed to ship a CPU with integrated GPU that is faster than discrete GPU but it is all proprietary and we don't have discreet GPU numbers to compare it to. Until Apple starts shipping high performance highly integrated GPU in their desktop CPUs, keep doing pitchfork for Mac Pro to use stock PCIe receptacles and to officially support 3rd party GPUs and to have a high thermal ceiling.

The new Radeon Pro SSG includes integrated two terabyte NVMe and treats the onboard video memory as another level of cache. Depending on perspective the card now has two terabytes of 'memory' instead of 16GB. So now we have to worry about proprietary integrated hard drive too.

ARM laptops running full Windows have already hit the market. I've heard good things though haven't tried it myself. ARM announced Cortex-A75 it will be their first processor core designed for 3+ ghz and isn't necessarily a 'mobile' processor (they designed it knowing laptops would ship with Cortex processors running full Windows OS).

Seems like we are coming to the end of an era. I'll enjoy it while it lasts I guess.

 

[CodeJingle]

A terribly unproductive 'gpu unboxing' and something something 'tower enclosure' video.

Also don't overthink it. Sheet metal is really easy to drill. I have the mounts, standoffs, rubber feet, fans. I could do something 'out of the box' (in the box). Drill a few holes in the tower. Throw in a few baking sheets for good measure. Go nuts with a temporary solution. Don't need to involve Chinese manufacturing partners for a custom solution yet. The tower isn't a solution for thermals as much as it is a solution to it being impossible to probe the Mac Pro while trapped in the trash can. The final solution for thermals could be an equivalent to eGPU box with a wire trunk (a lot of wires) going directly into the Mac Pro (might have to drill a hole into the side of the cylindrical cover). Don't know yet what the final solution will be but should be more worried about 'is it going to work at all' when I plug in one or two external Radeon GPU.

https://www.dropbox.com/s/jrat7dbyg9nacyt/20180328_193957.mp4?dl=0