MP 1,1-5,1 testing TB3 AIC with MP 5,1

[Woof Woof]

I've tried searching the thread, but is the Titan Ridge card a PCIe x4 card, and limited to ~20Gbit over TB3 because of the Gen2 slot in the cMP?

 

[joevt]

I've tried searching the thread, but is the Titan Ridge card a PCIe x4 card, and limited to ~20Gbit over TB3 because of the Gen2 slot in the cMP?

Thunderbolt 3 add-in cards are PCIe 3.0 x4.

PCIe 3.0 x4 is 31.5 Gbps (3938 MB/s) max but usually allows up to 3500 MB/s (28 Gbps) after considering PCIe protocol overhead (look for benchmarks of the fastest NVMe drives or the Host To Device or Device To Host speeds of graphics cards).

Thunderbolt 3 is limited to 22 Gbps (2750 MB/s) of PCIe traffic per Thunderbolt port. Both Thunderbolt 3 ports of a Thunderbolt 3 controller together can do about 23 Gbps (at least that's all I could get from a software raid). This means a Thunderbolt 3 controller does not use a PCIe 3.0 x4 link to its full potential even if you try to RAID both of its ports together. I don't know why (technical, power, heat, size?). Actually, I haven't tried a Thunderbolt + USB 3.1 gen 2 RAID yet. That might be interesting - it could be a way to get closer to PCIe 3.0 x4 max.

Mac Pro only has PCIe 2.0. PCIe 2.0x4 is 16 Gbps (2000 MB/s) max but usually allows up to 1600 MB/s (13 Gbps). Even in a PCIe 2.0 x4 slot, PCIe traffic over Thunderbolt 3 is faster than Thunderbolt 2 (by a small amount). Some PC Laptops connect their Thunderbolt 3 controller to PCIe 2.0 x4 or PCIe 3.0 x2 (both similar).

The rest of the 40 Gbps of a Thunderbolt 3 cable can be used by DisplayPort as the Thunderbolt controller has separate inputs for two DisplayPort signals.

There are ways to add PCIe 3.0 slots to a Mac Pro to get the full performance of the Thunderbolt 3 controller. Anything with a PCIe 3.0 switch with a PCIe 2.0 x8 or greater upstream link could be used to do that. This usually means a two or four M.2 or U.2 carrier PCIe cards with adapters or a PCIe expansion box. Expensive.

 

[Woof Woof]

Thunderbolt 3 add-in cards are PCIe 3.0 x4.

There are ways to add PCIe 3.0 slots to a Mac Pro to get the full performance of the Thunderbolt 3 controller. Anything with a PCIe 3.0 switch with a PCIe 2.0 x8 or greater upstream link could be used to do that. This usually means a two or four M.2 or U.2 carrier PCIe cards with adapters or a PCIe expansion box. Expensive.

That is exactly where I was going with the question. I was curious if anyone had used a PCIe switch to get a Gen3 x4 slot out of the deal.

I looked at amfeltec, my goto for oddball risers, and they don't do a M.2 to PCIe x4. There are items like these that are close: https://minerparts.com/product/pcie-4x-to-ngff-m-2-adapter-with-4-pin-power-cable/

Now to just wait for someone with a switch card and a Titan Ridge card to give it a go.

Edit: Finally going through and reading all the posts. I see this was an early thought.

 

[wjesse]

There are ways to add PCIe 3.0 slots to a Mac Pro to get the full performance of the Thunderbolt 3 controller. Anything with a PCIe 3.0 switch with a PCIe 2.0 x8 or greater upstream link could be used to do that. This usually means a two or four M.2 or U.2 carrier PCIe cards with adapters or a PCIe expansion box. Expensive.

That is exactly where I was going with the question. I was curious if anyone had used a PCIe switch to get a Gen3 x4 slot out of the deal.

I looked at amfeltec, my goto for oddball risers, and they don't do a M.2 to PCIe x4. There are items like these that are close: https://minerparts.com/product/pcie-4x-to-ngff-m-2-adapter-with-4-pin-power-cable/

Now to just wait for someone with a switch card and a Titan Ridge card to give it a go.

Edit: Finally going through and reading all the posts. I see this was an early thought.

I tested my GC-Titan Rideg with HightPoint SSD7102, like this

Unfortunately, the link speed shows 2.5GT/s and link width is X4, use a PCIe switch to get a Gen3 x4 seems not useful?
I didn't warm boot from Windows 10.

 

[joevt]

I see this was an early thought.

Not just a thought - there are pictures! I'm thinking that an M.2 to U.2 adapter with a U.2 cable and a U.2 to PCIe x4 adapter would be better than what I tried with the amfeltec (similar to wjesse's Highpoint) because the U.2 cables are more flexible and can be detached from both ends.

There's a super expensive but much nicer method to add PCIe 3.0 slots (I haven't tried it with a Mac Pro yet):
https://www.bhphotovideo.com/c/product/980261-REG/dynapower_usa_na255a_xgpu_netstor_6_slot_pcie.html

I tested my GC-Titan Rideg with HightPoint SSD7102, like this

Unfortunately, the link speed shows 2.5GT/s and link width is X4, use a PCIe switch to get a Gen3 x4 seems not useful?
I didn't warm boot from Windows 10.

You're looking at the wrong devices. Even a real Mac shows 2.5 GT/s x4 for those devices (but real Mac's don't list those devices in the PCI section). Those devices are internal to the Thunderbolt controller, so Intel doesn't bother with setting the link speed register to something reasonable. You have to look at the speed of the upstream Titan Ridge bridge. The lspci command will give the current link speed values - or look for my lspcitree.sh script. In my Mac mini 2018, The upstream Titan Ridge bridge has an Alpine Ridge 4C 2015 device id [8086:1578]. The NHI is 15eb and the USB controller is 15ec just like in your screen shot. Downstream bridges are 15ea. Or you could try a benchmark. Anything over 1000 MB/s will prove that it is at least 5.0 GT/s. Anything over 2000 MB/s will prove that it is 8.0 GT/s.

 

[zozomester]

Still waiting for the delivery. It is shipped, but it will take another few weeks to arrive!

Please share your experience if the card came out. I look forward.
Thank!

 

[wjesse]

Or you could try a benchmark. Anything over 1000 MB/s will prove that it is at least 5.0 GT/s. Anything over 2000 MB/s will prove that it is 8.0 GT/s.

I still don't have a chance to try a benchmark, I can't warm boot from Windows 10 drive to make TB3 work, there's an issue stop me, cMP can't find my Windows boot drive when SSD7102 is plug in PCIe slot.

 

[byally]

Dear community! I have a questions about sleep facility in cMP with Titan Ridge equipped:
1) is it working for usb 3 devices (no thunderbolt no warm boot) - can I put a flash disk then sleep then again wake?
2) is it working for TB devices connected to mac os after warm boot? or one have to reboot every time?
3) do i need some special steps to connect usb 3 devices? can i connect usb 3 flash disk anytime or should i connect it only during boot process etc

Please share your experience.

Thank you!!!

 

[Woof Woof]

What the hell, I'm going to hop in and join the party. Going to try the Amfeltec squid with the Titan Ridge card.

I'm liking the ADT-Link riser that wjesse tried. They have that riser with a right angle x4 connector, so I can try mounting the Titan Ridge card upside down in the 4th slot, and draw power for the riser from the SATA connector right above it. I can use a blank PCIe card for support.

Probably steal USB 2 for one of the TB3 connectors from the Bluetooth jack since I don't care about bluetooth.

I've got an Apple Thunderbolt display and TB3 <-> TB2 adapter laying around at the office.

What would be the best way to test link speed of TB3? I was thinking about a TB3 to NVME adapter, but I don't know which might be reliable.

 

[joevt]

What the hell, I'm going to hop in and join the party. Going to try the Amfeltec squid with the Titan Ridge card.

I'm liking the ADT-Link riser that wjesse tried. They have that riser with a right angle x4 connector, so I can try mounting the Titan Ridge card upside down in the 4th slot, and draw power for the riser from the SATA connector right above it. I can use a blank PCIe card for support.

Probably steal USB 2 for one of the TB3 connectors from the Bluetooth jack since I don't care about bluetooth.

I've got an Apple Thunderbolt display and TB3 <-> TB2 adapter laying around at the office.

What would be the best way to test link speed of TB3? I was thinking about a TB3 to NVME adapter, but I don't know which might be reliable.

I tried four of those ADT-Link risers in the amfeltec. I think my problem was that folding the cables too much reduced their quality. Or maybe spacing four of them at 0.8" was adding too much stress to the cables. I cut a rectangular hole in the back of the amfeltec for the cables to exit the case (since the amfeltec is low profile, there was plenty of room). I think a M.2 to U.2 adapter would be better, because then you can buy any length U.2 cable, and the cable is more flexible, and can be disconnected at either end - but the U.2 connector on the amfeltec side might take more than one slot of space? The HighPoint SSD7120 probably solves that problem. On the other side, the U.2 to PCIe x4 adapters I've seen seem to be wider than necessary.

About the Apple Thunderbolt display: I'm not sure it works properly with the GC-TITAN RIDGE. I believe Thunderbolt 1 devices do work (Apple Thunderbolt to FireWire or Gigabit Ethernet adapters) but maybe there's an issue with PCIe + DisplayPort traffic over Thunderbolt 1 from a GC-TITAN RIDGE? I don't know for sure that there is a problem.

 

[cdf]

Probably steal USB 2 for one of the TB3 connectors from the Bluetooth jack since I don't care about bluetooth.

Nice. I'm currently working on a hub solution for taking USB 2 from the bluetooth header. I don't believe that this has actually been done yet. A possible complication is that the header does not provide 5V but rather 3.3V. Hopefully, the Titan Ridge only needs the data lines... If you try this, please let us know!

 

[Woof Woof]

Nice. I'm currently working on a hub solution for taking USB 2 from the bluetooth header. I don't believe that this has actually been done yet. A possible complication is that the header does not provide 5V but rather 3.3V. Hopefully, the Titan Ridge only needs the data lines... If you try this, please let us know!

Damn, I was on the hunt today for various bits and pieces, and ran across a 4 pin USB to dual 9-pin USB 2 internal header HUB. Tiny little thing too. If I find it again, I'll post it. Hmm... I'll have to put a meter on that connector, and see if it is 3.3v.

 

[cdf]

I'll have to put a meter on that connector, and see if it is 3.3v.

It is. My meter showed 3.32

 

[Woof Woof]

I tried four of those ADT-Link risers in the amfeltec. I think my problem was that folding the cables too much reduced their quality. Or maybe spacing four of them at 0.8" was adding too much stress to the cables. I cut a rectangular hole in the back of the amfeltec for the cables to exit the case (since the amfeltec is low profile, there was plenty of room). I think a M.2 to U.2 adapter would be better, because then you can buy any length U.2 cable, and the cable is more flexible, and can be disconnected at either end - but the U.2 connector on the amfeltec side might take more than one slot of space? The HighPoint SSD7120 probably solves that problem. On the other side, the U.2 to PCIe x4 adapters I've seen seem to be wider than necessary.

About the Apple Thunderbolt display: I'm not sure it works properly with the GC-TITAN RIDGE. I believe Thunderbolt 1 devices do work (Apple Thunderbolt to FireWire or Gigabit Ethernet adapters) but maybe there's an issue with PCIe + DisplayPort traffic over Thunderbolt 1 from a GC-TITAN RIDGE? I don't know for sure that there is a problem.

It looks like you are using the gen 3 Squid (I see the fan). I grabbed the Gen 2 squid (gen 3 nvme connections) because amfeltec support said it was faster than the gen 3 model in the classic Mac Pros. Might be interesting to see if it behaves any different.

Yeah, the ADT-Link riser is OTW from China, and I think you are right... using an M.2 adapter might be more reliable, but block the slot above or below. I didn't realize the ADT was chinesium, or I might have kept looking. The best risers I have used use 3M twinax cable that can be bent and mutilated. I've done it, putting really hard crimps in a gen 3 x16 riser, and it just kept on going.

 

[joevt]

It looks like you are using the gen 3 Squid (I see the fan). I grabbed the Gen 2 squid (gen 3 nvme connections) because amfeltec support said it was faster than the gen 3 model in the classic Mac Pros. Might be interesting to see if it behaves any different.

Amfeltec support is wrong. The problem with the gen 3 was that it ran at gen 1 speed in the Mac Pro but there is a workaround for that to make it run at gen 2 speed, and the workaround is not required anymore when you use the Mojave firmware update for the MacPro4,1 and MacPro5,1. Read the amfeltec thread or the Highpoint thread. The gen 3 cards are faster because while the upstream in a Mac Pro must be gen 2, the downstream (NVMe drives) can be gen 3, giving up to twice the performance of NVMe drives in a gen 2 only card.

 

[Woof Woof]

Yeh, I had a small argument with the sales guy at Amfeltec. I had made a stupid assumption the gen 2 card was still gen 3 downstream.

 

[crjackson2134]

Yeh, I had a small argument with the sales guy at Amfeltec. I had made a stupid assumption the gen 2 card was still gen 3 downstream.

Ha Ha Ha...
Got to love it when stuff like this happens... It's a humbling experience when you realize you were wrong.
It happens to all of us sometimes...

 

[Aiwi]

Tested the following today:
RTX 2070 -(dual DP to)-> Titan Ridge -(Thunderbolt to)-> Ultrafine 5K

• No boot screen
• No image in macOS Mojave 10.14.5
• Working 5K60 image in Windows 10 1903 (after power cycling the Ultrafine 5K using a plug with a switch)
• No brightness/sound/video settings in Mojave or Windows

We will have to wait for RTX drivers in order to make this usable in macOS. (duh) Sad to not have a boot screen though.

---

For the fun of it I also tried the following combination:
RTX 2070 -(dual DP to)-> Titan Ridge -(Thunderbolt to)-> Ultrafine 5K
RTX 2070 -(single DP to)-> "Regular 1440p Monitor"

• Boot screen only on "Regular 1440p Monitor"
• Image only on "Regular 1440p Monitor" in macOS Mojave 10.14.5
• Working 5K60 image on Ultrafine and 1440p on "Regular 1440p Monitor" in Windows 10 1903
• No brightness/sound/video settings in Mojave or Windows

---

Mac Pro 5,1 X5690 24GB
144.0.0.0
Slot 4. SATA3 adapter with a Samsung 850
Slot 3. Gigabyte Titan Ridge
Slot 2. NVMe adapter with a Samsung 970 Evo
Slot 1. Nvidia RTX 2070

 

[joevt]

No brightness/sound/video settings in Mojave or Windows

I believe there are apps for Windows and MacOS that can change the brightness of the LG UltraFine 5K using DDC/CI (over DisplayPort) instead of HID (over USB).

USB requires PCIe communication - which should work in Windows, but maybe Windows doesn't have a LG UltraFine 5K driver for brightness control? Check the Windows Device Manager to see if the PCIe devices of the LG UltraFine 5K are working (including the USB controller). Or maybe having to power cycle the display means the PCIe devices can't work without proper hot plug support. Maybe the PCIe devices will appear if you don't power cycle the display - use the second display to check the Device Manager. Maybe the display can be power cycled after the computer starts up, but before Windows loads?

 

[Aiwi]

I believe there are apps for Windows and MacOS that can change the brightness of the LG UltraFine 5K using DDC/CI (over DisplayPort) instead of HID (over USB).

USB requires PCIe communication - which should work in Windows, but maybe Windows doesn't have a LG UltraFine 5K driver for brightness control? Check the Windows Device Manager to see if the PCIe devices of the LG UltraFine 5K are working (including the USB controller). Or maybe having to power cycle the display means the PCIe devices can't work without proper hot plug support. Maybe the PCIe devices will appear if you don't power cycle the display - use the second display to check the Device Manager. Maybe the display can be power cycled after the computer starts up, but before Windows loads?

You're probably right! I've decided to return the Titan Ridge and the Ultrafine already though. So I can't check. Not worth loosing boot screen and the unaccelerated Mojave support. Will hold off until there are 5K using a single DP 1.4 connection instead.

Might reconsider if Apple decides to launch an aluminium 6K display on WWDC. That one will probably be thunderbolt only..

 

[joevt]

You're probably right! I've decided to return the Titan Ridge and the Ultrafine already though. So I can't check. Not worth loosing boot screen and the unaccelerated Mojave support. Will hold off until there are 5K using a single DP 1.4 connection instead.

Might reconsider if Apple decides to launch an aluminium 6K display on WWDC. That one will probably be thunderbolt only..

The LG 34WK95U-W 34" 21:9 UltraWide 5K is a DisplayPort 1.4 display. It also supports dual cable DisplayPort 1.2 through Thunderbolt like the UltraFine 5K. The display is only 2160 tall instead of 2880, but this allows it to use 10 bpc where as DisplayPort 1.4 would only 2880p to do 8 bpc (DisplayPort 1.4 only has 75% the bandwidth of dual cable DisplayPort 1.2). DisplayPort 1.4 with Display Stream Compression (DSC) is required for higher bit depths / resolutions / refresh rates. I don't know of any displays using DSC and I don't know if MacOS or Windows drivers supports it.

 

[Rastafabi]

[…] Will hold off until there are 5K using a single DP 1.4 connection instead. […]

Here you are.

 

[Aiwi]

Here you are.

Cheers! Will hold off until after the WWDC though, you never know what "must have" monitor might be released by the Cupertino team

 

[Synchro3]

Here you are.

This monitor is only 8 Bit: https://www.displayspecifications.com/en/model/c0a11069

 

[joevt]

This monitor is only 8 Bit: https://www.displayspecifications.com/en/model/c0a11069

It's worse than that. It's 6 bit with FRC. It can't do 10 bpc with DisplayPort 1.4 at 5K without Display Stream Compression.