Amfeltec x16 PCIe with 4 SSDs: 5900+ MB/s

Dr. Stealth · Jan 5, 2018

Pkavicky said:
I'm seriously

on the verge of getting the Amfeltec card but cannot find AHCI blades anywhere. I know it's been asked in the past, but if there are new leads on M.2 AHCI cards. I'd go for the popular 4 x 512 gig cards on the Amfeltec. Any pointers most welcome. I'm just not finding them.

I found mine on eBay (4) x 512GB. They come and go. Mostly they are used, pulled from older laptops, but I was able to buy 4 new blades in original packaging.

Also just an FYI while searching. (sorry if you already know this)... The last two digits of the part numbers...

MZHPV512HDGL-00000 - OEM Samsung
MZHPV512HDGL-000D1 - OEM Dell
MZHPV512HDGL-000H1 - OEM H.P.
MZHPV512HDGL-000L1 - OEM Lenovo

They are all the exact same Samsung blades.....

May make it easier to find.

Pkavicky · Jan 6, 2018

Very helpful, thanks.

Dr. Stealth said:
I found mine on eBay (4) x 512GB. They come and go. Mostly they are used, pulled from older laptops, but I was able to buy 4 new blades in original packaging.

Also just an FYI while searching. (sorry if you already know this)... The last two digits of the part numbers...

MZHPV512HDGL-00000 - OEM Samsung
MZHPV512HDGL-000D1 - OEM Dell
MZHPV512HDGL-000H1 - OEM H.P.
MZHPV512HDGL-000L1 - OEM Lenovo

They are all the exact same Samsung blades.....

May make it easier to find.

View attachment 745431

ekwipt · Jan 6, 2018

how do these OEM compare to 960 EVO and Pro?

]edit sorry found 1829 MB/s vs 2200 MB/s

itdk92 · Apr 22, 2018

joevt said:
The first two Barefeats articles used 512 MB SM951 AHCI M.2 drives.

The first article dealing with the gen 2 card concluded that NVMe would not be faster than AHCI because the Gen 2 card is limited to PCIe 2.0 x4 1500 MB/s. It also concluded that you wouldn't be able to boot from NVMe. That might not be entirely correct: it may be possible to copy an NVMe EFI driver to make that possible (as some have done on Hackintosh).

The second article dealing with the gen 3 card concluded that you would get best performance in a classic Mac Pro using the gen 2 card. That is not entirely correct because you can get full/better performance in the classic Mac Pro from individual M.2 devices (no raid) using the gen 3 card. It also stated that the speed advantage of NVMe flash is nullified on the Mac Pro tower. That is referring to the gen 2 card though. The tests in the article show that even an AHCI drive can exceed 2GB/s in the gen 3 card. The article states "the PCIe 3.0 Carrier Board's controller gets 'lost in translation" which is a terribly uninformative description of the problem. There is a solution to that problem but it is not mentioned.

There is a third article http://barefeats.com/hard225.html using 512 MB 960 Pro NVMe drives in High Sierra. It seems to downplay the speed advantage of the gen 3 card for individual M.2 devices. The tests show the speed advantage but the conclusion is not very enthusiastic about it - for people that don't plan to raid.

For my tests, I used four 1TB 960 Pro NVMe drivers as stated in my first tests in #212 using Apple Raid (Disk Utility.app). The updated tests in #214 used SoftRaid which gave much better results.

I was quoted $465.00 USD + $59.66 USD taxes for the card SKU-086-34 (I'm in Canada) on Oct 2. The card is expensive because of the PEX 8732 switch chip (about $170) I'm not sure what the other $300 is for. Cards that require PCIe bifurcation don't use a switch chip (they use the motherboard's switching capabilities for the slot) and are therefore much less expensive ($200?).

Sounds great.

So you would generally recommend Amfeltec Squid Gen 3 for individual drives? (without any other modificaions)

How about NVMe RAIDs?

How can you be sure this commands work on a 4.1/5.1 Mac Pro? Have you tried?

I don’t know how much Mac Pro 3.1 and Hack counts in verifying this.

Thank you in advance!

ezylstra · Apr 22, 2018

itdk92 said:
Sounds great.

How can you be sure this commands work on a 4.1/5.1 Mac Pro? Have you tried?

Yes.
[doublepost=1524418351][/doublepost]

joevt said:
I would like to see how SoftRaid handles that compared to Disk Utility because on my Hackintosh with 4 drives, SoftRaid was 49% better than Disk Utility.

joevt, did you find a solution to execute the setpci commands at boot?

itdk92 · Apr 22, 2018

ezylstra said:
Yes.
[doublepost=1524418351][/doublepost]

Have you tried on a 4.1/5.1?

ezylstra · Apr 22, 2018

itdk92 said:
Have you tried on a 4.1/5.1?

Yes.

I used the commands to downgrade my graphics card from PCIe v2 to PCIe v1 speed. If that works, going from PCIe v1 to PCIe v2 will work.

itdk92 · Apr 22, 2018

ezylstra said:
Yes.

I used the commands to downgrade my graphics card from PCIe v2 to PCIe v1 speed. If that works, going from PCIe v1 to PCIe v2 will work.

Don't want to be that guy, but can you provide proof? Does this work with GPUs, too?

ezylstra · Apr 22, 2018

This is the output from downgrading my GPU speed. I can do this back and forth all day. It is my GPU on a MacPro4,1->5,1.

thames-mac-pro:sbin neweric$ rootport=00:7
thames-mac-pro:sbin neweric$ linkstatus=$(sudo ./setpci -s $rootport CAP_EXP+12.w); echo '#
Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4
& 31))
#
Before: PCIe 2.0 x16
thames-mac-pro:sbin neweric$ speed=1
thames-mac-pro:sbin neweric$ sudo ./setpci -s $rootport CAP_EXP+30.w=$speed:F
thames-mac-pro:sbin neweric$ sudo ./setpci -s $rootport CAP_EXP+10.w=20:20
thames-mac-pro:sbin neweric$ linkstatus=$(sudo ./setpci -s $rootport CAP_EXP+12.w); echo '#
Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4
& 31))
#
Before: PCIe 1.0 x16
thames-mac-pro:sbin neweric$

itdk92 · Apr 22, 2018

ezylstra said:
This is the output from downgrading my GPU speed. I can do this back and forth all day. It is my GPU on a MacPro4,1->5,1.

thames-mac-pro:sbin neweric$ rootport=00:7
thames-mac-pro:sbin neweric$ linkstatus=$(sudo ./setpci -s $rootport CAP_EXP+12.w); echo '#
Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4
& 31))
#
Before: PCIe 2.0 x16
thames-mac-pro:sbin neweric$ speed=1
thames-mac-pro:sbin neweric$ sudo ./setpci -s $rootport CAP_EXP+30.w=$speed:F
thames-mac-pro:sbin neweric$ sudo ./setpci -s $rootport CAP_EXP+10.w=20:20
thames-mac-pro:sbin neweric$ linkstatus=$(sudo ./setpci -s $rootport CAP_EXP+12.w); echo '#
Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4
& 31))
#
Before: PCIe 1.0 x16
thames-mac-pro:sbin neweric$

What does the actual System Report say?

h9826790 · Apr 22, 2018

itdk92 said:
What does the actual System Report say?

Better to test the GPU link speed by CL!ng, system report not often wrong.

https://itunes.apple.com/hk/app/cl-ing/id1244392031?l=en&mt=12

ezylstra · Apr 22, 2018

System Report will always report the original speed. It cannot be relied on to report any changes.

You imply the output from setpci is lying. I have no good way to prove to you my GPU is running at v1 speed other than that output. I'm running the original graphics cards, so they provide a poor test. Better would be to use a graphics card that has problems and runs at v1 speeds.

If I do the same with a Lycom DT-120 with an sm961 that negotiates the wrong speed on my Xserve3,1, I can show you the performance difference before and after. In that case it is clear that the speed reported by setpci is correct. It reports PCIe v1 before running and gives 700 MB/sec range for reads and writes, and reports PCIe v2 and 1500 MB/sec after.

If you are still curious, I suggest you do the work, do the test, and post the results.

joevt · Apr 22, 2018

itdk92 said:
So you would generally recommend Amfeltec Squid Gen 3 for individual drives? (without any other modificaions)

How about NVMe RAIDs?

It depends on the speed of the NVMe drive. You should be able to decide from the benchmarks I gave and benchmarks for the drives you want to use.

itdk92 said:
How can you be sure this commands work on a 4.1/5.1 Mac Pro? Have you tried?

I don’t know how much Mac Pro 3.1 and Hack counts in verifying this.

The commands worked on two different computers: older and newer than the Mac Pro 4.1/5.1. There's no reason they shouldn't also work on the Mac Pro 4.1/5.1 because they all use standard PCIe registers.

ezylstra said:
joevt, did you find a solution to execute the setpci commands at boot?

I haven't looked. You could put the code in an EFI boot loader like rEFInd, or a kext, or a startup script. I guess a startup script would be the easiest way. An EFI boot loader would ensure the OS reports the correct speed. A kext might do the same if it runs early enough. I didn't test if the speed changes after sleep/wake. If it does, then the EFI boot loader method isn't good enough. A kext could handle that. I'm not sure if a startup script can be executed after wake? That would be something to look into (launchd, AppleScript, Automator, Scenario.app, Login items, etc.)

h9826790 · May 8, 2018

ezylstra said:
This is the output from downgrading my GPU speed. I can do this back and forth all day. It is my GPU on a MacPro4,1->5,1.

thames-mac-pro:sbin neweric$ rootport=00:7
thames-mac-pro:sbin neweric$ linkstatus=$(sudo ./setpci -s $rootport CAP_EXP+12.w); echo '#
Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4
& 31))
#
Before: PCIe 2.0 x16
thames-mac-pro:sbin neweric$ speed=1
thames-mac-pro:sbin neweric$ sudo ./setpci -s $rootport CAP_EXP+30.w=$speed:F
thames-mac-pro:sbin neweric$ sudo ./setpci -s $rootport CAP_EXP+10.w=20:20
thames-mac-pro:sbin neweric$ linkstatus=$(sudo ./setpci -s $rootport CAP_EXP+12.w); echo '#
Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4
& 31))
#
Before: PCIe 1.0 x16
thames-mac-pro:sbin neweric$

Hi I would like to run some tests as well. May I know how to find out the rootport?

e.g. Is my 1080Ti at root port 00:3?

joevt · May 8, 2018

h9826790 said:
Hi I would like to run some tests as well. May I know how to find out the rootport?

e.g. Is my 1080Ti at root port 00:3?

Yes.

The lspci tree output (-t) doesn't show the info for the root ports / bridges.

My pcitree.sh script outputs all devices in tree format and shows the root-ports / bridges on separate lines in the tree output. It has to be rewritten to work in Windows.

The following command lists all devices including root ports / bridges:

Code:

lspci.exe -nn

Get more info on the root port (bus 0, device 3) with this command:

Code:

lspci.exe -nnvv -s 00:3

You should see something like "Express Root Port (Slot+)" in one of the capabilities.

Use the following command to get info for the graphics card (bus 5, device 0, function 0):

Code:

lspci.exe -nnvv -s 05:0.0

LnkCap is the max speed and width
LnkSta is the current speed and width
LnkCtl2 is the target speed and width for a root port / upstream port

For the Nvidia graphics card, LnkSta is 2.5GT/s when not doing any rendering. It should go up to 5GT/s (in PCIe 2.0 slot) or 8GT/s (in PCIe 3.0 slot) when rendering begins. GPU-Z has a test button to start rendering (click the ? button)

The target speed of the root port should affect the speed of the graphics card endpoint.

h9826790 · May 8, 2018

joevt said:
Yes.

The lspci tree output (-t) doesn't show the info for the root ports / bridges.

My pcitree.sh script outputs all devices in tree format and shows the root-ports / bridges on separate lines in the tree output. It has to be rewritten to work in Windows.

The following command lists all devices including root ports / bridges:

Code:

lspci.exe -nn

Get more info on the root port (bus 0, device 3) with this command:

Code:

lspci.exe -nnvv -s 00:3

You should see something like "Express Root Port (Slot+)" in one of the capabilities.

Use the following command to get info for the graphics card (bus 5, device 0, function 0):

Code:

lspci.exe -nnvv -s 05:0.0

LnkCap is the max speed and width
LnkSta is the current speed and width
LnkCtl2 is the target speed and width for a root port / upstream port

For the Nvidia graphics card, LnkSta is 2.5GT/s when not doing any rendering. It should go up to 5GT/s (in PCIe 2.0 slot) or 8GT/s (in PCIe 3.0 slot) when rendering begins. GPU-Z has a test button to start rendering (click the ? button)

The target speed of the root port should affect the speed of the graphics card endpoint.

Thanks for the detailed explanation, will do the test and report back.

h9826790 · May 8, 2018

Quick report back.

I am 100% sure I get the correct root port because 00:3 is the only port that has LnkSta 2.5GT/s x16. Which must be my graphic card.

Before running

Code:

setpci -s 00:3 CAP_EXP+30.w=2:F

I got

Code:

                LnkCap: Port #3, Speed 5GT/s, Width x16, ASPM not supported
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch+ ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug- Surprise+
                        Slot #192, PowerLimit 0.000W; Interlock+ NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd Off, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible+
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB

And after apply the command, it become

Code:

                LnkCap: Port #3, Speed 5GT/s, Width x16, ASPM not supported
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch+ ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug- Surprise+
                        Slot #192, PowerLimit 0.000W; Interlock+ NoCompl-
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Off, PwrInd Off, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible+
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range BCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd+
                         AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                         AtomicOpsCtl: ReqEn- EgressBlck-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB

Therefore, the target link speed does changed. However, no matter how I stress the card (including using GPU-Z), the LnkSta won't change. Still stay at 2.5GT/s x16.

joevt · May 8, 2018

h9826790 said:
Quick report back.

I am 100% sure I get the correct root port because 00:3 is the only port that has LnkSta 2.5GT/s x16. Which must be my graphic card.

Before running

Code:

setpci -s 00:3 CAP_EXP+30.w=2:F

I got

Code:

LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-

And after apply the command, it become

Code:

LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-

Therefore, the target link speed does changed. However, no matter how I stress the card (including using GPU-Z), the LnkSta won't change. Still stay at 2.5GT/s x16.

After you set the target link speed, you need to start retrain with

Code:

setpci -s 00:3 CAP_EXP+10.w=20:20

If that doesn't work, post information about your motherboard.

h9826790 · May 9, 2018

joevt said:
After you set the target link speed, you need to start retrain with

Code:

setpci -s 00:3 CAP_EXP+10.w=20:20

If that doesn't work, post information about your motherboard.

Tested, no change, same as before, target link speed changed, but link status won't change (even GPU under stress). Anyway, not quite sure what you need to know about the motherboard. This is my Machine Profile.

zozomester · Jun 18, 2018

I have a problem that when Amfeltec squid 4 ssd raid0 is inside, then Windows 10 is frozen and constantly restarts.
Error message: Cashe Manager
- Slot 1: GTX 980
- Slot 2: Amfeltec Squid with High Sierra
- Slot 3: Hyperx Predator with Windows 10
- Slot 4: KT4004 USB 3

As I remove the Amfeltec card, the problem is over.
Thank!

handheldgames · Jun 18, 2018

ezylstra said:
System Report will always report the original speed. It cannot be relied on to report any changes.

You imply the output from setpci is lying. I have no good way to prove to you my GPU is running at v1 speed other than that output. I'm running the original graphics cards, so they provide a poor test. Better would be to use a graphics card that has problems and runs at v1 speeds.

If I do the same with a Lycom DT-120 with an sm961 that negotiates the wrong speed on my Xserve3,1, I can show you the performance difference before and after. In that case it is clear that the speed reported by setpci is correct. It reports PCIe v1 before running and gives 700 MB/sec range for reads and writes, and reports PCIe v2 and 1500 MB/sec after.

If you are still curious, I suggest you do the work, do the test, and post the results.

Thanks for sharing. Do you have a .sh file you can share? It would be great to use a modified version in a login script.

joevt · Jun 19, 2018

h9826790 said:
Tested, no change, same as before, target link speed changed, but link status won't change (even GPU under stress). Anyway, not quite sure what you need to know about the motherboard. This is my Machine Profile.

I thought you were using a PC because of the command line window screenshot. Do you have the same problem of not being able to change the graphics card speed in both Windows (BootCamp?) and macOS?

I did a couple tests with my Mac Pro 2008 and Nvidia 680 Mac edition. Below is my PCIe device tree containing the following:

Slot 1: PCI Express Port 5: GeForce GTX 680
Slot 2: PCI Express Port 1: Empty
Slot 3: Downstream Port E2: CalDigit FASTA-6GU3 Plus
Slot 4: Downstream Port E1: Sonnet Tempo SSD Pro Plus

I used my pcitree.sh script (attached) instead of "sudo lspci -tvnn" to get bridge information and the link speed/width of each device (max and current values).

Code:

sudo ~/amfeltec_tests/pcitree.sh
#=========================================================================================
-[0000:00]-\
            +-00.0                                                                                  # g1x4           0000:00:00.0 [8086:4003] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset Memory Controller Hub
            +-01.0-[01]----\                                                                        # g2x16 > g1x0   0000:00:01.0 [8086:4021] [0604] (rev 20) PCI bridge: Intel Corporation 5400 Chipset PCI Express Port 1
            +-05.0-[02]----\                                                                        # g2x16 > g1x16  0000:00:05.0 [8086:4025] [0604] (rev 20) PCI bridge: Intel Corporation 5400 Chipset PCI Express Port 5
            |               +-00.0                                                                  # g2x16 > g1x16  0000:02:00.0 [10de:1180] [0300] (rev a1) VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680]
            |               \-00.1                                                                  # g2x16 > g1x16  0000:02:00.1 [10de:0e0a] [0403] (rev a1) Audio device: NVIDIA Corporation GK104 HDMI Audio Controller
            +-09.0-[03-0e]-\                                                                        # g2x4 > g1x4    0000:00:09.0 [8086:4029] [0604] (rev 20) PCI bridge: Intel Corporation 5400 Chipset PCI Express Port 9
            |               +-00.0-[04-0d]-\                                                        # g1x8 > g1x4    0000:03:00.0 [8086:3500] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port
            |               |               +-00.0-[05-08]-\                                        # g1x4           0000:04:00.0 [8086:3510] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1
            |               |               |               \-00.0-[06-08]-\                        # g2x4 > g1x4    0000:05:00.0 [111d:8061] [0604] (rev 01) PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch
            |               |               |                               +-02.0-[07]----\        # g2x4 > g2x2    0000:06:02.0 [111d:8061] [0604] (rev 01) PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch
            |               |               |                               |               \-00.0  # g2x2           0000:07:00.0 [1b4b:9182] [0106] (rev 13) SATA controller: Marvell Technology Group Ltd. Device
            |               |               |                               \-04.0-[08]----\        # g2x4 > g2x2    0000:06:04.0 [111d:8061] [0604] (rev 01) PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch
            |               |               |                                               \-00.0  # g2x2           0000:08:00.0 [1b4b:9182] [0106] (rev 13) SATA controller: Marvell Technology Group Ltd. Device
            |               |               +-01.0-[09-0c]-\                                        # g1x4           0000:04:01.0 [8086:3514] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2
            |               |               |               \-00.0-[0a-0c]-\                        # g2x4 > g1x4    0000:09:00.0 [12d8:2308] [0604] PCI bridge: Pericom Semiconductor Device
            |               |               |                               +-01.0-[0b]----\        # g2x2           0000:0a:01.0 [12d8:2308] [0604] PCI bridge: Pericom Semiconductor Device
            |               |               |                               |               \-00.0  # g2x2           0000:0b:00.0 [1b21:1242] [0c03] USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
            |               |               |                               \-02.0-[0c]----\        # g2x2           0000:0a:02.0 [12d8:2308] [0604] PCI bridge: Pericom Semiconductor Device
            |               |               |                                               \-00.0  # g2x2           0000:0c:00.0 [1b21:0625] [0106] (rev 01) SATA controller: ASMedia Technology Inc. Device
            |               |               \-02.0-[0d]----\                                        # g1x4           0000:04:02.0 [8086:3518] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E3
            |               |                               +-00.0                                  # g1x4           0000:0d:00.0 [8086:1096] [0200] (rev 01) Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper)
            |               |                               \-00.1                                  # g1x4           0000:0d:00.1 [8086:1096] [0200] (rev 01) Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper)
            |               \-00.3-[0e]----\                                                        # g1x8 > g1x4    0000:03:00.3 [8086:350c] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge
            +-0f.0                                                                                  # g0x0           0000:00:0f.0 [8086:402f] [0880] (rev 20) System peripheral: Intel Corporation 5400 Chipset QuickData Technology Device
            +-10.0                                                                                  #                0000:00:10.0 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.1                                                                                  #                0000:00:10.1 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.2                                                                                  #                0000:00:10.2 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.3                                                                                  #                0000:00:10.3 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.4                                                                                  #                0000:00:10.4 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-11.0                                                                                  #                0000:00:11.0 [8086:4031] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset CE/SF Registers
            +-15.0                                                                                  #                0000:00:15.0 [8086:4035] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-15.1                                                                                  #                0000:00:15.1 [8086:4035] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-16.0                                                                                  #                0000:00:16.0 [8086:4036] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-16.1                                                                                  #                0000:00:16.1 [8086:4036] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-1b.0                                                                                  # g0x0           0000:00:1b.0 [8086:269a] [0403] (rev 09) Audio device: Intel Corporation 631xESB/632xESB High Definition Audio Controller
            +-1c.0-[0f]----\                                                                        # g1x1 > g1x0    0000:00:1c.0 [8086:2690] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1
            +-1c.1-[10]----\                                                                        # g1x1 > g1x0    0000:00:1c.1 [8086:2692] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 2
            +-1c.2-[11-12]-\                                                                        # g1x1           0000:00:1c.2 [8086:2694] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 3
            |               \-00.0-[12]----\                                                        # g1x1           0000:11:00.0 [104c:823e] [0604] PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express]
            |                               \-00.0                                                  #                0000:12:00.0 [104c:823f] [0c00] FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express]
            +-1c.3-[13]----\                                                                        # g1x1           0000:00:1c.3 [8086:2696] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 4
            |               \-00.0                                                                  # g1x1           0000:13:00.0 [14e4:4328] [0280] (rev 03) Network controller: Broadcom Limited BCM4321 802.11a/b/g/n
            +-1d.0                                                                                  #                0000:00:1d.0 [8086:2688] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1
            +-1d.1                                                                                  #                0000:00:1d.1 [8086:2689] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2
            +-1d.2                                                                                  #                0000:00:1d.2 [8086:268a] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3
            +-1d.3                                                                                  #                0000:00:1d.3 [8086:268b] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4
            +-1d.7                                                                                  #                0000:00:1d.7 [8086:268c] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller
            +-1e.0-[14]----\                                                                        #                0000:00:1e.0 [8086:244e] [0604] (rev d9) PCI bridge: Intel Corporation 82801 PCI Bridge
            +-1f.0                                                                                  #                0000:00:1f.0 [8086:2670] [0601] (rev 09) ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller
            +-1f.1                                                                                  #                0000:00:1f.1 [8086:269e] [0101] (rev 09) IDE interface: Intel Corporation 631xESB/632xESB IDE Controller
            +-1f.2                                                                                  #                0000:00:1f.2 [8086:2681] [0106] (rev 09) SATA controller: Intel Corporation 631xESB/632xESB SATA AHCI Controller
            \-1f.3                                                                                  #                0000:00:1f.3 [8086:269b] [0c05] (rev 09) SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller

I ran the following commands:

Code:

rootport=00:5
linkstatus=$(sudo setpci -s $rootport CAP_EXP+12.w); echo '#Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4 & 31))

speed=1
sudo setpci -s $rootport CAP_EXP+30.w=$speed:F
sudo setpci -s $rootport CAP_EXP+10.w=20:20
linkstatus=$(sudo setpci -s $rootport CAP_EXP+12.w); echo "#SetSpeed $speed: PCIe" $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4 & 31))

speed=2
sudo setpci -s $rootport CAP_EXP+30.w=$speed:F
sudo setpci -s $rootport CAP_EXP+10.w=20:20
linkstatus=$(sudo setpci -s $rootport CAP_EXP+12.w); echo "#SetSpeed $speed: PCIe" $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4 & 31))

Without a 3D graphics program running, the speed does not change:

Code:

#Before: PCIe 1.0 x16
#SetSpeed 1: PCIe 1.0 x16
#SetSpeed 2: PCIe 1.0 x16

When I execute the commands again while the Unigine Valley benchmark is running ( https://benchmark.unigine.com/valley ), I can change the speed:

Code:

#Before: PCIe 2.0 x16
#SetSpeed 1: PCIe 1.0 x16
#SetSpeed 2: PCIe 2.0 x16

h9826790 · Jun 19, 2018

joevt said:

I thought you were using a PC because of the command line window screenshot. Do you have the same problem of not being able to change the graphics card speed in both Windows (BootCamp?) and macOS?

I did a couple tests with my Mac Pro 2008 and Nvidia 680 Mac edition. Below is my PCIe device tree containing the following:

Slot 1: PCI Express Port 5: GeForce GTX 680
Slot 2: PCI Express Port 1: Empty
Slot 3: Downstream Port E2: CalDigit FASTA-6GU3 Plus
Slot 4: Downstream Port E1: Sonnet Tempo SSD Pro Plus

I used my pcitree.sh script (attached) instead of "sudo lspci -tvnn" to get bridge information and the link speed/width of each device (max and current values).

Code:

sudo ~/amfeltec_tests/pcitree.sh
#=========================================================================================
-[0000:00]-\
            +-00.0                                                                                  # g1x4           0000:00:00.0 [8086:4003] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset Memory Controller Hub
            +-01.0-[01]----\                                                                        # g2x16 > g1x0   0000:00:01.0 [8086:4021] [0604] (rev 20) PCI bridge: Intel Corporation 5400 Chipset PCI Express Port 1
            +-05.0-[02]----\                                                                        # g2x16 > g1x16  0000:00:05.0 [8086:4025] [0604] (rev 20) PCI bridge: Intel Corporation 5400 Chipset PCI Express Port 5
            |               +-00.0                                                                  # g2x16 > g1x16  0000:02:00.0 [10de:1180] [0300] (rev a1) VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680]
            |               \-00.1                                                                  # g2x16 > g1x16  0000:02:00.1 [10de:0e0a] [0403] (rev a1) Audio device: NVIDIA Corporation GK104 HDMI Audio Controller
            +-09.0-[03-0e]-\                                                                        # g2x4 > g1x4    0000:00:09.0 [8086:4029] [0604] (rev 20) PCI bridge: Intel Corporation 5400 Chipset PCI Express Port 9
            |               +-00.0-[04-0d]-\                                                        # g1x8 > g1x4    0000:03:00.0 [8086:3500] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port
            |               |               +-00.0-[05-08]-\                                        # g1x4           0000:04:00.0 [8086:3510] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1
            |               |               |               \-00.0-[06-08]-\                        # g2x4 > g1x4    0000:05:00.0 [111d:8061] [0604] (rev 01) PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch
            |               |               |                               +-02.0-[07]----\        # g2x4 > g2x2    0000:06:02.0 [111d:8061] [0604] (rev 01) PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch
            |               |               |                               |               \-00.0  # g2x2           0000:07:00.0 [1b4b:9182] [0106] (rev 13) SATA controller: Marvell Technology Group Ltd. Device
            |               |               |                               \-04.0-[08]----\        # g2x4 > g2x2    0000:06:04.0 [111d:8061] [0604] (rev 01) PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch
            |               |               |                                               \-00.0  # g2x2           0000:08:00.0 [1b4b:9182] [0106] (rev 13) SATA controller: Marvell Technology Group Ltd. Device
            |               |               +-01.0-[09-0c]-\                                        # g1x4           0000:04:01.0 [8086:3514] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2
            |               |               |               \-00.0-[0a-0c]-\                        # g2x4 > g1x4    0000:09:00.0 [12d8:2308] [0604] PCI bridge: Pericom Semiconductor Device
            |               |               |                               +-01.0-[0b]----\        # g2x2           0000:0a:01.0 [12d8:2308] [0604] PCI bridge: Pericom Semiconductor Device
            |               |               |                               |               \-00.0  # g2x2           0000:0b:00.0 [1b21:1242] [0c03] USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
            |               |               |                               \-02.0-[0c]----\        # g2x2           0000:0a:02.0 [12d8:2308] [0604] PCI bridge: Pericom Semiconductor Device
            |               |               |                                               \-00.0  # g2x2           0000:0c:00.0 [1b21:0625] [0106] (rev 01) SATA controller: ASMedia Technology Inc. Device
            |               |               \-02.0-[0d]----\                                        # g1x4           0000:04:02.0 [8086:3518] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E3
            |               |                               +-00.0                                  # g1x4           0000:0d:00.0 [8086:1096] [0200] (rev 01) Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper)
            |               |                               \-00.1                                  # g1x4           0000:0d:00.1 [8086:1096] [0200] (rev 01) Ethernet controller: Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper)
            |               \-00.3-[0e]----\                                                        # g1x8 > g1x4    0000:03:00.3 [8086:350c] [0604] (rev 01) PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge
            +-0f.0                                                                                  # g0x0           0000:00:0f.0 [8086:402f] [0880] (rev 20) System peripheral: Intel Corporation 5400 Chipset QuickData Technology Device
            +-10.0                                                                                  #                0000:00:10.0 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.1                                                                                  #                0000:00:10.1 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.2                                                                                  #                0000:00:10.2 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.3                                                                                  #                0000:00:10.3 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-10.4                                                                                  #                0000:00:10.4 [8086:4030] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FSB Registers
            +-11.0                                                                                  #                0000:00:11.0 [8086:4031] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset CE/SF Registers
            +-15.0                                                                                  #                0000:00:15.0 [8086:4035] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-15.1                                                                                  #                0000:00:15.1 [8086:4035] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-16.0                                                                                  #                0000:00:16.0 [8086:4036] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-16.1                                                                                  #                0000:00:16.1 [8086:4036] [0600] (rev 20) Host bridge: Intel Corporation 5400 Chipset FBD Registers
            +-1b.0                                                                                  # g0x0           0000:00:1b.0 [8086:269a] [0403] (rev 09) Audio device: Intel Corporation 631xESB/632xESB High Definition Audio Controller
            +-1c.0-[0f]----\                                                                        # g1x1 > g1x0    0000:00:1c.0 [8086:2690] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 1
            +-1c.1-[10]----\                                                                        # g1x1 > g1x0    0000:00:1c.1 [8086:2692] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 2
            +-1c.2-[11-12]-\                                                                        # g1x1           0000:00:1c.2 [8086:2694] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 3
            |               \-00.0-[12]----\                                                        # g1x1           0000:11:00.0 [104c:823e] [0604] PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express]
            |                               \-00.0                                                  #                0000:12:00.0 [104c:823f] [0c00] FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express]
            +-1c.3-[13]----\                                                                        # g1x1           0000:00:1c.3 [8086:2696] [0604] (rev 09) PCI bridge: Intel Corporation 631xESB/632xESB/3100 Chipset PCI Express Root Port 4
            |               \-00.0                                                                  # g1x1           0000:13:00.0 [14e4:4328] [0280] (rev 03) Network controller: Broadcom Limited BCM4321 802.11a/b/g/n
            +-1d.0                                                                                  #                0000:00:1d.0 [8086:2688] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1
            +-1d.1                                                                                  #                0000:00:1d.1 [8086:2689] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2
            +-1d.2                                                                                  #                0000:00:1d.2 [8086:268a] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3
            +-1d.3                                                                                  #                0000:00:1d.3 [8086:268b] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4
            +-1d.7                                                                                  #                0000:00:1d.7 [8086:268c] [0c03] (rev 09) USB controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller
            +-1e.0-[14]----\                                                                        #                0000:00:1e.0 [8086:244e] [0604] (rev d9) PCI bridge: Intel Corporation 82801 PCI Bridge
            +-1f.0                                                                                  #                0000:00:1f.0 [8086:2670] [0601] (rev 09) ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller
            +-1f.1                                                                                  #                0000:00:1f.1 [8086:269e] [0101] (rev 09) IDE interface: Intel Corporation 631xESB/632xESB IDE Controller
            +-1f.2                                                                                  #                0000:00:1f.2 [8086:2681] [0106] (rev 09) SATA controller: Intel Corporation 631xESB/632xESB SATA AHCI Controller
            \-1f.3                                                                                  #                0000:00:1f.3 [8086:269b] [0c05] (rev 09) SMBus: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus Controller

I ran the following commands:

Code:

rootport=00:5
linkstatus=$(sudo setpci -s $rootport CAP_EXP+12.w); echo '#Before: PCIe' $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4 & 31))

speed=1
sudo setpci -s $rootport CAP_EXP+30.w=$speed:F
sudo setpci -s $rootport CAP_EXP+10.w=20:20
linkstatus=$(sudo setpci -s $rootport CAP_EXP+12.w); echo "#SetSpeed $speed: PCIe" $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4 & 31))

speed=2
sudo setpci -s $rootport CAP_EXP+30.w=$speed:F
sudo setpci -s $rootport CAP_EXP+10.w=20:20
linkstatus=$(sudo setpci -s $rootport CAP_EXP+12.w); echo "#SetSpeed $speed: PCIe" $(( 0x$linkstatus & 15)).0 x$(( 0x$linkstatus >> 4 & 31))

Without a 3D graphics program running, the speed does not change:

Code:

#Before: PCIe 1.0 x16
#SetSpeed 1: PCIe 1.0 x16
#SetSpeed 2: PCIe 1.0 x16

When I execute the commands again while the Unigine Valley benchmark is running ( https://benchmark.unigine.com/valley ), I can change the speed:

Code:

#Before: PCIe 2.0 x16
#SetSpeed 1: PCIe 1.0 x16
#SetSpeed 2: PCIe 2.0 x16

I am using a cMP (as per my signature). Since web driver automatically enable PCIe 2.0 in macOS. Therefore, I only need to try to enable PCIe 2.0 in Windows. Bootcamp driver package is installed, but that's a natively installed Windows, not via bootcamp assistant.

joevt · Jun 19, 2018

h9826790 said:
I am using a cMP (as per my signature). Since web driver automatically enable PCIe 2.0 in macOS. Therefore, I only need to try to enable PCIe 2.0 in Windows. Bootcamp driver package is installed, but that's a natively installed Windows, not via bootcamp assistant.

So macOS properly switches between PCIe 1.0 and PCIe 2.0 for your Nvidia card depending on if a 3D app is running?

Whatever stops the Nvidia card from switching from PCIe 1.0 to PCIe 2.0 when no 3D app is running in macOS might be similar/related to why the Nvidia in Windows on the Mac doesn't switch from PCIe 1.0 to PCIe 2.0 even when a 3D app is running?

Do the people at macvidcards (netkas et al) know how their Mac BIOS enables PCIe 2.0 (Nvidia specific registers?) or is it just something in the MacBIOS that does it?

h9826790 · Jun 19, 2018

joevt said:
So macOS properly switches between PCIe 1.0 and PCIe 2.0 for your Nvidia card depending on if a 3D app is running?

Whatever stops the Nvidia card from switching from PCIe 1.0 to PCIe 2.0 when no 3D app is running in macOS might be similar/related to why the Nvidia in Windows on the Mac doesn't switch from PCIe 1.0 to PCIe 2.0 even when a 3D app is running?

Do the people at macvidcards (netkas et al) know how their Mac BIOS enables PCIe 2.0 (Nvidia specific registers?) or is it just something in the MacBIOS that does it?

Yes, my GPU can switch between PCIe 1.0 and PCIe 2.0 depends on demand in MacOS.

I consulted Nvidia, however it seems they can't understand that the issue only exist in Windows but not macOS. Therefore, can't be hardware limitation.

AFAIK, MVC know the exact switch in VBIOS to enable PCIe 2.0 in Windows. I still can't quite sure if VBIOS is a hard limit for the card to negotiate at PCIe 2.0 in Windows (because there has no such limitation in macOS).

Amfeltec x16 PCIe with 4 SSDs: 5900+ MB/s

macrumors 6502a

Attachments

macrumors newbie

macrumors 65816

macrumors 6502a

macrumors member

macrumors 6502a

macrumors member

macrumors 6502a

macrumors member

macrumors 6502a

macrumors P6

macrumors member

macrumors 604

macrumors P6

macrumors 604

macrumors P6

macrumors P6

macrumors 604

macrumors P6

macrumors 6502

macrumors 68000

macrumors 604

Attachments

macrumors P6

macrumors 604

macrumors P6

Our Staff