Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

thenewperson

macrumors 6502a
Mar 27, 2011
953
856
It's not something we could know, and given they just got PCIe 4 in their M1 Macs it could be a while. Maybe 2 years? As for TB, I'm guessing the next version will use it, though I'm not sure how things work there now given it's (seemingly) new role as 'USB4 but non-optional'.
 

dmccloud

macrumors 68030
Sep 7, 2009
2,990
1,727
Anchorage, AK
Given that PCIe 4 support is just now being rolled out (including for AMD and Intel-based systems), it will be a while before anyone starts supporting PCIe 5 in any significant manner.
 
  • Like
Reactions: bousozoku

leman

macrumors Core
Oct 14, 2008
19,302
19,284
Apple Silicon has no need for PCIe internally, so it’s only relevant to external connectors, meaning Thunderbolt. Current Thunderbolt3 is still based on PCIe 3.0 though. I suppose once the USB consortium decides to integrate newer versions of PCIe, Apple’s implementation will follow. It might take a while.

Why do you ask anyway?
 

Krevnik

macrumors 601
Sep 8, 2003
4,100
1,309
Where did we get PCIe 4 in M1? Maybe you are confusing it with USB4?

Apple did slip it into their little bullet point slide of SoC features:

M1_Features_575px.jpg
 

joevt

Contributor
Jun 21, 2012
6,689
4,086
Thanks for this, I never noticed! I wonder where exactly it is used...
PCIe is only used for WLAN and Bluetooth and Thunderbolt.

Looking at the ioreg of an M1 MacBook Air, the PCIe root port for WLAN and Bluetooth devices says it's capable of 16 GT/s x1 (PCIe 4.0) but the status says it's running at only 2.5 GT/s x1 (PCIe 1.0) which is strange because the devices (wlan and bluetooth) that are attached to the root port have capability and status of 5.0 GT/s x1 (PCIe 2.0).

So given the above, while the M1 Macs are capable of PCIe 4.0, it doesn't look look like it's actually used. It may be that PCIe 4.0 is used since the PCIe capability and status values in ioreg are usually not current. To get the current values, you need something like pciutils. pciutils needs to be updated to work with M1 Macs though.

The two PCIe root ports for the Thunderbolt ports are weird because they have capability 2.5 GT/s x16 and status 2.5 GT/s x1. A Thunderbolt device connected to one of the root ports has the correct upstream capability of 8 GT/s x4 (PCIe 3.0). Older Thunderbolt devices may have upstream capability of 2.5 GT/s x4 when the upstream is a Thunderbolt port. The status for the upstream is 2.5 GT/s x4 which is expected when the upstream is a Thunderbolt port (even though the Thunderbolt connection does ≈24 Gbps of PCIe data). As with all Thunderbolt controllers, the downstream devices of the Thunderbolt controller have capability and status of 2.5 GT/s x4 even though the controller is processing up to ≈24 Gbps of PCIe data. The downstream PCIe devices in a Thunderbolt controller are integrated inside the Thunderbolt controller so it doesn't need to be real PCIe and the link rate and link width don't need to be real. It's the same for Thunderbolt integrated inside the Ice Lake and Tiger Lake CPUs. The external PCIe devices in a Thunderbolt peripheral (such as a USB, FireWire, or Ethernet controller) will have real PCIe so the PCIe capability and status values will be real for the Thunderbolt downstream bridges that are connected to those real PCIe devices.

One thing interesting about the M1 Macs is that the three PCIe root ports are separate from each other. They each have the full range of PCIe bus numbers 0x00-0xFF available to them. All three root ports have address 0:0:0 - no other Mac ever had more than one PCIe device with the same PCIe bus:device:function numbers. The Thunderbolt ports only have bus numbers 0x01..0x80 reserved, but I don't see why that couldn't be 0x01..0xFF since 0x80 is an 8 bit number.

Code:
root port for wlan and bluetooth:
    | |   |     |   "pcidebug" = "0:0:0(1:1)"
    | |   |       |     "pcidebug" = "1:0:0"
    | |   |         |   "pcidebug" = "1:0:1"

root port for thunderbolt port 1:
    | |   |     |   "pcidebug" = "0:0:0(1:128)"
    | |   |         |   "pcidebug" = "1:0:0(2:4)"
    | |   |           | |   "pcidebug" = "2:2:0(3:3)"
    | |   |           |     |   "pcidebug" = "3:0:0"
    | |   |             |   "pcidebug" = "2:4:0(4:4)"

root port for thunderbolt port 2:
    | |   |     |   "pcidebug" = "0:0:0(1:128)"
 

guzhogi

macrumors 68040
Original poster
Aug 31, 2003
3,751
1,849
Wherever my feet take me…
Apple Silicon has no need for PCIe internally, so it’s only relevant to external connectors, meaning Thunderbolt. Current Thunderbolt3 is still based on PCIe 3.0 though. I suppose once the USB consortium decides to integrate newer versions of PCIe, Apple’s implementation will follow. It might take a while.

Why do you ask anyway?
Don't the internal SSDs use PCIe to some extent? Honest question.
 

Krevnik

macrumors 601
Sep 8, 2003
4,100
1,309
Yes, SSD is NVMe based. Don‘t be confused by macOS displaying a fancy „Apple Fabric“ name. It still is PCIe.

The controller isn't even NVMe though. It's NVMe-like, but it isn't an NVMe drive. There's no PCIe bus, and the driver, despite having NVMe in the name, isn't NVMe in the sense you are thinking of.

This has been true for A-series chips, and the T2 which use an embedded SSD controller. The only chips off the SoC are the NAND chips themselves.

PCIe is only used for WLAN and Bluetooth and Thunderbolt.

At least on the Mini, it looks like Ethernet and USB-A ports also hang off a PCIe bus there too. Not terribly surprising.
 

CMMChris

macrumors 6502a
Oct 28, 2019
850
793
Germany (Bavaria)
I know that the controller is on the SoC. But that doesn’t change anything about the fact The SSD being NVMe and PCIe.... Source: Linux.

Even Apple themselves called the connection „NVMe Controller“ initially. It was changed to Apple Fabric through and update at some point.
 

Krevnik

macrumors 601
Sep 8, 2003
4,100
1,309
I know that the controller is on the SoC. But that doesn’t change anything about the fact The SSD being NVMe and PCIe.... Source: Linux.

Even Apple themselves called the connection „NVMe Controller“ initially. It was changed to Apple Fabric through and update at some point.

The T2's SSD controller is accessed over PCIe (T2 gets a set of lanes to the Intel CPU), but that's no longer true with the M1. Since we're now in a single SoC, the M1 uses a proprietary I/O bus like the iPhone/iPad does. That I/O bus is what the T2 likely used internally to communicate with the embedded SSD controller.

EDIT: My source comes from the ioreg output from my M1 Mini, there's no PCIe controller associated with the "NVMe" driver being used, but rather an ARM I/O controller.
 

CMMChris

macrumors 6502a
Oct 28, 2019
850
793
Germany (Bavaria)
Dude, Corellium developed Linux support for the internal SSD. Go and try it out, dig through the system or check the source code. You then will realize that all the SSD stuff is indeed NVMe and the connection is PCIe based.

Even Apple themselves call their controller an NVMe controller
Bildschirmfoto 2021-03-19 um 19.13.58.png

Now guess what NVMe is... a PCIe extension to drive storage devices over PCIe with generic drivers.
Edit: Forgot to say that just because the device tree and IORegistry doesn't list the connection as PCI doesn't mean it technically isn't PCIe based.
 
Last edited:
  • Like
Reactions: jdb8167

Krevnik

macrumors 601
Sep 8, 2003
4,100
1,309
Dude, Corellium developed Linux support for the internal SSD. Go and try it out, dig through the system or check the source code. You then will realize that all the SSD stuff is indeed NVMe and the connection is PCIe based.

Even Apple themselves call their controller an NVMe controller

Now guess what NVMe is... a PCIe extension to drive storage devices over PCIe with generic drivers.

You really need to stop doubling down on this, you tell me to look at the code, and I find more evidence that you are wrong. :)

Corellium uses a virtual PCIe controller to make the M1 SSD work in Linux. Relevant commits:

Add virtual PCIe controller: https://github.com/corellium/linux-m1/commit/e501cce95dfcc65adb8c213f15efa9944899e940
Add ANS2 Controller support for SSD: https://github.com/corellium/linux-m1/commit/08f0fcce76579a58333016ec804b6f2d27e42296
Fixing device tree for the ANS2 controller: https://github.com/corellium/linux-m1/commit/8666075241e825fd6f7cec82ab7d7f2bcb626d7f

Corellium's comment about the SSD support when it asks someone configuring a kernel build:

Say Y here if you want to enable support for the NVMe-like ANS
controller
found in the Apple M1 SoC. It will be visible as a PCI
host bridge.

It's as I've said before. It is NVMe-like. It is not on PCIe however, and has some quirks that make it slightly different than standard NVMe.
 
  • Like
Reactions: TechnoMonk

Krevnik

macrumors 601
Sep 8, 2003
4,100
1,309
It doesn't matter if it's modified and got a different name glued on top of it. It still is PCIe / NVMe at its heart.

There's no PCIe involved at all. Rather, the SSD controller itself is directly accessible via MMIO like you'd expect with on-die secondary processors. This is not "PCIe based".

To further quote Correlium here, like you suggested I do:

This coprocessor does not, unfortunately expose a PCIe like interface. But it does have a MMIO range that is a mostly normal NVMe BAR, with a few quirks (handled in NVMe code).
 
  • Like
Reactions: TechnoMonk

guzhogi

macrumors 68040
Original poster
Aug 31, 2003
3,751
1,849
Wherever my feet take me…
Now guess what NVMe is... a PCIe extension to drive storage devices over PCIe with generic drivers.
Edit: Forgot to say that just because the device tree and IORegistry doesn't list the connection as PCI doesn't mean it technically isn't PCIe based.
For some reason, this reminds me of the first Macs with 802.11n wireless. Apple originally touted them as 802.11 a/b/g, then released a software patch to enable 'n' speeds. Due to some law (Sarbanes/Oxley I want to say?), Apple had to charge for the patch. There were a few arguments about it. The hardware was technically 'n', but Apple didn't advertise it as such. Crazy times. https://www.macrumors.com/2007/01/30/apple-quietly-releases-802-11n-enabler/

Anyways, regardless of what kind of connection the SSDs have, I'm curious to see when Thunderbolt will update its PCIe stuff. Apple seems to have killed off eGPU support with the M1 (correct me if I'm wrong), but I'm sure there are still other devices which can use the extra bandwidth of PCIe 4/5+.
 

Krevnik

macrumors 601
Sep 8, 2003
4,100
1,309
Anyways, regardless of what kind of connection the SSDs have, I'm curious to see when Thunderbolt will update its PCIe stuff. Apple seems to have killed off eGPU support with the M1 (correct me if I'm wrong), but I'm sure there are still other devices which can use the extra bandwidth of PCIe 4/5+.

Apple could support eGPU on the M1, but the fact that the GPU drivers needed haven't been ported to ARM suggests they aren't terribly interested. But I can't really blame them with all the stability issues I encountered with my eGPU setup, or the boot screen pain that Apple spent a whole OS release cycle seeing if they could improve with mixed results. And it wound up being incredibly niche, so it feels like a failed experiment like the "two lower power GPU" approach they took in the 2013 Mac Pro.

The catch is that without a version of TB that brings it up to PCIe 4/5, Apple's not got a lot of options there without essentially forking the standard. Thunderbolt is niche enough that it's probably not a great move.
 
  • Like
Reactions: CMMChris

joevt

Contributor
Jun 21, 2012
6,689
4,086
At least on the Mini, it looks like Ethernet and USB-A ports also hang off a PCIe bus there too. Not terribly surprising.
You're right. I was looking at the ioreg for a MacBookAir10,1. The Macmini9,1 has additional root ports for USB (xhci) and Ethernet (lan-1gb).

A Macmini9,1 has these pci devices (bus:device.function in hex; g4=16 GT/s, g3=8 GT/s, g2=5 GT/s, g1=2.5 GT/s):
Code:
├┬00:00.0-[03]      # g4x1 > g1x1
│├─03:00.0          # g2x1          wlan
│└─03:00.1          # g2x1          bluetooth
├┬00:01.0-[02]      # g4x1 > g2x1
│└─02:00.0          # g2x1          xhci
└┬00:02.0-[01]      # g4x1 > g1x1
 └─01:00.0          # g1x1          lan-1gb

└┬00:00.0-[01-80]   # g1x16 > g1x1  thunderbolt 0

└┬00:00.0-[01-80]   # g1x16 > g1x1  thunderbolt 1
Like the MacBookAir10,1, there appears to be 3 separate segments. The first segment is provided by a IOPCIBridge:AppleEmbeddedPCIE:AppleT810xPCIe:AppleT8103PCIe. The Thunderbolt segments are each provided by a IOPCIBridge:AppleEmbeddedPCIE:AppleT8103PCIeC.

Also like the MacBookAir10,1, nothing in the Macmini9,1 appears to use PCIe 4.0. The first three root ports may appear to support PCIe 4.0, but the devices connected to the root ports do not.

Interestingly, the USB controller appears to be based on the Fresco Logic FL1100. The driver has this class hierarchy:
AppleUSBHostController:AppleUSBXHCI:AppleUSBXHCIPCI:AppleUSBXHCIFL1100:AppleEmbeddedUSBXHCIFL1100:AppleT8101USBXHCIFL1100:AppleT8103USBXHCIFL1100

NVMe does not necessarily require a PCIe bus to function.
Right. For example, there exists USB to NVMe enclosures using a bridge chip. The computer talks to the USB device using the USB mass storage interface and the USB to NVMe bridge chip talks to the NVMe device using PCIe.

As CMMChris and Krevnik explained, the connection between the CPU and the SSD is like that. Even if the SSD is NVMe, The M1 Mac driver is not using a normal PCIe interface to talk to it.
Here's the class hierarchy:
Code:
M1 Mac:        IONVMeController:AppleNVMeController:AppleEmbeddedNVMeController:AppleANS2NVMeController:AppleANS2CGNVMeController:AppleANS3NVMeController (provider is RTBuddyService)
Mac mini 2018: IONVMeController:AppleNVMeController:AppleANS2Controller (provider is IOPCIDevice)
normal NVMe:   IONVMeController (provider is IOPCIDevice)
In the M1 Mac case, the provider of the IONVMeController is not a IOPCIDevice. It's called RTBuddy which has this class hierarchy IOSlaveProcessor:RTBuddy:RTBuddyV2. There's one for each of these roles:
ACIO0, ACIO1, AOP, PMP, SIO, ANS2, SMC, DCP, DCPEXT, GFX
 
  • Like
Reactions: TechnoMonk

mcnallym

macrumors 65816
Oct 28, 2008
1,182
911
Digging up an old thread - While Apple Silicon may not use PCIe internally, should we expect their SSD speeds to keep pace with PCIe5, especially now that they're shipping?
Providing they use faster nand and multiple chips then don’t see why Apple Storage shouldn’t keep pace.

however you will only see faster storage as you get the new SoC launches.
 

Pressure

macrumors 603
May 30, 2006
5,077
1,408
Denmark
Digging up an old thread - While Apple Silicon may not use PCIe internally, should we expect their SSD speeds to keep pace with PCIe5, especially now that they're shipping?
PCIe5 sequential read and write speeds doesn't really matter. You will still be bottlenecked by the much more important random read and write speeds, which isn't faster than PCIe3 or PCIe4 SSDs to begin with.

I can't find a single use case for 12,000MB/s sequential read and writes.
 

StellarVixen

macrumors 68040
Mar 1, 2018
3,177
5,640
Somewhere between 0 and 1
PCIe5 sequential read and write speeds doesn't really matter. You will still be bottlenecked by the much more important random read and write speeds, which isn't faster than PCIe3 or PCIe4 SSDs to begin with.

I can't find a single use case for 12,000MB/s sequential read and writes.
Neither can I. I thought it might serve as direct access memory, without need for RAM in some examples, but latency still isn’t nowhere near.

But hey, pumping up the numbers is one aspect of the "progress".
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.