(Theory) M1 Chipset simply not able to support PCIe via TB3 / TB3 Audio interfaces.

[Hessel89]

Like we knew, eGPU is not supported at the moment with M1, but what's less spoken about, is the fact that there is still no support for Thunderbolt 3 based Audio Interfaces.
Just like eGPU's, these audio interfaces utilise Thunderbolt 3's ability of having a direct connection to the CPU, via PCIe. Utilising TB3 this way enables those devices to run with extremely low latency.
Since no vendor has a working driver for the M1 yet, even 8 months after Apple announced the switch, I'm starting to think this is because the M1 chipset is inherently flawed when it comes to Thunderbolt 3.
Rather than offering a direct connection to the CPU via PCIe, data has to go through an USB 4 translation layer first, eliminating the actual benefits of Thunderbolt 3 for such situations, if not completely disabling it.

This is just a theory, I'm curious what you guys think.

Sidenote: it's not meant to bash the M1 Macs. I've been running an M1 Mac Mini with just 8GBs of ram and frankly the performance blew away. I just think it's good to talk about what it can and can't do at the moment.

Speaking for myself though: Unfortunately I had to return my M1 Mini because my studio is based around TB3, including an expensive TB3 audio interface. And I'm not the only one. Colleagues with TB3 Interfaces from different vendors are also still waiting.. Most USB interfaces on the other hand seem to work fine.

 

[Ritsuka]

Nope. If it doesn't work, it's simple because there is no driver. Did you try to contact your audio interface manufacture and ask if they have plans to support M1?

 

[Hessel89]

Nope. If it doesn't work, it's simple because there is no driver. Did you try to contact your audio interface manufacture and ask if they have plans to support M1?

Of course I have. My vendor has actually announced support and made the driver a priority since the day Apple announced the M1 but up till today still nothing. I believe them because I know colleagues have too with their vendors.
I posted this thread this discuss the theory I laid out, not to complain about the drivers still not being there, (cos frankly I don't care, I'm still on Intel) So if you have any opinion about that, feel free to chime in.

 

[leman]

Of course it supports PCIe. But audio interface manufacturers are literally the worst when it comes to updating the drivers. Which in case of M1 basically means rewriting the drivers using the new userland API if I understand correctly.

 

[Hessel89]

But audio interface manufacturers are literally the worst when it comes to updating the drivers.

Tell me about.. but in this case there does seem to be a correlation with eGPU support, which also require realtime PCIe streams. Could just be that either USB 4 or the M1 chipset simply doesn't allow for this? Other ways of utilising TB3 work fine, like connecting 5K screens, but those are using different protocols, like DisplayPort, which is also supported by USB C.

 

[leman]

Tell me about.. but in this case there does seem to be a correlation with eGPU support, which also require realtime PCIe streams. Could just be that either USB 4 or the M1 chipset simply doesn't allow for this? Other ways of utilising TB3 work fine, like connecting 5K screens, but those are using different protocols, like DisplayPort, which is also supported by USB C.

M1's PCI-e over Thunderbolt works perfectly fine, there are working drivers that utilize it (for example Promise Pegasus drivers).

As far as eGPUs go, that's a completely different story since they also require deep OS support. You need to be able to register the eGPU in the system and have your application communicate with it using the standard APIs. Personally, I am sure that lack of eGPU support is a deliberate policy decision from Apple to ensure streamlined developer experience (eGPUs violate the performance guarantees offered by Apple Silicon and introduce complex edge cases). In fact, I am fairly sure that you can write an eGPU driver for M1 using the existing APIs... but it will be rather useless since Apple does not allow pluggable Metal drivers.

 

[Deccr]

I’ve plugged in a couple of NVMe TB3 drives and they both show up as PCI-e drives when looking at About This Mac->Storage.

 

[thedocbwarren]

Interface works, eGPU doesn't only due to driver support.

 

[CMMChris]

And here's a solid proof for the M1's PCIe support over Thunderbolt...

 

[synicalx1]

Be very keen to keep an eye on this situation. I'm almost certainly going to get a M1 and eGPU support, even if it's a few months down the line, would make it an absolutely perfect setup for me (and a great way to re-purpose my 2060...)

 

[thedocbwarren]

Be very keen to keep an eye on this situation. I'm almost certainly going to get a M1 and eGPU support, even if it's a few months down the line, would make it an absolutely perfect setup for me (and a great way to re-purpose my 2060...)

It's possible. I wonder how it works given how interwoven the GPU is with the SoC.

 

[crazy dave]

Like we knew, eGPU is not supported at the moment with M1, but what's less spoken about, is the fact that there is still no support for Thunderbolt 3 based Audio Interfaces.
Just like eGPU's, these audio interfaces utilise Thunderbolt 3's ability of having a direct connection to the CPU, via PCIe. Utilising TB3 this way enables those devices to run with extremely low latency.
Since no vendor has a working driver for the M1 yet, even 8 months after Apple announced the switch, I'm starting to think this is because the M1 chipset is inherently flawed when it comes to Thunderbolt 3.
Rather than offering a direct connection to the CPU via PCIe, data has to go through an USB 4 translation layer first, eliminating the actual benefits of Thunderbolt 3 for such situations, if not completely disabling it.

This is just a theory, I'm curious what you guys think.

Sidenote: it's not meant to bash the M1 Macs. I've been running an M1 Mac Mini with just 8GBs of ram and frankly the performance blew away. I just think it's good to talk about what it can and can't do at the moment.

Speaking for myself though: Unfortunately I had to return my M1 Mini because my studio is based around TB3, including an expensive TB3 audio interface. And I'm not the only one. Colleagues with TB3 Interfaces from different vendors are also still waiting.. Most USB interfaces on the other hand seem to work fine.

Corellium claim that their Linux port for the M1 can talk to eGPUs using TB3. Unsure how far they’ve gotten with actually running off of it, but the TB hardware supports it just fine. Others have shown TB3 pcie network cards working on the Mac side. And others have mentioned other examples. So, hardware seems to be fine. Restrictions on more than 2 monitors and no eGPU seem to be due to lack of drivers/restrictions in software.

 

[jdb8167]

Corellium claim that their Linux port for the M1 can talk to eGPUs using TB3. Unsure how far they’ve gotten with actually running off of it, but the TB hardware supports it just fine. Others have shown TB3 pcie network cards working on the Mac side. And others have mentioned other examples. So, hardware seems to be fine. Restrictions on more than 2 monitors and no eGPU seem to be due to lack of drivers/restrictions in software.

To be useful though it would also have to integrate into Apple's Metal graphics stack. From what I understand this isn't open so any GPU driver would need to be written to support Metal with Apple's cooperation.

 

[leman]

Be very keen to keep an eye on this situation. I'm almost certainly going to get a M1 and eGPU support, even if it's a few months down the line, would make it an absolutely perfect setup for me (and a great way to re-purpose my 2060...)

Don’t get your hopes up. I don’t think it’s likely that Apple will allow official eGPU integration because it violates the Apple Silicon GPU programming model.

 

[jdb8167]

Don’t get your hopes up. I don’t think it’s likely that Apple will allow official eGPU integration because it violates the Apple Silicon GPU programming model.

Have you seen any spelunking on whether the low-level Metal frameworks that were used by the Intel Macs are still there? I don't know what the framework would be called. I might go looking to see if there is a way to get information on the private framework.

Without access to the framework(s), it isn't like that even an unofficial eGPU driver will be forthcoming.

 

[ArPe]

Don’t get your hopes up. I don’t think it’s likely that Apple will allow official eGPU integration because it violates the Apple Silicon GPU programming model.

Yes for eGPU to work they would have to disable UMA in real time so that the eGPU’s VRAM takes over. Very tricky to do and can be unstable.

 

[leman]

Have you seen any spelunking on whether the low-level Metal frameworks that were used by the Intel Macs are still there? I don't know what the framework would be called. I might go looking to see if there is a way to get information on the private framework.

Without access to the framework(s), it isn't like that even an unofficial eGPU driver will be forthcoming.

Haven't seen anything like that, no, but Apple never really exposed its GPU driver interface anyway. I don't think it would be difficult for them to add third-party driver support, but again, my point is that they wouldn't want to do it. Even if there is still an interface (and Appel GPU driver is not directly backed into the OS, which is also possible), not much you can do if Apple refuses to load your kernel extension.

Anyway, I'm fairly confident that a GPU driver can still be written, it just won't be able to interact with Metal or the base OS services. But if Nvidia wants to offer a CUDA driver for eGPUs, they could probably do it. Of course, that most certainly would be a fool's errand.

Yes for eGPU to work they would have to disable UMA in real time so that the eGPU’s VRAM takes over. Very tricky to do and can be unstable.

What you say don'ts make much sense. UMA has no relation to eGPUs, the driver simply needs to take care of data synchronization over the PCI-e bus. You don't need to "disable UMA", nor does the VRAM "take over", that's just a different pool of memory that you have to manage separately. Intel Macs have UMA too and they work just fine with eGPUs and dGPUs.

 

[jdb8167]

Haven't seen anything like that, no, but Apple never really exposed its GPU driver interface anyway. I don't think it would be difficult for them to add third-party driver support, but again, my point is that they wouldn't want to do it. Even if there is still an interface (and Appel GPU driver is not directly backed into the OS, which is also possible), not much you can do if Apple refuses to load your kernel extension.

They exposed their GPU driver interface to AMD and at one point to Nvidia. I was wondering if that framework was still lingering in Apple Silicon Big Sur. If it is, I can't figure out where it is. There are a bunch of candidates but I have no idea where to start.

Certainly a commercial GPU driver would be out of the question without Apple's cooperation but if someone wanted to hack a GPU driver and the Metal integration framework was still there, unchanged, and working, it becomes somewhat feasible. You can load any kernel extension you want if you are willing to reduce security.

Anyway, I'm fairly confident that a GPU driver can still be written, it just won't be able to interact with Metal or the base OS services. But if Nvidia wants to offer a CUDA driver for eGPUs, they could probably do it. Of course, that most certainly would be a fool's errand.

Sure. Since the PCIe connection is visible from the System Profile, it is almost certainly accessible but again, you are probably going to have to write a kernel extension and turn off security to load it. If someone was doing research with CUDA they may be willing to do that but then Nvidia would still have to write the driver. I doubt they can make any money doing that so it isn't going to happen.

 

[crazy dave]

To be useful though it would also have to integrate into Apple's Metal graphics stack. From what I understand this isn't open so any GPU driver would need to be written to support Metal with Apple's cooperation.

oh it wouldn’t be useful on macOS (not as things currently stand), this was for Linux running on the M1. I was just using it as an example that the TB3 hardware appears to be fully functional - that it isn’t the sticking point for why eGPUs don’t work in the macOS for the M1.

 

[leman]

They exposed their GPU driver interface to AMD and at one point to Nvidia. I was wondering if that framework was still lingering in Apple Silicon Big Sur. If it is, I can't figure out where it is. There are a bunch of candidates but I have no idea where to start.

Certainly a commercial GPU driver would be out of the question without Apple's cooperation but if someone wanted to hack a GPU driver and the Metal integration framework was still there, unchanged, and working, it becomes somewhat feasible. You can load any kernel extension you want if you are willing to reduce security.

Might be possible, but it really depends on how driver discovery is implemented. For what we know, Apple could just hard-code the driver list somewhere in the kernel, so you'd need to patch that list to load a new driver.

Not to mention that kernel extensions are on borrowed time. You can still load them now, but you won't be able to do that in a year or two. Apple has confirmed it in no unclear terms that they are removing support for kernel extensions altogether.

Sure. Since the PCIe connection is visible from the System Profile, it is almost certainly accessible but again, you are probably going to have to write a kernel extension and turn off security to load it. If someone was doing research with CUDA they may be willing to do that but then Nvidia would still have to write the driver. I doubt they can make any money doing that so it isn't going to happen.

You could probably even use DriverKit to do that... but what about performance?

I bet Alyssa Rosenzweig (https://rosenzweig.io) knows all there is to know about these things btw, you could ask her on twitter...

 

[crazy dave]

Anyway, I'm fairly confident that a GPU driver can still be written, it just won't be able to interact with Metal or the base OS services. But if Nvidia wants to offer a CUDA driver for eGPUs, they could probably do it. Of course, that most certainly would be a fool's errand.

I agree that it feels unlikely but, especially as a Mac user who also does CUDA programming, it would be nice. Probably not going to happen. However maybe if Linux works on the M-series chips ...

 

[jdb8167]

Might be possible, but it really depends on how driver discovery is implemented. For what we know, Apple could just hard-code the driver list somewhere in the kernel, so you'd need to patch that list to load a new driver.

Not to mention that kernel extensions are on borrowed time. You can still load them now, but you won't be able to do that in a year or two. Apple has confirmed it in no unclear terms that they are removing support for kernel extensions altogether.



You could probably even use DriverKit to do that... but what about performance?

I bet Alyssa Rosenzweig (https://rosenzweig.io) knows all there is to know about these things btw, you could ask her on twitter...

I doubt that Apple will remove the ability to load kernel extensions. They are just not going to sign them which makes them commercially non-viable since you would have to reduce security to load them at all. As long as you can experiment with unsigned kernels you will have the ability to load extensions and since Apple just added that feature to the M1 secure boot, I don't expect it to go away.

 

[ArPe]

What you say don'ts make much sense. UMA has no relation to eGPUs, the driver simply needs to take care of data synchronization over the PCI-e bus. You don't need to "disable UMA", nor does the VRAM "take over", that's just a different pool of memory that you have to manage separately. Intel Macs have UMA too and they work just fine with eGPUs and dGPUs.

If you plug in an eGPU everything graphical that is loaded into UMA has to be removed from system memory and transferred to the GPU’s VRAM in real time without error. It can be done but it can be unstable. Maybe that’s why there is no driver yet.

Stop comparing Intel’s iGPU implementation to Apple’s. The memory management is not the same. One has a fixed reserve and the other is dynamic.

 

[leman]

If you plug in an eGPU everything graphical that is loaded into UMA has to be removed from system memory and transferred to the GPU’s VRAM in real time without error. It can be done but it can be unstable. Maybe that’s why there is no driver yet.

It might surprise you, but that's exactly what GPU driver have been doing for decades and do today, with great success. Yes, GPU data synchronization is a complicated and channeling topic. But it's a solved problem.

Stop comparing Intel’s iGPU implementation to Apple’s.

Stop assuming that there is some weird magic to Apple's implementation that prevents it from using PCI-e for data transfers just like any other computer. On the memory organization level, there is little difference between an Intel SoC or an Apple SoC — it's just that the later is faster and smarter.

 

[leman]

I doubt that Apple will remove the ability to load kernel extensions. They are just not going to sign them which makes them commercially non-viable since you would have to reduce security to load them at all. As long as you can experiment with unsigned kernels you will have the ability to load extensions and since Apple just added that feature to the M1 secure boot, I don't expect it to go away.

I am not speculating though. Apple has deprecated kernel extensions in 2019 and has confirmed that support for kernel extensions will be completely removed in future versions of macOS. They been mentioning this in no unclear terms during the WWDC and also on the support forums.