MP 7,1 ESXi on Mac Pro 7,1

[mikeboss]

I was really hoping the on-board NICs would be similiar to those of the 2018 Mac Mini

welcome to the macrumors forums, william!

in fact, the NICs in the 2019 Mac Pro are similar to the one in Mac mini 2018. according to the ifixit teardown Apple uses 2x aQuantia AQtion AQC107-B1-C multi-gigabit ethernet controllers. maybe Apple uses a newer revision of the controller. I'd expect that the latest AQC107 drivers from Aquantia will support the NICs of Mac Pro 2019...

Mac Pro 2019 Teardown

The Mac is back and more Pro than ever, throwing away the cylindrical “trash can” design in favor of something that resembles a computer. Its...

www.ifixit.com

 

[lamw]

welcome to the macrumors forums, william!

Thanks!

in fact, the NICs in the 2019 Mac Pro are similar to the one in Mac mini 2018. according to the ifixit teardown Apple uses 2x aQuantia AQtion AQC107-B1-C multi-gigabit ethernet controllers. maybe Apple uses a newer revision of the controller. I'd expect that the latest AQC107 drivers from Aquantia will support the NICs of Mac Pro 2019...

Ah, that's good to know. I knew there was a teardown but was on mobile and was going to look at the system later but figure a support bundle could be useful. If its AQC-based, then I think its worth giving the current driver a try @CNDCodewarrior. I know they've had a few updates since publishing.

I honestly thought they were going to stick with basic 1GbE broadcom but it looks like Apple is slowly switching over to all AQC/Marvell NICs now

 

[Mago]

Ok, but would be possible to concurrently have linux and macOS, as with T2 equiped Mac even can't have a multi boot Linux install, maybe t2 won't allow access to the imMP pcie SSD having to install macOS in a Hackintosh-like setup along Linux

 

[SecuritySteve]

Ok, but would be possible to concurrently have linux and macOS, as with T2 equiped Mac even can't have a multi boot Linux install, maybe t2 won't allow access to the imMP pcie SSD having to install macOS in a Hackintosh-like setup along Linux

The grammar here is pretty rough, so forgive me if I'm answering the wrong question.

All Apple has to do is white list the Linux / ESXi OS with the T2 to get multi-boot to work. That may be in the works, but we have no way of knowing for sure.

 

[SecuritySteve]

Thanks!



Ah, that's good to know. I knew there was a teardown but was on mobile and was going to look at the system later but figure a support bundle could be useful. If its AQC-based, then I think its worth giving the current driver a try @CNDCodewarrior. I know they've had a few updates since publishing.

I honestly thought they were going to stick with basic 1GbE broadcom but it looks like Apple is slowly switching over to all AQC/Marvell NICs now

Hello William, any update on your end? I've been checking Virtuallyghetto every day while we wait for the rack mounted Mac Pro to be available.

 

[lamw]

Hello William, any update on your end? I've been checking Virtuallyghetto every day while we wait for the rack mounted Mac Pro to be available.

Not yet Steve. I'm still waiting for @CNDCodewarrior to confirm his test which may be another week. Working with him offline for now. If someone can provide support bundle, that could help. I have also reached out to Aquantia folks to see if this particular device is supported w/their current driver.

 

[CNDCodewarrior]

I've started a new job and it's pretty far north, living the camp life. I'll be home this upcoming Friday and should have some time to play with it on Saturday. I've got a intel nic that I'll install in the pro so the unmodified installer should be able to finish then I can get @lamw the support bundle.

Sorry for the delay guys but lugging the beast on the plane wasn't an option

 

[CNDCodewarrior]

Ok so got ESXi 6.7 update 3 installed on the pro using an intel nic.

Install went normally, it saw one of my internal PCIE cards for the install and it saw the 2 cards on my highpoint raid card (Raid not supported)

Here's the storage adaptors list

Local storage

The intel nic's

And just general info

So as expected no internal apple storage is visible, the built in nic's don't have drivers as expected. But if you have a storage card that is supported by vmware you will be able to install it and you would be able to install any NIC's you needed to get communications working.

Generated the support file and sent it to @lamw

Hope this helps folks!

 

[SecuritySteve]

Ok so got ESXi 6.7 update 3 installed on the pro using an intel nic.

View attachment 888060

Install went normally, it saw one of my internal PCIE cards for the install and it saw the 2 cards on my highpoint raid card (Raid not supported)

View attachment 888061

Here's the storage adaptors list

View attachment 888062

Local storage

View attachment 888063

The intel nic's

View attachment 888064

And just general info

View attachment 888065

So as expected no internal apple storage is visible, the built in nic's don't have drivers as expected. But if you have a storage card that is supported by vmware you will be able to install it and you would be able to install any NIC's you needed to get communications working.

Generated the support file and sent it to @lamw

Hope this helps folks!

Brilliant, thank you so much. What intel nic did you use?

 

[CNDCodewarrior]

Just a basic 2 port 1gig card, sorry don't have the exact model as it's still in the pro and the baby's sleeping right now so can't take it apart.


Type: Ethernet Controller
Bus: PCI
Vendor ID: 0x8086
Device ID: 0x105e
Subsystem Vendor ID: 0x8086
Subsystem ID: 0x115e
Revision ID: 0x0006
Link Width: x4
BSD name: en12
Kext name: AppleIntel8254XEthernet.kext
Location: /System/Library/Extensions/IONetworkingFamily.kext/Contents/PlugIns/AppleIntel8254XEthernet.kext

Version: 3.1.5

Drivers are loaded for it in OS X too.

But anything that has drivers for ESXi will work with it.

 

[lamw]

Thanks to @CNDCodewarrior and couple other customers, we found that the built-in NICs aren't automatically recognized by ESXi, however once you incorporate the Aquantia ESXi Driver (which you can search on my blog "Aquantia 10GbE ESXi Driver for Apple 2018 Mac Mini", then ESXi will pickup those NICs. I've shared these finding on my blog as several other customers have asked and will use that to maintain latest news/updates.

 

[SecuritySteve]

So with the SMC blocking, is it currently not possible to boot a macOS VM on ESXi on the Mac Pro 7,1 - even with the Aquantia ESXi Driver and a PCIe storage card for ESXi?

 

[lamw]

That's correct. AQC driver only enables the NIC (as thats purely a driver issue which we've had since Mac Mini 2018). The PCIe storage or any PCIe device for that matter isn't affected by the T2 (with exception of Thunderbolt connected devices).

 

[SecuritySteve]

Ouch. So unless Apple actually reaches out to VMware (or VMware reaches out to Apple / throws them money?) macOS VMs on ESXi might be a thing of the past?

 

[Bakafish]

Ouch. So unless Apple actually reaches out to VMware (or VMware reaches out to Apple / throws them money?) macOS VMs on ESXi might be a thing of the past?

There are 2 SATA and one USB3 ports internally that can be used for appropriate storage to boot ESX, not being able to leverage the T2 controlled storage is not really a big hurdle for anyone interested in low level virtualization. Best practices is always for external network attached storage for VM's/data and the built in storage is neither very cost effective nor high performance (compared to PCI based SSD) and has other well documented shortcomings. Booting from a standardized internal USB key would serve most non-commercial applications of ESX for this platform. Existing SATA or PCI based storage could be leveraged where that isn't considered robust enough, so not sure why you are expressing this much angst.

As a side note, the demand for ESX support on Apple's platform is miniscule and extremely fringe case at best. No one is buying Apple boxes for general ESX hosts, the only benefit is for OS X virtual hosting (since Fusion and Parallels have good client solutions.)

Having said that, I'd be interested in hearing any other need for ESX specifically on the Mac.

 

[lamw]

the demand for ESX support on Apple's platform is miniscule and extremely fringe case at best. No one is buying Apple boxes for general ESX hosts, the only benefit is for OS X virtual hosting (since Fusion and Parallels have good client solutions.)

I'll share my Reddit reply since this is a common miss-conception on why customers are interested in ESXi on Apple HW. Its certainly "smaller" chunk of the overall x86 market from a VMware standpoint, but its certainly not a fringe use case nor miniscule from the customers I've spoken with over the past 7+years having been involved in this space from VMware

If you wish to participate in Apple's eco-system of developing/building MacOS/iOS applications, per Apple's EULA, it must be done on Apple HW platform. For basic development, your standard Apple laptop is sufficient but once you need to build, test, integrate, scale, CI/CD, etc. then laptops/desktops with Fusion on your desk isn't going to cut it for most Enterprises. This is the target audience/use case when it comes to running ESXi on Apple HW and you can either do it yourself like many of our customers do OR you can consume hosted service from MacStadium (who also runs vSphere on their bare-metal offering that they provide to customers). Past customers who've shared their success story and use of MacStadium and vSphere has included Uber, Dropbox, Capital One, Box, Travis CI to just name a few and there's many many others that can't share.

Several years back, I wrote a blog series sharing some of our customer uses cases for MacOS on vSphere if you're interested in some of the specific use case (which are still very applicable in 2020). Simply search "Community stories of VMware & Apple OS X in Production" and you'll see find 10-part series. I would link but don't want to get banned again

 

[deconstruct60]

That's correct. AQC driver only enables the NIC (as thats purely a driver issue which we've had since Mac Mini 2018). The PCIe storage or any PCIe device for that matter isn't affected by the T2 (with exception of Thunderbolt connected devices).

followed by
[automerge]1579709764[/automerge]

Ouch. So unless Apple actually reaches out to VMware (or VMware reaches out to Apple / throws them money?) macOS VMs on ESXi might be a thing of the past?

IF the macOS VM image is on the PCIe storage device that is not affected by the T2. It is this still blocked?

Once ESXi is up and running the T2 isn't particularly in play. At least from a drive perspective. Is there is "leak through" or native access to the secure enclave can see a problem there. But once the EFI firmware is running on the copy provided by the T2 then it isn't particularly in the "boot loop" anymore.

 

[Bakafish]

I'll share my Reddit reply since this is a common miss-conception on why customers are interested in ESXi on Apple HW. Its certainly "smaller" chunk of the overall x86 market from a VMware standpoint, but its certainly not a fringe use case nor miniscule from the customers I've spoken with over the past 7+years having been involved in this space from VMware

If you wish to participate in Apple's eco-system of developing/building MacOS/iOS applications, per Apple's EULA, it must be done on Apple HW platform. For basic development, your standard Apple laptop is sufficient but once you need to build, test, integrate, scale, CI/CD, etc. then laptops/desktops with Fusion on your desk isn't going to cut it for most Enterprises. This is the target audience/use case when it comes to running ESXi on Apple HW and you can either do it yourself like many of our customers do OR you can consume hosted service from MacStadium (who also runs vSphere on their bare-metal offering that they provide to customers). Past customers who've shared their success story and use of MacStadium and vSphere has included Uber, Dropbox, Capital One, Box, Travis CI to just name a few and there's many many others that can't share.

Several years back, I wrote a blog series sharing some of our customer uses cases for MacOS on vSphere if you're interested in some of the specific use case (which are still very applicable in 2020). Simply search "Community stories of VMware & Apple OS X in Production" and you'll see find 10-part series. I would link but don't want to get banned again

For context I should disclose I work for Dell/EMC (former VMware Engineer), have a VCAP and support Enterprise virtualization in Asia-Pacific, so when I say "fringe" and you say "smaller" we are looking at it from completely different perspectives. The number of production Mac based deployments is a miniscule fraction (likely 0.01% of total ESX market) compared to commercial server manufacturers. I'm not trying to diminish the value and use cases you attribute, but it isn't even really a 'market' for VMware, the only reason it works at all is that there is so little effort needed to make it work and that there is still a bit of love of Apple HW inside the VMware engineering department (they always provided me the latest Mac hardware (PowerPC at that time) when I worked there) and they have slipped it in so to speak.

The main point I was trying to make is that "MacStadium" (and I assume your firm) have the capability to workaround the minor limitations in ESX deployment on Mac outlined above, and likely do not care that the T2 storage isn't accessible as I'm sure they use some form of network based enterprise storage (as I said is best practice.) I would contend that anyone who was investing in one or more Pro's with the intent of creating a "continuous integration" cluster could likewise easily deal with the T2 being inaccessible. I gave 3 solutions (in order of cost & reliability), USB3, SATA, PCI based SSD (including potentially a RAID which some may feel is appropriate for locally based data.)

(Up to) 8TB of mediocre speed, non redundant, local storage that has no recoverability is a joke. T2 storage is garbage from an enterprise (ESX) perspective. Apple and VMware would be doing a disservice to support it as someone out there would inevitably use it for data and pay the iron price. Unbundled ESX network drivers is not a hardship to a professional, there is nothing I see here that requires Apple or VMware to intervene or throw money at. I feel it is a vanishingly small market from their perspective, and they are not going to waste engineering resources to make it easier for non-professionals to misapply it as a solution.

 

[AidenShaw]

(Up to) 8TB of mediocre speed, non redundant, local storage that has no recoverability is a joke. T2 storage is garbage from an enterprise (ESX) perspective.

I have a small vSphere cluster. Four 2U dual-socket ProLiant servers, 3 TiB RAM total. Quad 10 Gbps Ethernet on each. A pair of 208v 5000 KvA APC sine-wave UPS systems to handle brief power hits.

The ESXi systems use 32 GB SD cards for the ESXi system drive - they are essentially diskless. (It's very common for enterprise commodity servers to have an SD or microSD slot on the mobo for exactly this use case.) ESXi uses about a GB on each card, the other 31 GB is over-provisioning - although ESXi is basically in-memory, there are a few logfile flushes per hour that write to the SD card.

Four 24TB VMFS 5 filesystems on redundant 16 Gbps FibreChannel. (Dual FC HBAs on every system, dual controllers on the SAN RAID controller, dual cables to each storage shelf, dual-ported disks, each array is RAID-60 with hot spares.) Everything is set up so that under normal use load is balanced between the redundant paths - so we effectively have 32 Gbps to storage.

The cluster is over-sized so that only three hosts are really needed. For maintenance, we vMotion all of the VMs off the system to be worked on - and no user even realizes that we've shut a server down. We're a research and development lab, so we don't worry about HA (high availability).

And I'd expect that Bakafish would agree that this is a small vSphere setup.

 

[chrfr]

...The main point I was trying to make is that "MacStadium" (and I assume your firm) have the capability to workaround the minor limitations in ESX deployment on Mac outlined above...

Lamw works for VMware.

 

[tsialex]

Finally some good news about ESXi support for MM8,1 and MP7,1:

Update on ESXi on Apple Mac Mini 2018 & Mac Pro 2019

Although there has not been any news in some time regarding the support for ESXi on the latest Apple Mac Mini 2018 and the recently released Apple Mac Pro 2019, there has definitely been work happe…

www.virtuallyghetto.com

 

[SecuritySteve]

Finally some good news about ESXi support for MM8,1 and MP7,1:

Update on ESXi on Apple Mac Mini 2018 & Mac Pro 2019

Although there has not been any news in some time regarding the support for ESXi on the latest Apple Mac Mini 2018 and the recently released Apple Mac Pro 2019, there has definitely been work happe…

www.virtuallyghetto.com

It's alive! Huzzah!

 

[lamw]

Looks like you guys beat me to sharing the good news, I had a ton of emails to send this morning letting our customers know the good news and dropping a note here was on my to-do list

 

[AlexBoid]

Hey guys, does the on-board NICs should work with ESXi 7.0 +? The Aquantia drivers are there, and NICs are detected by ESXi (7.0 u2d), no problems here, but sadly no network activity from adapter, it isn't able to get DHCP address or ping a gateway in case of setting static IP. No outgoing packets from NICs whatsoever. Does someone have a different behaviour with those?