MP 1,1-5,1 modifying MacPro 5,1 firmware to change memory operation from 'interleaved' to NUMA

[broly]

hello all,

longtime user/reader, first time poster.

i am posting to ask others about the memory configuration settings encoded in the 4,1/5,1 firmware.

i have learned that the firmware has limited the system to a single NUMA node (instead of two), meaning the dual processors' memory is operating in 'interleaved' mode.

i am aware that in specific use cases, enabling NUMA can hurt performance by 20-30%, however i have an application that uses intel's parallel studio (an eigensolver, to be specific) and it cannot use more than a single CPU.

is it at all possible to make modifications to the firmware and flash it? has anyone tried anything like this?

i really want to experiemnt with the memory operating in full NUMA mode, and i am told it's strictly a firmware limitation that is preventing it from doing so.

any information would be great.

thanks/

 

[tsialex]

While everything is possible, the cost of developing and implementing such alteration on how the platform works with dual CPUs would never be worth of it. Why won't you just buy a single CPU tray that will cost you less than $80 with shipping on eBay, and be done with it, since it's an application that don't work with multiple CPUs?

Mac Pro uses NUMA to share RAM between both Xeon processors, changing it would limit you to the same RAM constrains that single CPU trays have anyway.

 

[startergo]

While everything is possible, the cost of developing and implementing such alteration on how the platform works with dual CPUs would never be worth of it. Why won't you just buy a single CPU tray that will cost you less than $80 with shipping on eBay, and be done with it, since it's an application that don't work with multiple CPUs?

Mac Pro uses NUMA to share RAM between both Xeon processors, changing it would limit you to the same RAM constrains that single CPU trays have anyway.

How will this do?

sudo nvram boot-args="cpus=1"

Limits the number of active processors in the system to the set level. Apple's developer tools have an option to enable or disable some of the CPUs on the system, but you can do this manually by running this command and specifying the number of CPU cores to use. In some cases, such as with laptop systems, this might help preserve power, though is likely not useful for much else unless you are testing and programming.

Or should it be:
boot-args="cpus=12" for 12 cores out of 24 cores for instance.

 

[tsialex]

How will this do?

Or should it be:
boot-args="cpus=12" for 12 cores out of 24 cores for instance.

If I remember correctly, I'm too feverish to read the documentation to confirm if I'm right, this works at the kernel task scheduler and not at the hardware level and won't change the memory NUMA access.

A way to cheaply test if a single CPU tray will bring performance is to remove CPU B and run the dual CPU tray as a single CPU. SMC will make all the fans work full RPM, but it's a simple test to do for checking if it's worth to pursue a single tray.

 

[broly]

hi,

well, the application still does use the *entire* single CPU, just not both of them.

so a single tray won't really solve the problem because i am getting performance like that already.

i thought the setting of interleaved mode is a simple edit to the firmware? surely these kinds of schemes are onboard and not entirely on the firmware (as this would take a bit of space, which i assume is valuable).

in short, i was hoping you would know if the firmware stores information about the memory operation mode and if it can be modified.

i want to see the performance of both CPUS in NUMA mode (even though i'm aware there are tradeoffs).

 

[tsialex]

hi,

well, the application still does use the *entire* single CPU, just not both of them.

so a single tray won't really solve the problem because i am getting performance like that already.

i thought the setting of interleaved mode is a simple edit to the firmware? surely these kinds of schemes are onboard and not entirely on the firmware (as this would take a bit of space, which i assume is valuable).

in short, i was hoping you would know if the firmware stores information about the memory operation mode and if it can be modified.

i want to see the performance of both CPUS in NUMA mode (even though i'm aware there are tradeoffs).

I still stand by assessment, the cost of any changes that will need to be done/tested, even if someone does this for the intellectual challenge, won't be worth. This is not adding something like the NVMe and other tweaks that we did in the past, but changing the way the EFI firmware initialises the CPUs and the memory controllers. You will need someone with serious RE background and good knowledge of TIANO implementation, Tylersburg platform and IDA.

Go for a newer Mac Pro, like a MP6,1 12-core, where you don't have to worry about NUMA penalties.

Anyway, good luck whatever you choose to do.

 

[broly]

i appreciate your recommendations.

i wouldn't worry about the skills part. while i may not be versed in the specific areas you're speaking of, i do not think it will be too difficult to learn them. but i understand what you are saying.

isn't there any resource you could point me to that discusses the general aspect of this firmware and how it operates? i thought this forum would be the best place to ask someone like yourself, who has selflessly kept users (like myself) apprised of the updates.

i didn't even know (until one week ago, when i saw your master post) that the MP 5,1 had firmware updates. i was on something well before HIgh sierra.i think it was 0059 or something really old, so obviously i am grateful for all of the work you have done. and i do not intend to disrespect you by asking more questions that seem to disregard your recommendation.

just hoping there are some resources out there that can discuss how this firmware operates and what it does.

regarding getting a newer mac pro. i honestly was going to get one later this year, and was quite excited about it. however with:

• the deprecatoin of the IOKit driver family (i mean it's still there on high sierra, it's just annoying seeing xcode give me 100s of warnings)
• the abolishment of the waitqueue API (after 10.4)
• the excruciating and seemingly non-sensical difficulties in linking system calls (like os_unfair_lock_lock, or pthreads) to driver kexts (even when Xcode reports successful compilation and the right header files are included)

suggests that i may actually leave OS X much sooner than i originally thought.

the IOKit family is extremely cool, and it's just typical modern apple to destroy what was probably the most promising desktop OS ever created.

i was hoping to port the Asus Xonar driver family to OS X (see github[dot]com/i3roly/CMI8788. i can actually get it to talk to the card no problem. i am just fed up with having difficulty linking kernel-level libraries), but the kind of changes they've made make me wonder if they even care about where mac was supposed to thrive.

don't get me wrong it's very encouraging to see they've released a new PCI-E-based device and obviously have to support it for some time, but when you look underneath it's just so underwhelming. APFS is garbage and forcing users to adopt it is just typical apple.

 

[flowrider]

I'm too feverish to read the documentation to confirm if I'm right,

Jeez man - TURN OFF your Mac and GO to SLEEP. You need the rest. We can get along with out you for awhile. Chill!!!!

Lou

 

[AidenShaw]

Go for a newer Mac Pro, like a MP6,1 12-core, where you don't have to worry about NUMA penalties.

Or, better yet, don't worry about NUMA.

The Intel systems don't have huge performance differences between "near" and "far" memory - especially since the big caches mean that cache hits are the same speed for both "near" and "far".

 

[tsialex]

i appreciate your recommendations.

i wouldn't worry about the skills part. while i may not be versed in the specific areas you're speaking of, i do not think it will be too difficult to learn them. but i understand what you are saying.

isn't there any resource you could point me to that discusses the general aspect of this firmware and how it operates? i thought this forum would be the best place to ask someone like yourself, who has selflessly kept users (like myself) apprised of the updates.

i didn't even know (until one week ago, when i saw your master post) that the MP 5,1 had firmware updates. i was on something well before HIgh sierra.i think it was 0059 or something really old, so obviously i am grateful for all of the work you have done. and i do not intend to disrespect you by asking more questions that seem to disregard your recommendation.

just hoping there are some resources out there that can discuss how this firmware operates and what it does.

regarding getting a newer mac pro. i honestly was going to get one later this year, and was quite excited about it. however with:

• the deprecatoin of the IOKit driver family (i mean it's still there on high sierra, it's just annoying seeing xcode give me 100s of warnings)
• the abolishment of the waitqueue API (after 10.4)
• the excruciating and seemingly non-sensical difficulties in linking system calls (like os_unfair_lock_lock, or pthreads) to driver kexts (even when Xcode reports successful compilation and the right header files are included)

suggests that i may actually leave OS X much sooner than i originally thought.

the IOKit family is extremely cool, and it's just typical modern apple to destroy what was probably the most promising desktop OS ever created.

i was hoping to port the Asus Xonar driver family to OS X (see github[dot]com/i3roly/CMI8788. i can actually get it to talk to the card no problem. i am just fed up with having difficulty linking kernel-level libraries), but the kind of changes they've made make me wonder if they even care about where mac was supposed to thrive.

don't get me wrong it's very encouraging to see they've released a new PCI-E-based device and obviously have to support it for some time, but when you look underneath it's just so underwhelming. APFS is garbage and forcing users to adopt it is just typical apple.

Apple EFI 1.10 is not exactly a fork, but a parallel development of what would later become the TIANO EDK and EDKII. Since Apple implementation is not open source, you should start with the EDK and TIANOCORE documentations since a lot of things are identical or similar. Tylersburg and 5520 Intel white papers too.

NUMA Linux code/papers will be useful too to see how the implementation is done at the OS level, Alan Cox and friends code back in the day was very interesting btw.