MP 1,1-5,1 MacPro4,1 hangs booting any macOS except other OS's

[MacNB2]

Sad day. One of my MacPro's all of a sudden does not complete to boot but hangs.
Logic Board failure ???
I am using MartinLo's OpenCore setup.
I have several drives with Mountain Lion, El Capitan, Yosemite, High Sierra and Mojave.

Boot starts, progress bar starts and soon slows down. After 15 minutes...yes 15 minutes...still the same.

Next I booted Mojave natively without OpenCore...same...hangs.
I did NVRAM rest five times followed by SMC rest...to no avail.
Next I booted Verbose natively to check for errors and noticed :



Seems to be hanging due to FireWire: busy timeout[2], (60s): 'IOFireWireLocalNode'

Next I ran Apple Hardware Diagnostics by holding D at boot chime and it ran without hanging.
I ran the extensive test and after 25 minutes it found nothing wrong :



Not sure if AHD actually fully tests each peripherals.
Next I booted Linux (eOS) via OpenCore and it boots fine and runs great.
Thinking it might be an NVRAM issue, I booted GRML linux (off USB) to use the flashrom to dump the BootROM as suggested here by @Macschrauber.
It dumped fine and used their rom utility to check the dump.
There's nothing wrong with NVRAM.

I then searched here but found no other similar issues.
Today I saw someone had the same problem with their Mac mini here with Firewire (and MacPro's has the same Firewire chip).
They deleted the IOFireWireFamily.kext.
But OpenCore has a neat feature to disable kexts

So I configured OpenCore to block the IOFireWireFamily.kext like this:


Now...all the macOS's boot perfectly.
But I have lost firewire ports (that I was using for transferring videos from DV camcoders).

The other issue is I can no longer boot macOS natively but only via OpenCore.

Anyone know any idea why all the macOS Firewire drivers hang this way if there's a h/w fault ?
Anyway to test the firewire H/W further and is it fixable ?
Or am I toast and need a replacement logic board ?

 

[prefuse07]

From what you've said, I have a feeling your bootrom is corrupted -- Have you had it rebuilt yet?

Also, have you tried looking at the diagnostic LEDs?

They can tell you many things, and can help you diagnose the status of your lobo

 

[Macschrauber]

The test is easy, as you have a backup of your bootrom.

NEVER DO THIS IF YOU HAVE NO BOOTROM BACKUP:

flash the generic MP51.fd firmware file extracted from a 10.14.6 full Installer for test sake.

The Mac Pro has no serial and identity after it but for a test boot this could be ok.

Flash it back after this with grml or Dosdudes RomTool. My dumper does not support flashing from a Mac Pro with no serial number. Maybe I will change that as underlaying tasks changed.

 

[bookemdano]

I wonder if your firewire controller got some ESD or something from the camcorder--or did you not have that connected recently?

If your firewire controller is fried, it looks like you could add one via PCIe card - see here: https://forums.macrumors.com/threads/firewire-pci-expansion-card.2235816/

 

[MacNB2]

From what you've said, I have a feeling your bootrom is corrupted -- Have you had it rebuilt yet?

Also, have you tried looking at the diagnostic LEDs?

They can tell you many things, and can help you diagnose the status of your lobo

Thanks. I checked the bootrom with the dump utility it tells me the VSS1 & VSS2 are healthy. No I have not had it rebuilt.
Nor upgraded to 144.0.0.0 just crossed flashed to 5,1. The system was working fine.

The Diagnostics LEDs look OK after booting:

 

[MacNB2]

The test is easy, as you have a backup of your bootrom.

NEVER DO THIS IF YOU HAVE NO BOOTROM BACKUP:

flash the generic MP51.fd firmware file extracted from a 10.14.6 full Installer for test sake.

The Mac Pro has no serial and identity after it but for a test boot this could be ok.

Flash it back after this with grml or Dosdudes RomTool. My dumper does not support flashing from a Mac Pro with no serial number. Maybe I will change that as underlaying tasks changed.

Thx will try it.
Can you remind me the linux GRML flashrom command line to flash MP51.fd ?
Or is it OK to flash from Mojave with RomTool (after booting via OpenCore) ?
After flashing MP51.fd, I just run your rom tool to get a dump and analysis ?

This is the current dump log:

 

[MacNB2]

I wonder if your firewire controller got some ESD or something from the camcorder--or did you not have that connected recently?

If your firewire controller is fried, it looks like you could add one via PCIe card - see here: https://forums.macrumors.com/threads/firewire-pci-expansion-card.2235816/

Thx for the suggestions. Yes I had used the front firewire ports a few days ago from the camcorder. May be it was ESD. Camcoder is fine though and still working.
Yes I can add a PCIe firewire card but they too will require Mac's firewire kexts which I have disabled so would not work the add-in card(unless they come with their own drivers).
But the issue is unable to boot macOS natively (without OpenCore)
Well I could do that but would have to delete the Firewire drivers.

 

[prefuse07]

But the issue is unable to boot macOS natively (without OpenCore)

That's why I think it has something to do with your bootrom...

 

[bookemdano]

Perhaps investigate if linux is loading drivers for the firewire controller. If not, then that explains why it boots successfully, just like how you were able to load Mojave after you disabled the firewire kext.

If linux is loading drivers and the controller shows up maybe you can test for functionality. Looks like DVgrab exists to dump DV via firewire on linux, or maybe you have something like an older external HDD with a firewire interface.

Similarly you could also try installing Windows 10 or 11--I believe it has native drivers for the Mac Pro's firewire controller. If not then they would be in the bootcamp package, which is trivial to get.

Just a general troubleshooting thing, and may not make any difference here, but for a problem like this I like to strip the computer down to "bare bones" as much as possible to eliminate variables. So yank all PCI cards except the GPU, and pull all SATA drives except the one you intend to boot from. Also disconnect any peripherals other than USB mouse/kb and one monitor. You could also try pulling all but one stick of RAM (per CPU). A bad RAM stick can cause all kinds of weird issues--although it's probably unlikely in this case since you ran the hardware test and RAM is one thing it does check.

Anyway, like I said maybe it won't change the outcome but it would be good to know for certain so that you don't end up "barking up the wrong tree" so to speak.

 

[MacNB2]

That's why I think it has something to do with your bootrom...

I can boot natively without OpenCore IF I delete the IOFireWireFamily.kext from any of the macOS's.
So it's that kext that's hanging trying to initialise or setup the firewire h/w.

 

[prefuse07]

update: I just re-read your OP, and it looks like you did try a deeeeep PRAM and SMC reset, so scratch my last post.

 

[tsialex]

This is not BootROM related.

I had this twice before, one time I had to replace the backplane and the other was a short with the front panel board that just needed a front panel replacement.

Test your front panel board with another Mac and see if the same problem moves/also happens with the other Mac, then test the front panel board from the known working Mac with the problematic one. Edit: Thinking better, just do the opposite, test a known working front panel board first, or disconnect it altogether and see if the problem continues. Also, do a clean install of HighSierra/Mojave with a spare disk with a known working Mac Pro. Do not continue your diagnostics with the current install.

Don't forget to check the RTC BR2032 battery voltage and replace it if below 3.00V, test with a multimeter/voltmeter.

 

[Macschrauber]

The Mac Pro runs without the front panel. Just pull the cable and short the jumpstart pads near the diagnostic switch to start the machine.

A healthy VSS Store is not a guarantee for a working nvram at all. If the content of a variable is wrong no tool can tell that. It just has no logical errors. Like checking a drive for errors.

 

[tsialex]

Firewire firmware, like the 82574L Intel Ethernet controllers, are completely independent of the BootROM with it's own SPI flash, so, I stand that this specific issue is not BootROM related, but hardware related.

Since the MAC Addresses for the Ethernet and Firewire controllers are shown in the AHT Hardware Profile picture, this eliminate a blank MAC address related issue, a relatively common defect with Ethernet and Firewire controllers.

 

[Macschrauber]

I bet also for a hardware failure, the flashing of an "empty nvram volume" mp51.fd is just a diagnostic step.

 

[tsialex]

I bet also for a hardware failure, the flashing of an "empty nvram volume" mp51.fd is just a diagnostic step.

I'd agree with that for anything that have NVRAM configuration settings like BT, but the OP already did a multiple NVRAM reset and AFAIK there are no directly related NVRAM configuration settings for FireWire, just things like kernel debug via FireWire kdp_match_name=firewire.

Btw, you can completely bypass the NVRAM VSS store powering the Mac Pro down, disconnect the main power cable, removing the RTC battery, waiting a minute and then powering the Mac Pro without the RTC battery.

 

[MacNB2]

Good suggestions everyone. I have tried all but one.

Perhaps investigate if linux is loading drivers for the firewire controller. If not, then that explains why it boots successfully, just like how you were able to load Mojave after you disabled the firewire kext.

If linux is loading drivers and the controller shows up maybe you can test for functionality. Looks like DVgrab exists to dump DV via firewire on linux, or maybe you have something like an older external HDD with a firewire interface.

Looks like Linux drivers are not loading...at least they don't hang the system like macOS

After booting linux, I checked the devices with lspci command:

macnb@MacPro-eOS:~$ lspci -v |grep -E -i "(1394|firewire|texas)"
0b:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01) (prog-if 00 [Normal decode])
0c:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01) (prog-if 10 [OHCI])
    Kernel modules: firewire_ohci

So linux see's the two firewire devices (one on Front Panel & one Logic Board). But there's an initialisation error in the Kernel log:

macnb@MacPro-eOS:~$ dmesg |grep -E -i "(1394|firewire)"
[    3.754541] firewire_ohci 0000:0c:00.0: failed to read phy reg 2
[    3.754566]  read_phy_reg+0x8d/0xa0 [firewire_ohci]
[    3.754569]  ohci_enable+0x33c/0x4d0 [firewire_ohci]
[    3.754575]  fw_card_add+0x4f/0xa0 [firewire_core]
[    3.754577]  pci_probe+0x45f/0x673 [firewire_ohci]
[    3.754611]  fw_ohci_init+0x4a/0x1000 [firewire_ohci]
[    3.754780] firewire_ohci: probe of 0000:0c:00.0 failed with error -16
macnb@MacPro-eOS:~$

There's a read error and the driver init to fail with error -16.

I looked further in the kernel log and looks like there's a panic in between the "failed to read phy reg 2" and "error -16"...but at least linux does not hang or crash the boot:

macnb@MacPro-eOS:~$ dmesg
....
....
[    3.754541] firewire_ohci 0000:0c:00.0: failed to read phy reg 2
[    3.754549] CPU: 12 PID: 278 Comm: systemd-udevd Tainted: G          I       5.4.0-146-generic #163~18.04.1-Ubuntu
[    3.754550] Hardware name: Apple Inc. MacPro5,1/Mac-27AD2F918AE68F61, BIOS 9144.0.9.0.0 06/08/18
[    3.754551] Call Trace:
[    3.754561]  dump_stack+0x6d/0x8b
[    3.754566]  read_phy_reg+0x8d/0xa0 [firewire_ohci]
[    3.754569]  ohci_enable+0x33c/0x4d0 [firewire_ohci]
[    3.754575]  fw_card_add+0x4f/0xa0 [firewire_core]
[    3.754577]  pci_probe+0x45f/0x673 [firewire_ohci]
[    3.754582]  local_pci_probe+0x47/0xa0
[    3.754584]  pci_device_probe+0x10b/0x1b0
[    3.754587]  really_probe+0xf5/0x440
[    3.754589]  driver_probe_device+0x11b/0x130
[    3.754591]  device_driver_attach+0x58/0x60
[    3.754593]  __driver_attach+0xeb/0x130
[    3.754594]  ? device_driver_attach+0x60/0x60
[    3.754595]  bus_for_each_dev+0x74/0xb0
[    3.754598]  ? kmem_cache_alloc_trace+0x213/0x230
[    3.754600]  driver_attach+0x1e/0x20
[    3.754601]  bus_add_driver+0x159/0x240
[    3.754603]  ? 0xffffffffc057e000
[    3.754604]  ? 0xffffffffc057e000
[    3.754606]  driver_register+0x60/0x100
[    3.754607]  ? 0xffffffffc057e000
[    3.754608]  __pci_register_driver+0x5a/0x60
[    3.754611]  fw_ohci_init+0x4a/0x1000 [firewire_ohci]
[    3.754615]  do_one_initcall+0x4a/0x200
[    3.754617]  ? _cond_resched+0x19/0x40
[    3.754619]  ? kmem_cache_alloc_trace+0x213/0x230
[    3.754623]  do_init_module+0x4f/0x20f
[    3.754627]  load_module+0x1c9b/0x22d0
[    3.754630]  __do_sys_finit_module+0xfc/0x120
[    3.754632]  ? __do_sys_finit_module+0xfc/0x120
[    3.754634]  __x64_sys_finit_module+0x1a/0x20
[    3.754636]  do_syscall_64+0x57/0x190
[    3.754639]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[    3.754640] RIP: 0033:0x7f98c7d15539
[    3.754642] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05
 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1f f9 2c 00 f7 d8 64 89 01 48
[    3.754643] RSP: 002b:00007ffd619429e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[    3.754645] RAX: ffffffffffffffda RBX: 0000563a6b2f2910 RCX: 00007f98c7d15539
[    3.754646] RDX: 0000000000000000 RSI: 00007f98c79f4105 RDI: 000000000000000d
[    3.754646] RBP: 00007f98c79f4105 R08: 0000000000000000 R09: 00007ffd61942b00
[    3.754647] R10: 000000000000000d R11: 0000000000000246 R12: 0000000000000000
[    3.754648] R13: 0000563a6b2fe4c0 R14: 0000000000020000 R15: 0000563a6b2f2910
[    3.754780] firewire_ohci: probe of 0000:0c:00.0 failed with error -16

I also installed DVgrab and it cannot see the attached camera:

macnb@MacPro-eOS:~$ dvgrab
Error: no camera exists

macnb@MacPro-eOS:~$

So I took the Linux drive to another working MacPro and booted it with the camera attached.
DVgrab transferred a clip no problem:

macnb@MacPro-eOS:~$ dvgrab
Found AV/C device with GUID 0x0800460102820000
Waiting for DV...
Capture Started
"dvgrab-001.dv":   207.78 MiB 1513 frames timecode 45:85:85.45 date 2067.02.15 22:26:25
Capture Stopped

macnb@MacPro-eOS:~$

So not looking good for the H/W.

This is not BootROM related.

I had this twice before, one time I had to replace the backplane and the other was a short with the front panel board that just needed a front panel replacement.

View attachment 2183556

Test your front panel board with another Mac and see if the same problem moves/also happens with the other Mac, then test the front panel board from the known working Mac with the problematic one. Edit: Thinking better, just do the opposite, test a known working front panel board first, or disconnect it altogether and see if the problem continues. Also, do a clean install of HighSierra/Mojave with a spare disk with a known working Mac Pro. Do not continue your diagnostics with the current install.

I took out the CPU cage and disconnected the cable that connects the logic board to the Front Panel board.
Rebuilt and started the system by shorting the jumpstart solder pads as suggested above by @Macschrauber.
Problem still exists.

So it not the fireware in the Front Panel but the main logicboard.

I installed a fresh High Sierra drive on a working Mac and installed into this Mac and the problem still exists.

Don't forget to check the RTC BR2032 battery voltage and replace it if below 3.00V, test with a multimeter/voltmeter.

I tested the RTC battery and it measured 3.01V so looks like that is OK.

I bet also for a hardware failure, the flashing of an "empty nvram volume" mp51.fd is just a diagnostic step.

This is the only step I have not tried. But sadly I am resigned to a h/w failure.

Can you remind me the linux GRML flashrom command line to flash MP51.fd ?
Or is it OK to flash from Mojave with RomTool (after booting via OpenCore) ?
After flashing MP51.fd, can I just run your rom tool to get a dump and analysis ?

 

[Macschrauber]

Can you remind me the linux GRML flashrom command line to flash MP51.fd ?
Or is it OK to flash from Mojave with RomTool (after booting via OpenCore) ?
After flashing MP51.fd, can I just run your rom tool to get a dump and analysis ?

for a 4.1 with assumed SST25VF032B

flashrom -p internal:laptop=this_is_not_a_laptop -c SST25VF032B -w thefilename.bin

 

[MacNB2]

for a 4.1 with assumed SST25VF032B

flashrom -p internal:laptop=this_is_not_a_laptop -c SST25VF032B -w thefilename.bin

OK thx.
for using GRML, do I need to shutdown and reboot the Mac into firmware flash mode (long press of power button) ?

 

[bookemdano]

Good work troubleshooting--sucks about the outcome though.

And yeah you're probably right that adding a firewire controller via PCIe may not help as the driver may still hang trying to talk to the built-in controller. Since it seems to be fried anyway, maybe there's a good way to completely kill it so it doesn't cause the driver to hang. In that case you could go with an add-on controller to gain back firewire functionality.

The only other method I can think of is sort of Rube Goldbergian. If you have or are willing to buy a Titan Ridge card to add Thunderbolt 3 (which requires some mods to work properly) you could pick up a TB3->TB2 adapter and a TB2->Firewire adapter and gain back firewire capability that way.

Of course by the time you buy all that stuff you might be able to just buy a replacement backplane board and be done with it. Or dedicate your DV-ripping duties to your other Mac Pro.

 

[MacNB2]

Good work troubleshooting--sucks about the outcome though.

And yeah you're probably right that adding a firewire controller via PCIe may not help as the driver may still hang trying to talk to the built-in controller. Since it seems to be fried anyway, maybe there's a good way to completely kill it so it doesn't cause the driver to hang. In that case you could go with an add-on controller to gain back firewire functionality.

The only other method I can think of is sort of Rube Goldbergian. If you have or are willing to buy a Titan Ridge card to add Thunderbolt 3 (which requires some mods to work properly) you could pick up a TB3->TB2 adapter and a TB2->Firewire adapter and gain back firewire capability that way.

Of course by the time you buy all that stuff you might be able to just buy a replacement backplane board and be done with it. Or dedicate your DV-ripping duties to your other Mac Pro.

Yes it's a shame that Apple does not allow users to disable built-in peripherals like on regular PC's (whose BIOS's allow one to turn ON/OFF internal WiFi, GPU, LAN, PXE, etc). That's why that guy with the Mac mini ended up literally de-soldering the firewire chip 😳

I can work around the lack of built-in firewire but would have to keep track of blocking the firewire kext using OpenCore.

Yes it's a dilemma...do I keep spending on this old girl or save up for AS ???

 

[Macschrauber]

OK thx.
for using GRML, do I need to shutdown and reboot the Mac into firmware flash mode (long press of power button) ?

yes, it's a hardware thing

if "internal:laptop=this_is_not_a_laptop" is not working then try "internal", it depends on the version of flashrom

as you read before use the same programmer

 

[MacNB2]

yes, it's a hardware thing

if "internal:laptop=this_is_not_a_laptop" is not working then try "internal", it depends on the version of flashrom

as you read before use the same programmer

I flashed MP51.fd via GRML simply using:
flashrom -p internal -c SST25VF032B -w MP51.fd

Unfortunately this boot rom does not solve the firewire issue.

Looks like the logic board is faulty (the well the firewire part).

My OCD head in me tells me spend money and get a replacement but serial number will not match the sticker.
My stubbun head tells me to work around the issue with Opencore.
....hmmm a dilemma

 

[tsialex]

My OCD head in me tells me spend money and get a replacement but serial number will not match the sticker.

This is not a problem, if in the future you'll get a replacement backplane, I can match it following the exact same procedure that Apple did back when MacPro5,1 was still supported and needed a replacement backplane. You will have the exact same SSN/HWC/SON.

 

[MacNB2]

This is not a problem, if in the future you'll get a replacement backplane, I can match it following the exact same procedure that Apple did back when MacPro5,1 was still supported and needed a replacement backplane. You will have the exact same SSN/HWC/SON.

Thx. good to know.