Resolved cMP 2009 CPU tray issue

[JedNZ]

Last week, and then again last night, I restarted after applying a software update to my cMP (last night it was a LittleSnitch updater that requires a restart) and it was very sluggish to get to the startup chime. Last night on a second restart there was simply no startup chime at all.

Primary cMP - 12-Core X5680 - see my sig for the rest - this is my main machine.

I have also have a backup, secondary cMP - Hexa-core X5680 - used as a media and internet server.

Fans run, and optical drive and hard drives spin up. I'm doubting it's a PSU issue.
No LEDs indicating on the CPU tray.

Here's what I've tried already - sorry it's going to be a little verbose:
1. Took out ram from CPU-B - restarted - No Sign Of Life (NSOL) - fans run, but don't roar.
2. Put ram back into CPU-B, and removed ram from CPU-A. NSOL - fans run, but don't roar.
3. Replaced PRAM battery - CR2032. Pressed the button to the left of the PRAM battery. NSOL - fans run, but don't roar.
4. Unplugged the power cord, and held down the power button for 10+ seconds. Booted back up - NSOL - fans run, but don't roar.
5. Removed/disconnected all drives and PCI-e cards. Try to boot up from Firewire external drive. NSOL - fans run, but don't roar.
6. Placed my CPU tray into my other cMP 2009 (Hexa-core - single CPU). NSOL - fans run, but don't roar.
7. Swapped in my ram from my Hexa cMP to my 12-core - NSOL - fans run, but don't roar.
8. Remove CPU-A, inspect - no obvious signs of damage - clean CPU and heatsink, apply Arctic Silver5. Reinstall, restart - NSOL - fans run, but don't roar.
9. Remove CPU-A CPU and remove Northbridge heatsink. Inspect - no obvious sign of damage - both push-pin rivets in excellent condition - clean heatsink and Northbridge chip thingy, apply Arctic Silver5. Broke one of the push-pin rivets, so replace it with one of the push-pins I bought off eBay (had to cut 3mm off the top - used the original rivet for the one that hides under the CPU heatsink, and the replacement one that is exposed). Restart - NSOL - fans run, but don't roar.
10. Remove, inspect, clean and re-install CPU-B. NSOL - fans run, but don't roar.
11. Put my 12-core tray back in my hexa-core cMP to see if that changes anything - nope - NSOL - fans run, but don't roar.
12. Put the 12-core CPU tray back in my 12-Core cMP. Get a red LED Ram indicator on ram slot 7. cMP starts up and boots to Recovery HD. Fans roar. Hopes are high, but on restart NSOL - fans run, but don't roar.
13. Put Hexa-core CPU tray into 12-core cMP. Boots fine - all ram shows up. Fans roar but then settle down after a minute or two. Restart - same result. Shutdown - put 4 x 8GB ram modules from my 12-core CPU tray into the Hexa-core tray - restart - no problems all working.

Final result - the Hexa-core CPU tray is now in my primary cMP and working fine with all my drives and PCIe adapters etc.

So, what I learned:
Can't be a PSU issue - Hexa-core CPU tray boots fine in my primary cMP.
Ram not an issue - works in my Hexa-core CPU tray in my primary cMP.
New PRAM battery - so that's not an issue.
All my hard drives, PCI-e cards etc are all working in my primary cMP when running my Hexa-core CPU tray.

So I'm guessing it's something on the 12-core CPU tray.

When I press the Diagnostics LED (on the backplane, to the left of the PCIe fan):
I get a green LED for the third one down that has PSU PWROK to the right of it.
I get an orange LED for the 4th one down that has 5V STBY to the right of it.

I also get two green LEDs to the left of the DIAG LED button - one is the top one with SYS PG to the left of it, and a green LED for the GPU OK.

The only issues I've had leading up to the failure have been the long wait for the startup chime (bong) - temps have all been around average - no crashes or unexpected shutdowns.

Any ideas on what the issue might be? Any other troubleshooting ideas I can try?

 

[JedNZ]

Where does the firmware exist on a cMP - on the backplane motherboard, or on the CPU tray somewhere?

Just wondering if the firmware may have become corrupted if it lives somewhere on the CPU tray. If so, is there a way to reapply/flush out/restore the firmware update?

 

[h9826790]

Where does the firmware exist on a cMP - on the backplane motherboard, or on the CPU tray somewhere?

Just wondering if the firmware may have become corrupted if it lives somewhere on the CPU tray. If so, is there a way to reapply/flush out/restore the firmware update?

100% on logic board. CPU tray independent.

 

[ActionableMango]

Being a 2009 dual CPU upgrade, and that the problem is on the CPU tray, that makes me suspicious. Many people have problems there.

When you inspected the CPUs, did you inspect the sockets too?

Delidded or lidded?

 

[adam9c1]

The CPU tray has firmware too.

Doubt this will fix your issue, but it's free:
power up the cMP with the single CPU and zap PRAM.
Shut down and swap in the dual socket tray.

Another thing to try is this:
remove CPU B and heatsink and try to run with A only.
Also remove both CPUS and install CPU B into A slot only.

 

[h9826790]

The CPU tray has firmware too.

Not entirely wrong, the CPU tray carry the SMC firmware, but not the Mac firmware.

The 4,1 / 5,1 firmware is 100% stored on the Logicboard.

I flash my 4,1 to 5,1 with my factory 4,1 CPU tray.

And then get a new 2009 CPU tray directly from Apple. My Mac is still 5,1 on 1st boot with the new CPU tray.

Apple didn’t give me a 5,1 CPU tray because on all documents shows that they only treat it as a 2009 Mac Pro.

The SMC on the tray is the 2009 SMC.

So, the only explanation is that the 5,1 firmware has nothing to do with the CPU tray.

 

[adam9c1]

Well, I have a 2009 cMP (flashed to 5,1) and got a tray marked as 2009 but turns out it's a 2010, and when it's in the machine fans run at full speed.
The tray functions fine in a true 2010 cMP.

 

[h9826790]

Well, I have a 2009 cMP (flashed to 5,1) and got a tray marked as 2009 but turns out it's a 2010, and when it's in the machine fans run at full speed.
The tray functions fine in a true 2010 cMP.

That’s because the SMC version mismatch. Nothing to do with 4,1 / 5,1 firmware.

The 2009 SMC can boot with both 4,1 or 5,1 firmware. When you flash a 4,1 to 5,1, the SMC won’t chnage. That means the SMC is totally indepent to the Mac’s firmware.

There is no “4,1 CPU Tray” or “5,1 CPU tray”, but only “2009 CPU tray” or “2010/2012 CPU tray”. The reason behind is because the tray has nothing to do with the firmware version, but the SMC version.

Whenever the SMC version doesn’t match, the fan will go full speed as a safety precaution. This happen regardless the firmware version.

 

[JedNZ]

ActionableMango: delidded - can not see any signs of damage using a magnifying glass onto the CPU and on the CPU tray where it sits.

Adam9c1: Have done a PRAM reset using the Hexa-core, then put the 12-Core tray back in but still NSOL.
Just in the process of removing CPU-B and seeing if it will start up with just CPU-A in the CPU-A socket (i.e. CPU-B completely removed). Then doing as you suggest - swapping CPU-B into the CPU-A socket.

h9826790: I appreciate the info about 4,1 and 5,1 firmware being located on the logic board (backplane), and SMC living on the CPU tray. Is there any way of resetting or re-installing the SMC firmware on the tray?

 

[JedNZ]

Okay - put CPU from CPU-B into CPU-A socket and NSOL on restart. Decided to hold down the power button on next attempt at restart, and the machine boots up. Yay! Ram is registered (3 x 4GB modules in slots 1 to 3 on CPU-A). Fans are roaring but this is due to CPU-B not being installed.

Going to install the original CPU-A into the CPU-B socket and see if it starts up with both CPUs. If it doesn't then I guess the original CPU-A is the problem.

 

[tsialex]

Going to install the original CPU-A into the CPU-B socket and see if it starts up with both CPUs. If it doesn't then I guess the original CPU-A is the problem.

I had the exactly same problem, CPU-A stopped working (X5677 from China, still lidded). Finally, tried one by one on my single core tray and confirmed a bad processor.

 

[JedNZ]

Final diagnosis after multiple attempts at getting the original CPU-A going in CPU-B socket: the CPU must be fried/damaged. On the hunt now for a replacement X5680, but much happier that I've got it working with just one CPU installed. Thought my system was completely hosed.

 

[JedNZ]

Replacement X5680 arrived. Delidded it - broke my crappy quality vice doing it! Cleared the solder off, and cleaned it up with very fine sandpaper, cleaned with Arctic Clean Remover and Purifier. Installed it in CPU-B (didn't install any ram in the CPU-B side). Started it up and CPU-B isn't showing at all - just CPU-A and the ram in CPU-A.

Hmmm.
Reset PRAM – CPU-B still not showing.
Unplug power, wait 20secs, plug power back in, wait 10 secs, Restart – CPU-B still not showing.
Shutdown, then hold down power on button for 15 secs to prompt the firmware thing. Get the firmware "beeep". Restart – CPU-B still not showing.
Pull out CPU-B (unscrew heatsink etc)- check thermal paste and reapply on an area with thin coverage. Restart – CPU-B still not showing.
Tighten screws for CPU-B heatsink. Restart – SUCCESS!! 12 cores showing. Fans roaring.
Shut down, install ram in CPU-B ram slots. Restart – CPU-B ram not showing. What the?!
Reset PMU / PRAM / SMC routines again.
Restart - CPU-B ram still not showing. Double what the?!! Fans aren't roaring.
Pull CPU tray out - DUH! Didn't push the ram in - it was just sitting in the slots. Grrr!! Push the ram in.
I am now the user of a 12 core. Yay!

Sensors showing CPU-A are about 10˚C higher than CPU-B.
CPU-A ram are about 10˚ higher than CPU-B ram.
Have adjusted BOOSTA to 2000rpm (using Macs Fan Control) - tempt came down 4˚ (as shown below).
Northbridge about 5˚C less than before - probably due to new thermal paste and rivets.

No doubt the thermal paste will need to cure, but pretty happy with how it looks - let me know if any of the sensors don't appear right if you have a similar (dual CPU) system.

iStats Menus



Interestingly, Macs Fan Control is showing slightly different sensor temps



And TG Pro sensor readings