2012 12-core Constantly Shutting Down

[Auggie]

This started happening a couple weeks ago when it began shutting down whenever it went into deep sleep. Disabling deep sleep kept the problem at bay for awhile, but then it started shutting down while using the computer and performing disk intensive tasks (I was installing Adobe CC suite). Resetting SMC and PRAM again seemed to keep the problem at bay... for a few days.

It doesn't matter what OS or which internal drive I boot off (SSD or one of the regular 3.5" drives): it will shutdown whenever it goes to deep sleep or typically when performing disk intensive tasks such as large file transfers or big software installs (such as a clean OS X install). It may also shutdown when using external USB drives or other high-power draw devices (e.g. Corsair K70 Lux with its two USB plugs).

I pulled all hard drives and booted my primary startup one off a USB drive dock, and although it runs for awhile, it will eventually shutdown; it just doesnt happen as frequently.

A quick AHT showed no issues, but when I started the long test, it would always crash about 10 minutes into it. CPU temps are typically sub-60 degrees Celsius, and the heat sinks aren't painful to the touch heat-wise.

I removed all RAM chips (have 8x8GB) and swapped in different pairs but the problem persists.

I've got an Apple Genius Bar appointment for the weekend but would like to solicit ideas as to the culprit: I think it's either Power Supply, one or both 3.46 CPUs, or motherboard.

Suggestions?

 

[h9826790]

Check the north bridge chip. The rivets of it's own heatsink (under the CPU heatsink) may be broken.

 

[pastrychef]

I doubt the Genius will touch the Mac Pro once he/she realizes the CPUs have been swapped.

You can use whatever software you used to get CPU temp readings to also check Northbridge temps. But shut down during sleep probably eliminates Northbridge overheating...

 

[Auggie]

I doubt the Genius will touch the Mac Pro once he/she realizes the CPUs have been swapped.

You can use whatever software you used to get CPU temp readings to also check Northbridge temps. But shut down during sleep probably eliminates Northbridge overheating...

Yea, I'm worried about that aspect. But at the moment, I have to give it a shot. Worst case is have to get a genuine pair of 2.4's then go back again...

 

[ActionableMango]

I suspect the same things that you do.

The service manual has a few steps you can do to check for a dead power supply. But it sounds like your problem is intermittent, not a dead power supply, so I don't think that procedure will help unless you cannot turn it back on right after a shutdown.

Service manual suggests going to a minimum hardware configuration (i.e. remove all unnecessary hardware including external drives) and seeing if problem persists still.

 

[SurfNorway]

Have you tried running Apple Hardware Test on the machine, and leave it run for the extended version? That puts the machine under a stress test and may give you a clue or two as to where the problem lies. AHT is free, and is probably already installed on your machine, depending on the vintage, the OS it shipped with, and if you've installed a new OS since then (not just incremental upgrades).

 

[Auggie]

Have you tried running Apple Hardware Test on the machine, and leave it run for the extended version? That puts the machine under a stress test and may give you a clue or two as to where the problem lies. AHT is free, and is probably already installed on your machine, depending on the vintage, the OS it shipped with, and if you've installed a new OS since then (not just incremental upgrades).

As I stated in the original post, I enabled the "long" test and it crashed around 10 minutes into it. I tried 3 times and same shutdown around that mark. So it just won't stay on long enough to do the "extended" test.

I suspect the same things that you do.

The service manual has a few steps you can do to check for a dead power supply. But it sounds like your problem is intermittent, not a dead power supply, so I don't think that procedure will help unless you cannot turn it back on right after a shutdown.

Service manual suggests going to a minimum hardware configuration (i.e. remove all unnecessary hardware including external drives) and seeing if problem persists still.

Again, as stated in original message, I did all that, removing all possible internal devices and external devices except the absolute minimum (Apple aluminum USB keyboard, Logitech Revolution MX USB receiver, Apple 27" Monitor mini-DisplayPort connector and it's USB plug) and all drives except the obvious of at least ONE startup drive. I tried this startup both internally mounted and externally through a self-powered USB drive dock, but the end result is the same: eventually an immediate shutdown. Presently, the moment I log into an account, it will shutdown during the desktop initialization phase.

I can boot off an install CD, but if I leave it alone it's base booted system apparently has a power-saving feature that will eventually put it to sleep after a period of inactivity, ergo, it shuts down. Otherwise, I can't do anything appreciable booted off an OS X install disc (when I start an install it shutdowns 5-15 minutes into the install) so it's not really valid test condition for my hardware issue.

I hadn't yet unplugged the CD drive, but it seems to be working fine as I can boot of CD's from it, so I doubt it's the culprit. My assumption is that the power management system is "sensing" an over-drawn power condition and is shutting down the system to protect itself. Now whether there is indeed a power issue (short?) or bad sensor, it's certainly hardware related.

I've been waiting for a stable period of time in order to monitor the north bridge chip, but it's no longer staying on long enough to determine if it's overheating or not.

 

[fhturner]

My assumption is that the power management system is "sensing" an over-drawn power condition and is shutting down the system to protect itself. Now whether there is indeed a power issue (short?) or bad sensor, it's certainly hardware related.

Do you have another CPU tray you can swap for comparison? It sounds like it could be a CPU issue or overtemp to me, like perhaps the heat sink has shifted and not making good contact, or perhaps the fan inside a heatsink has failed. You might also try pulling 1 CPU and running w/ just a single to test. I believe that will ramp up the fans to full blast, so obviously you'd just be doing that to test. After testing the single CPU, swap them. Maybe w/ one or the other, the problem is there, but gone when you swap.

 

[ActionableMango]

I don't understand the theories behind overtemp if it is also shutting down while during sleep.

 

[Auggie]

Do you have another CPU tray you can swap for comparison? It sounds like it could be a CPU issue or overtemp to me, like perhaps the heat sink has shifted and not making good contact, or perhaps the fan inside a heatsink has failed. You might also try pulling 1 CPU and running w/ just a single to test. I believe that will ramp up the fans to full blast, so obviously you'd just be doing that to test. After testing the single CPU, swap them. Maybe w/ one or the other, the problem is there, but gone when you swap.

Although it could be a CPU issue, I don't think it's heat related because after being shutdown all night or all day and I turn it, it may still shutdown after a few minutes after boot up; hardly enough time for it to overheat into a condition forcing abrupt shutdown. Even so, in my previous testing of monitoring CPU temps before a shutdown, it was last shown at a comfortable sub-60 degree range.

And as 'Mango said, the "heat" theory doesn't explain it shutting down immediately after going into deep sleep.

My first concern: should I or should I NOT bring the computer to a Genius Bar with the 3.46's installed? And if I do bring it in and they stop all troubleshooting once they are aware of the non-Apple CPU's, will they continue to work on it later if I return with Apple CPU's installed? Or will it by then be "black-listed?" These machines should typically be long passed AppleCare coverage so I'm not sure why they wouldn't continue working on it because it will definitely be a customer charge, unless it is absolutely determined that the 3.46's are defective.

 

[pastrychef]

My first concern: should I or should I NOT bring the computer to a Genius Bar with the 3.46's installed? And if I do bring it in and they stop all troubleshooting once they are aware of the non-Apple CPU's, will they continue to work on it later if I return with Apple CPU's installed? Or will it by then be "black-listed?" These machines should typically be long passed AppleCare coverage so I'm not sure why they wouldn't continue working on it because it will definitely be a customer charge, unless it is absolutely determined that the 3.46's are defective.

My guess is that it will be flagged as having been "tampered" with and won't work on it even at your cost.

 

[Auggie]

My guess is that it will be flagged as having been "tampered" with and won't work on it even at your cost.

<sigh>. Yea, we're both guessing. The problem is if Apple does refuse service, and continues to do afterwards, then I'm screwed. Perhaps it's best that I just cancel and wait for a pair of original 2.4's shows up somewhere.

 

[pastrychef]

I just did a quick check on E5620 prices on Ebay. They are going for about $10 a pair now. I think it would be much safer to stick these in there before making your visit to the Genius Bar.

 

[Auggie]

Ya think Apple would notice that those are not genuine Apple-supplied CPUs?

 

[pastrychef]

No, I don't think they would know.

 

[h9826790]

Although it could be a CPU issue, I don't think it's heat related because after being shutdown all night or all day and I turn it, it may still shutdown after a few minutes after boot up; hardly enough time for it to overheat into a condition forcing abrupt shutdown. Even so, in my previous testing of monitoring CPU temps before a shutdown, it was last shown at a comfortable sub-60 degree range.

And as 'Mango said, the "heat" theory doesn't explain it shutting down immediately after going into deep sleep.

My first concern: should I or should I NOT bring the computer to a Genius Bar with the 3.46's installed? And if I do bring it in and they stop all troubleshooting once they are aware of the non-Apple CPU's, will they continue to work on it later if I return with Apple CPU's installed? Or will it by then be "black-listed?" These machines should typically be long passed AppleCare coverage so I'm not sure why they wouldn't continue working on it because it will definitely be a customer charge, unless it is absolutely determined that the 3.46's are defective.

Obviously it very depends on how the staff handle it. When I took my flash 4,1 to Apple Store, they just fix it without asking anything. However, I did (and had to) put the original CPU back in, otherwise, my W3690 won't even work with the 4,1 logic board (their test / replacement parts).

In your case, when they know your CPU is user upgraded, some staffs will ask you to go home and put the original CPU back in. There are some sharing by other members (not necessary in Mac Pro's forum, and not necessary about CPU) that the Apple staff told them to "revert" the machine and come back again. Never heard a black list yet (void warranty is another issue).

Since they can't even consider able to test your machine properly with the out of spec (e.g. TDP) CPU. So, even they accept it, they must put the original CPU back in for the tests. That also means they have to take out your CPU anyway. TBH, I don't want to risk my CPU. You have no idea how those technicians handle your non stock CPU. They can just throw it away because it's "foreign object", and should never exist in your cMP. And then charge you for the "new" replacement CPUs accordingly. Even they give you back the 5690, but tell you that the CPU is faulty (because they accidentally kill it), you can't say anything, because you can't prove the CPU was good when you gave them your cMP.

Anyway, I don't think they will care about if the CPU's are the real original one. Even though they can see it. As long as you put the correct model CPU in, they will accept and test it. Just like my case, I gave them my cMP with the W3520, however, with newly applied AS5 thermal paste. They said nothing about it. So, I am quite sure they don't care.

Technically, you don't need to give them a stock config machine. Just like you drop your notebook onto the ground, the whole machine fallen apart, you can still take the debris to there and ask them to fix it. All they do is just give you a new one, and charge you accordingly. We only need to keep it as stock config if we ask for warranty. In your case, I can't see why they "cannot" accept, you are not ask for warranty, but a paid repair service. As I said, they can always swap out your CPUs, and put the "good" one in, and charge you accordingly. In this point of view, you are welcome to give them a non-stock config cMP, because they can charge you more. In fact, you are the person that want to keep everything stock to avoid unnecessary charge (and possible lost a pair of X5690).

 

[rpmurray]

When you first had this happen did you check the console logs to see if it had any hints about what was going on?

 

[Auggie]

When you first had this happen did you check the console logs to see if it had any hints about what was going on?

Logs had showed nothing unusual or out-of-the-ordinary: it will be doing its thing (installing software, file copying or whatnot) then log just stops at the moment of shutting down. New log entries just shows the boot-up processing...
[doublepost=1489254832][/doublepost]

Obviously it very depends on how the staff handle it. When I took my flash 4,1 to Apple Store, they just fix it without asking anything. However, I did (and had to) put the original CPU back in, otherwise, my W3690 won't even work with the 4,1 logic board (their test / replacement parts).

In your case, when they know your CPU is user upgraded, some staffs will ask you to go home and put the original CPU back in. There are some sharing by other members (not necessary in Mac Pro's forum, and not necessary about CPU) that the Apple staff told them to "revert" the machine and come back again. Never heard a black list yet (void warranty is another issue).

Since they can't even consider able to test your machine properly with the out of spec (e.g. TDP) CPU. So, even they accept it, they must put the original CPU back in for the tests. That also means they have to take out your CPU anyway. TBH, I don't want to risk my CPU. You have no idea how those technicians handle your non stock CPU. They can just throw it away because it's "foreign object", and should never exist in your cMP. And then charge you for the "new" replacement CPUs accordingly. Even they give you back the 5690, but tell you that the CPU is faulty (because they accidentally kill it), you can't say anything, because you can't prove the CPU was good when you gave them your cMP.

Anyway, I don't think they will care about if the CPU's are the real original one. Even though they can see it. As long as you put the correct model CPU in, they will accept and test it. Just like my case, I gave them my cMP with the W3520, however, with newly applied AS5 thermal paste. They said nothing about it. So, I am quite sure they don't care.

Technically, you don't need to give them a stock config machine. Just like you drop your notebook onto the ground, the whole machine fallen apart, you can still take the debris to there and ask them to fix it. All they do is just give you a new one, and charge you accordingly. We only need to keep it as stock config if we ask for warranty. In your case, I can't see why they "cannot" accept, you are not ask for warranty, but a paid repair service. As I said, they can always swap out your CPUs, and put the "good" one in, and charge you accordingly. In this point of view, you are welcome to give them a non-stock config cMP, because they can charge you more. In fact, you are the person that want to keep everything stock to avoid unnecessary charge (and possible lost a pair of X5690).

Yea, I think it's staff-dependent upon what action they take. I've always had positive experiences and a no-questions-asked type of interaction when it comes to any modifications or upgrades performed.

I'm not at all concerned that parts will be indiscriminately thrown away or replaced without consulting with the customer, at which point I may be allowed any requests. Of course, if a CPU is bad (or both), I wouldn't have them replace it anyways as I know they wouldn't replace it with like-speed CPU's. In that case, I'd just shop for a replacement CPU and replace it myself. I'm only concerned they won't touch it at all and in the future, though the probability of them "shunning" the machine henceforth is not likely to happen.

Still, the concern will always gnaw at me so I've gone ahead and cancelled the appointment and will be shopping for OEM-spec CPUs.

BTW, those cheap E5620's are only 4-core processsors; the original dual-CPU 5,1's came with 6-core processors, so I need a minimum of E5645's (2.4GHz). A quick check on eBay has them going for about $30 each, which I just ordered a pair.

These "temporary" CPUs should determine if the 3.46's are the culprits (I'm hoping they are because I no longer need Apple and can just order replacement 3.46's), but if not, then Apple will hopefully find the root cause and estimate the damage to my wallet. I'm praying that it's not the motherboard or main board as I think those would be the most expensive repairs, which also may be problematic because the SN data plate was removed by the previous owner when he replaced the case.

 

[fhturner]

I don't understand the theories behind overtemp if it is also shutting down while during sleep.

Yeah, duh...forget I mentioned the overtemp part! But I would still try just pulling a CPU and testing that way. I see you're acquiring some stock CPU components, but you can test by simply pulling a single CPU before you resort to outright swapping them. Maybe you get lucky and CPU B that you pull first is the bad one, so you don't have to go through swapping 2 others in and then swapping back later.

 

[Auggie]

But I would still try just pulling a CPU and testing that way. I see you're acquiring some stock CPU components, but you can test by simply pulling a single CPU before you resort to outright swapping them. Maybe you get lucky and CPU B that you pull first is the bad one, so you don't have to go through swapping 2 others in and then swapping back later.

Yea, I'll do that when the other CPU's come in. I still want to have "stock" CPU's on hand for just this type of scenario where I might have to bring it in to Apple. They're cheap so it's a matter of convenience having a fall-back immediately accessible.

When I'm ready for the swap, I'll pull one CPU and boot the machine. As you say, maybe I'll "get lucky" and find out it was the "bad" CPU. If not, then there's no real time or effort wasted. By the second CPU replacement I will indeed know if it's a bad CPU or not.

 

[nft]

I have a similar issue, 2010 Mac Pro w/2x2.66 and have not been able to fix it. However mine randomly shuts down and immediately restarts. Do you have any error logs for the shutdown?

 

[h9826790]

I have a similar issue, 2010 Mac Pro w/2x2.66 and have not been able to fix it. However mine randomly shuts down and immediately restarts. Do you have any error logs for the shutdown?

Sounds like a power failure (e.g. PSU related). If that's due to overheat, it should be a hard shutdown but no auto re-start.

 

[Auggie]

I have a similar issue, 2010 Mac Pro w/2x2.66 and have not been able to fix it. However mine randomly shuts down and immediately restarts. Do you have any error logs for the shutdown?

Sounds like a power failure (e.g. PSU related). If that's due to overheat, it should be a hard shutdown but no auto re-start.
View attachment 692077

Having that preference checked would restart my computer a few minutes after it randomly shutdown. It's not root cause of my shutdowns as I eventually disabled it and the problem persists so I didn't mention it, but I normally have that option set.

My logs showed no discernible trace or hint of impending shutdown, so it appears that it's occurring from a hardware issue that does not generate log entries or the effect is so sudden there was no time to record an entry.

 

[h9826790]

Having that preference checked would restart my computer a few minutes after it randomly shutdown. It's not root cause of my shutdowns as I eventually disabled it and the problem persists so I didn't mention it, but I normally have that option set.

My logs showed no discernible trace or hint of impending shutdown, so it appears that it's occurring from a hardware issue that does not generate log entries or the effect is so sudden there was no time to record an entry.

Oh, sorry, I should be more specify that's reply to nft's case.

 

[Auggie]

In case anyone was keeping score, my 2.4 CPUs arrived along with hex wrench, chemical cleaners and thermal paste:

1) Removed the front CPU (left side when looking into open case) and booted up without it. Still crashes when going into deep sleep.

2) Installed the replacement front CPU and removed the rear CPU (right side when looking into open case) and it would not boot; blinking power button LED. I didn't track the pattern or frequency. FYI, I had kept all 8 RAM modules installed.

3) Installed replacement rear CPU, bringing total CPUs back to two. Computer boots...

But now it shuts down constantly, more so than it ever did: during the boot cycle, after it boots, during a short AHT test. Speaking of the AHT test, it now seems to "pause" while testing memory and eventually shuts down. Before, I was able to always complete the short AHT test; it was only the long test where it would shut down roughly 5-10 minutes into the process.

Reseting SMC and PRAM has no effect to the shut downs. Both 2.4 CPUs are detected and do seem to be running fine as far as I can tell.

As I had tested the RAM modules a couple weeks ago, swapping in only a random pair at a time with no effect to shut downs, I still don't think they're the cause of the issue, but it does make me wonder why AHT seems to get stuck testing memory now and shuts down; there is the possibility a module(s) got damaged during the troubleshooting and hardware swaps. But with 8 of them, I can deal with that later.

Regardless, it's now pretty much a forgone conclusion that it's not CPU related.

I'm beginning to lean towards a power supply issue as the frequency of shut downs has been increasing since the initial symptoms of just when it went into deep sleep.

So my next step is scheduling a Genius Bar appointment; that will probably have to wait until April.