NVMe with ST-NGFF2013-C; Vega Internal GPU; Mac Pro 2013 (6,1)

[CodeJingle]

external enclosure for nvme delivers in two days. my boss bakes me a wonderful 'all foods' birthday cake. i received the usb-c to usb-a cable.

i am wiping the oem ssd and doing a clean install of whatever macos version i get from command-option-r then re-upgrading to mojave. i'll do a non-lazy partition wipe so it should be scanning sectors. this is not the nvme there were no crash issues with this drive but just-in-case.

 

[mbosse]

external enclosure for nvme delivers in two days. my day job finally gets air conditioner for the factory floor. my boss bakes me a wonderful 'all foods' birthday cake. i received the usb-c to usb-a cable.

i am wiping the oem ssd and doing a clean install of whatever macos version i get from command-option-r then re-upgrading to mojave. i'll do a non-lazy partition wipe so it should be scanning sectors. this is not the nvme there were no crash issues with this drive but just-in-case.

View attachment 773662

View attachment 773663

View attachment 773664

View attachment 773665

View attachment 773668

Not sure I understood everything you wrote, but I think I got the most essential: Happy Birthday!

Magnus

 

[CodeJingle]

i had to plug the drive into an ubuntu machine to wipe it. diskutil refused to allow me to wipe the drive stating as its reason that i can't erase a bootable disk. i had two physical hard drives plugged in, diskutil said for both of them that i was not allowed to erase them, that they were both either bootable or both set as the startup disk. the experience was unacceptable. the disk utilities should be much better now that apple has their own first party file system. but diskutil isn't better. sad
[doublepost=1533056301][/doublepost]

Happy Birthday!

thanks!

 

[crjackson2134]

i had to plug the drive into an ubuntu machine to wipe it. diskutil refused to allow me to wipe the drive stating as its reason that i can't erase a bootable disk. i had two physical hard drives plugged in, diskutil said for both of them that i was not allowed to erase them, that they were both either bootable or both set as the startup disk. the experience was unacceptable. the disk utilities should be much better now that apple has their own first party file system. but diskutil isn't better. sad

I’ve run into that several times too, I think it sees the recovery files and/or partition and is over zealous about protecting them.

I always keep bootable Ubuntu media handy as a last resort.

I purchased Paragon Hard Disk Manager and it can zap them 99.999% of the time when booted to an alternate drive. It did refuse to allow me to select the delete / wipe options once on an APFS recovery group (I don’t know what to call it in an APFS system, it’s all in the same disk container, same partition).

For that problem, it was an Ubuntu solution. I have since found tha Paragon can delete it, but it requires a reboot after the OS is gone. It seems to latch onto base.dmg and only a reboot can clear that when it happens.

There are no issues on anything except APFS drives. And it’s easy to work around with a reboot.

 

[CodeJingle]

oh god this is bs ...

[-bash-3.2# diskutil repairVolume /dev/disk0s2

Started file system repair on disk0s2

Repairing storage system

Performing fsck_apfs -y -x /dev/disk0s2

Checking volume

Checking the container superblock

Checking the EFI jumpstart record

error: ?: invalid o_oid (0x0)

error: verification/reading of the EFI jumpstart record failed

EFI jumpstart record is invalid

The volume /dev/disk0s2 could not be verified completely

Storage system check exit code is 8

Error: -69716: Storage system verify or repair failed

Underlying error: 8: Exec format error

-bash-3.2#



omg my other boss (not the one that made the cake; i have two bosses) had only borrowed his parents' air conditioner for 48 hours not actually bought one for the company. i rescind my thanks to him and reappropriate it to his parents. another low point of the day.

 

[CodeJingle]

ok it looks like this will work now. a quasi full-disk wipe and reinitialize is such a pain. that took maybe four hours. and i'll have to do this again for nvme!


[doublepost=1533081038][/doublepost]so my cpu overdrive light came on when installing macos then one or two minutes later the computer goes into a low-level emergency shutdown which then corrupts the hard drive. this is probably what happened when i was running boot camp assistant and didn't realize it. [without the hard drive corruption] same behavior as during the thermal interface test.

the cpu isn't hot so again i'm thinking this is based on some over-voltage or over-current protection.

since i can't swap out the processor i'll have to either disable turbo boost or bypass the monitor.

 

[CodeJingle]

anyone know a quick way to disable turbo boost on mac pro (late 2013)?

for example, does one of the magnet-based switches littered all over the boards, does one of those disable turbo boost?

 

[AidenShaw]

anyone know a quick way to disable turbo boost on mac pro (late 2013)?

for example, does one of the magnet-based switches littered all over the boards, does one of those disable turbo boost?

Why disable extra performance?

All of my systems have a BIOS setting to enable/disable Turbo mode - whether you're in legacy or UEFI mode. Most of them separate Turbo (running faster than the base clock) from SpeedStep (dropping below base under low load). Most of my dev/test systems have Turbo enabled and SpeedStep disabled. (When doing timing tests, you don't want the CPU clock jumping all over the place.)

 

[CodeJingle]

Why disable extra performance?

unless you are joking, what is the boot option to change bios settings on a mac, like what is the keyboard shortcut at boot time to enter the ui for modifying bios settings?

you need context before you reply. besides the post you are directly referencing, read the one or two posts prior for context. please answer the question in addition to asking why.

i am in internet recovery mode. i *must* temporarily disable turbo boost. any way to disable from terminal? also, there are a slew of magnet switches in the mac pro, they are there for hardware debugging. i would bet one of the magnet switches forces turbo boost to disable. so which one?

my cpu cooling rating is 600 tdp, but after several straight minutes of turbo boost the hardware does an emergency shutdown. but the cpu is still cool to the touch. so the issue is probably over-voltage or over-current. i debug the problem by temporarily disabling turbo boost. it isn't a permanent fix so no one needs to lecture me about it. just help me to achieve the next goal on my path towards a better permanent solution.

 

[CodeJingle]

Intel Power Gadget free for Mac to monitor thermals and power usage. Exactly what I need.

https://software.intel.com/en-us/articles/intel-power-gadget-20

There is a way to switch turbo boost on and off dynamically I had heard about it on a recent atp podcast (or possible the talk show podcast), though I don't think they referenced a specific app or tool for doing it.

http://atp.fm/episodes/284

Googling for a macos software tool to toggle turbo mode receives some good results with a minimum of required sifting. 'Turbo Boost Switcher' is one such tool, that modifies the 'Turbo Boost MSR register'. This helps, sure, but how this helps me in internet recovery mode I have no idea.

http://www.rugarciap.com/turbo-boost-switcher-for-os-x/

So apparently it is easy to modify internal xeon registers with a macos kernel extension. This is helpful to know in terms of building an affordable hardware-based full-bandwidth PCIe 3.0 pass-thru packet capture tool. Shouldn't require a kernel completely rebuilt from scratch, just a few kernel extensions.

new problem - anyone know how to load a kernel extension after booting from a recovery partition or after booting into internet recovery mode? not sure if this also require disabling sip.

 

[flygbuss]

Don't know if that helps, you've probably already seen it, but this is the app plus code (you have to compile yourself).

https://github.com/rugarciap/Turbo-Boost-Switcher

 

[AidenShaw]

you need context before you reply. besides the post you are directly referencing, read the one or two posts prior for context. please answer the question in addition to asking why.

The poster should also supply some context, and not assume that the reader will go back and read a week's worth of mostly tangential posts. (A whole bunch of birthday cake stuff.)

Your question was "anyone know a quick way to disable turbo boost on mac pro (late 2013)?" If you'd asked "anyone know a quick way to disable turbo boost on mac pro (late 2013) for debugging?" we wouldn't be here.

Please supply some context, rather than expecting that readers will read a week's worth of posts before replying.

 

[CodeJingle]

"anyone know a quick way to disable turbo boost on mac pro (late 2013)?"

You [still] didn't answer the question. You shouldn't need context to answer a simple question. Someone else did chime in with a satisfactory answer, and that person wasn't you (it was flygbuss).

I go to the front page of macrumors and look at the forum thread for any first-party post by macrumors and I am reminded of the toxic environment that is most of the internet. It is good to answer people's questions directly in addition to attempting to uncover the underlying problem in case they asked the wrong question or to help them perceive the problem from more than one perspective. I thought you knew me well enough, to ascertain that I know what I am asking. I don't need you to redirect the question, in case I didn't know what I was talking about and turbo boost didn't actually have anything to do with my problem. Which is not the case, unless I am daft. I've been working on this project 8 months now, when I ask a straight question a straight answer is appreciated.

The problem I see most often, interacting with other people, is a lack of matching wit for wit. You'll see this reading 'The Purloined Letter.' If you can gauge someone's intelligence, and it seems fair, then don't patronize them, give them the straight answer. If you are able to gauge and it comes back low then you start hand-holding. It is quite inefficient to default to hand-holding, or to continue hand-holding in an ongoing interaction even if you know the person you are interacting with is fairly intelligible.

https://en.m.wikipedia.org/wiki/The_Purloined_Letter

https://americanliterature.com/author/edgar-allan-poe/short-story/the-purloined-letter

 

[CodeJingle]

https://github.com/rugarciap/Turbo-Boost-Switcher

thanks yes i linked earlier to the paid version of the rugarciap app/plugin, i guess they also have an open source version. i just bought a pro license for 'Turbo Boost Switcher.' i'll try installing macos on the external drive through a usb 2 hub instead of connecting directly to usb 3 hopefully the downgraded speed keeps the processor from going turbo boost. then after macos is installed i can use 'Turbo Boost Switcher.'



got nvme enclosure set up. the drive is recognized on ubuntu. haven't tested yet on macos.

 

[CodeJingle]

The nvme controller is doing something weird with the partitions. I've seen this before (so not *that* weird). Basically when interfacing a drive directly then switching to putting it behind a usb controller, or vice-versa, sometimes requires erasing the drive and re-partitioning it. Not sure why. Don't want to do that yet. Will have to reconnect the drive directly in the internal slot to back it up before wiping it for external use.

Wiping and installing macos fresh on the oem drive in an external enclosure is working with the usb crippled from 3.0 to 2.0 to not cause an emergency shutdown. Though not by going through a secondary hub or switch. Instead I am using a usb 2.0 cable to connect the drive directly. This method gives optimal stability while still limiting the throughput.

I can do turbo boost emergency shutdown testing after finishing the clean install of high sierra and mojave, and setting up the turbo boost switch tool and intel power gadget.

 

[crjackson2134]

The nvme controller is doing something weird with the partitions. I've seen this before (so not *that* weird). Basically when interfacing a drive directly then switching to putting it behind a usb controller, or vice-versa, sometimes requires erasing the drive and re-partitioning it. Not sure why. Don't want to do that yet. Will have to reconnect the drive directly in the internal slot to back it up before wiping it for external use.

Wiping and installing macos fresh on the oem drive in an external enclosure is working with the usb crippled from 3.0 to 2.0 to not cause an emergency shutdown. Though not by going through a secondary hub or switch. Instead I am using a usb 2.0 cable to connect the drive directly. This method gives optimal stability while still limiting the throughput.

I’ve seen this many times, including spinning drives. I’ve spoken to multiple drive MFGs about this and have been told it has to do with communication protocols of the different interfaces. I don’t know if that’s accurate but it’s not particularly unique as you have already stated.

Just chiming in to confirm it’s probably not indicative of a particular issue related to your project.

 

[CodeJingle]

I am running Prime95 torture test for several hours at 99%+ CPU utilization with Turbo Boost disabled to make sure I have stable thermals before running again with Turbo Boost enabled. CPU temperature seems stable so far, hovering around 70 celsius or 158 fahrenheit. Monitoring stats with Intel Power Gadget, iStat Menus, and Turbo Boost Switcher.

 

[CodeJingle]

yeah running prime95 torture test with 16 threads for four hours straight is no problem with turbo boost disabled (on a processor rated for 150 watts with 8 cores and hyper threading enabled). core cpu temperature was stable around 70 degrees celsius for the entire run.

but the cpu package total power was bouncing back and forth between 114 watts and 124 watts which is already very close to the alleged maximum sustained output power of the portion of the hardware system driving the cpu which is 130 watts.

so i already have a strong indication based on this data that sustained usage of turbo boost probably isn't running into a thermal issue but is running into a power issue. i knew it was a possible issue using a 150 watt processor in a 130 watt board but it seemed others had tried it and none of them had run into significant issues. seems like the wattage rating of the cpu and board aren't just for thermals.

this is all tentative feedback until i actually run the test again with turbo boost enabled. i could be wrong.

 

[CodeJingle]

looking at the latest apple macbook pro throttlegate/thermalgate debacle, and the hacks that came out of that before apple's fix, there is a way to directly modify the settings for the internal power management system. i'll probably have to do something like this to fix my turbo boost issue.

https://www.reddit.com/r/macbookpro/comments/91256u/optimal_cpu_tuning_settings_for_i9_mbp_to_stop/

the voltageshift tool claims to support haswell and broadwell only. mac pro (late 2013) is ivy bridge which is one generation before haswell. so voltageshift as-is may not work with mac pro (late 2013).

https://github.com/sicreative/VoltageShift

 

[CodeJingle]

I have a couple videos I'll upload them later. Some text-based updates:

Booting the Mac Pro (late 2013) with only one GPU has some undocumented side effects:
CPU speed is 1.2 GHz
CPU core count is probably set (not yet verified)
Fan is fixed to full-speed
Internal LED that represents a thermal or power condition is on
Performance of the one GPU seems awful

The special boot mode where you plug in the AC cable while also holding down the power button might trigger the same mode without having to remove a GPU (haven't tested). There is also a magnetic switch next to the GPU connection, trigger it to disable the GPU without having to remove it.

I'm guessing these distinctions are pure hardware and not firmware logic. There are a ton of relays and electronic switches it should be possible to make all those changes occur when the one GPU is removed without having to add special case in the firmware.

I was unable to repro the emergency shutdown, at least not yet. For the graph from one thread at full turbo boost to 16 threads at no turbo boost, the power consumption of the CPU was always less than 130 watts even though the CPU is rated at 150 watts. It seems then the CPU is not causing the emergency shutdown.

I couldn't find any good macos GPU stress test so I put on hold to stress test CPU + GPU simultaneously. Spending more time with these 20+ thermal monitors it seems the GPUs are getting much hotter that I previously thought. So that may be causing the emergency shutdown even though the documentation doesn't specify that either GPU can trigger the same thermal/power condition as the CPU.

Also it seems certain drive failures remain (not nvme issue). After pulling the power cord too early I ran into an issue with 'verifying allocated space' and the alleged fix 'performing deferred repairs' is applied every time. So literally every time I run diskutil I get the response 'performing deferred repairs' but it never seems to actually fix the problem. I've seen that on a few drive failures over the years, that whatever is wrong with the drive or volume, repair or first aid says it was fixed, but the issue is still reported every time I scan.

 

[CodeJingle]

Epic's Unreal Engine always supports the latest graphics standards and is industry-leading in terms of supporting new and updated APIs as they are released. Metal 2 first shipped on High Sierra and was added in Unreal Engine 4.18 ready for preview testing in September 2017 around the time High Sierra shipped.

https://www.unrealengine.com/en-US/blog/vr-on-macos-early-access-support-in-unreal-engine

So I am betting Fortnite, being Epic's 1st party title and built heavily on top of their Unreal Engine, will be a good stress test for GPU. I searched online and couldn't find any GPU stress tools based on Metal 2. With opengl being deprecated it is no longer a good option for programming a stress test for macos based on the opengl API. Really right now on macos Metal 2 is the only good API for direct GPU programming, including burn-in/stress testing.

If anyone has or knows about a good option for GPU stress testing and they know the internal implementation of said stress tool is using Metal 2 please post on the thread or DM me thanks!

-----

Also I still purposely crippled my external hard drive speed to USB 2 for Turbo Boost testing, unable to reproduce the emergency shutdown condition. I may redo with the external drive being driven at USB 3.

 

[CodeJingle]

i’ve confirmed the 130 watt cpu rating of the mac pro (late 2013) is a limitation of both electric and heat. but not in the way i envisioned previously. for example the e5-2687w v2 in the mac pro never consumes more that 130 watts of power. even though it is a 150 watt processor, it is artificially limited by the computer’s hardware and firmware and is never delivered more that 130 watts.

this sounds bad at first, but turns out to be great in practice, since the electrical limitation also limits the heat output to the equivalent 130 tdp. the 2687w processor at this limitation compares favorably to the e5-2667 v2. min heat for 2667 is probably slightly less but max heat for both is the same. 2687w runs at full base clock of 3.4 ghz even for all cores simultaneously (when running all 8 cores may be throttled slighly like by 100 mhz but that isn’t picked up by istat menus). turbo boost never hits full 4 ghz even when running only one core but i consider that a reasonable tradeoff, especially because i don’t expect to be often running only one or two cores.

can anyone elso also confirm or deny this, that the power output to 2687w is artifically limited on the mac pro but is not limited on a different e5 v2 motherboard that is rated for 150+ watts?

 

[flygbuss]

Apperently Rise of Tomb Raider comes with built-in benchmarks for metal 2 on a Mac.
That might be a stress test?

https://barefeats.com/directx_versus_metal_2.html
https://xsreviews.co.uk/news/games-with-built-in-benchmarks-2017/

 

[CodeJingle]

Apperently Rise of Tomb Raider comes with built-in benchmarks for metal 2 on a Mac.
That might be a stress test?

https://barefeats.com/directx_versus_metal_2.html
https://xsreviews.co.uk/news/games-with-built-in-benchmarks-2017/

seems like rise of the tomb raider has three fixed cutscenes generated in real-time displayed along side some stats which include the frames per second. probably not a good gpu burn-in test but gets the job done for consistent comparison point between different gpu while under a decent load.

 

[jaysee_au]

Hi everyone. Been reading this thread for a while, planning to upgrade my 6,1. I ordered the ST-NGFF2013-C and picked up a Samsung 960 Pro second hand. Managed to get the SSD into the adapter, and both into the Mac Pro, but the screw doesn't seem to align with the hole on the motherboard.

See below - the Sinetech adapter is the perfect length, but with the SSD in it, the SSD sticks out slightly past the end.

See image here: https://www.dropbox.com/s/z78mihyn28xqhq4/IMG_0973.JPG?dl=0

As you can see above, the original screw to hold it in can work, just with both SSD blade & adapter it's not fully screwed in, and on an angle. I'm worried about the screw touching the SSD memory chip too in this configuration...

If instead, I just screw the adapter in, it fits perfectly. Is that what people are doing? If so, how then do you keep the SSD in the adapter?

See image here: https://www.dropbox.com/s/tmstvwjq7wmkw7h/IMG_0975.JPG?dl=0

I've managed to get the system working like this, but obviously it's not a long term solution!

See image here: https://www.dropbox.com/s/vs2dhvasatgfrpj/IMG_0978.JPG?dl=0

I ordered the heatsink (EKWB EK-M.2) linked in the first post too, but with that installed it's even less likely to fit. I think I could probably get away with the heatsink on top of the drive, but not the pad / plate below as someone else mentioned above.

I have tested the drive in the machine as per last image above, and it appears to be working fine from the initial install / setup assistant, system information sees it as a 5GT / x4 NVMe drive, and Blackmagic reported around 1500/sec read/write.

Anyone got any suggestions??

Edit: not sure why the images aren't showing...