MP 1,1-5,1 Latebloom - An experimental workaround for the 11.3+ race condition

[LucMac]

Before I start testing... What kind of Backup is mostly recommend? TimeMachine? CCC?
Thank you @Syncretic for your time and effort with latebloom!!!

My Specs:
Currently on macOS 11.2.3 via OC 0.6.6
Quad Core 3,2 GHz
24 GB RAM Tripple Channel
512 GB Samsung 860 Pro on Sonnet PCI Card
5x Seagate 8TB HDDs
Inateck USB 3.0 + Asrock USB-C 3.1 PCI Cards
Sapphire Radeon RX 580 flashed by MVC
Broadcom BCM94360CD Airport + BT
LG BluRay/DVD Drive

 

[eVasilis]

Before latebloom, I hadn't even been able to install 11.3 or 11.4. No matter how many times I tried, the installers would hang and nothing ever progressed no matter how many forced reboots I did. Yesterday, I was able to complete the installation of 11.4 (did not even bother with 11.3) with only a couple of forced reboots. After installation all three cold reboots I did went fine but 4 out of 5 warm ones ended up in the apple prohibitory symbol. The settings initially where latebloom=100, lb_range=20 and lb_debug=1 and after, latebloom=250, lb_range=20 and and lb_debug=1. I do not know how corrupt data can become after so many crashes that is why I plan to try once more installing and booting 11.4, this time with latebloom=300, lb_range=50 and and lb_debug=1. Still, as far as I am concerned this is definite improvement. A big thank you to syncretic!

A better overview of my hardware: MacPro 5,1 (mid 2010); 2 X Intel Xeon CPU X5650; 48BG RAM (triple channel); Samsung 850 Evo SSD (SATA bay 1); Mojave spinner (SATA bay 2); Data disks (bays 3 and 4); Inateck KT 4004 USB 3 card PCIe slot 4; Radeon RX580 pulse 8GB PCIe slot 1; Stock WiFi and Bluetooth modules.

 

[sfalatko]

From the start we noticed that latebloom is a lot more efficient to overcome KPs at cold boot. Warm reboots are much more susceptible to KPs.

@tsialex - thanks. I reread the first post and realized I had missed that.

In your testing (post #42) where the values you reported for only cold boot or for either? And if for either would you recommend when testing individual machines for the best values that we only use warm boot to find the best value or cold boot?

 

[StarPlayrX]

Time machine is slow to do a full restore while it's great for incremental tape drive style backups. CCC all around good. If you know how to use ASR in the command line and your disks are NOT sealed then that works well. My software built into Bigmac2 recovery / installer disks uses ClonetoolX which uses a combination of ASR and hdiutil under the hood and its fully compatible with BigMac2 / Hax.dylib DoNotSeal installs.

if the disk is not sealed OR if the ASR doesn't think your seal is broken, you can create a bootable backup with ASR which is built in to the OS.

cd /Volumes
ls -a
sudo asr -s sourceDiskName -t targetDiskName -er -nov

examples:

sudo asr -s / -t macOS -er -nov
sudo asr -s 'Macintosh HD' -t macOS -er -nov

If not in recovery, omit sudo

@LucMac​

 

[macnrolla]

Before I start testing... What kind of Backup is mostly recommend? TimeMachine? CCC?

I really like CCC, because you can easily create a bootable backup on another disk. Then you can boot that disk, run the upgrade, tests, etc. If something goes wrong, just boot into your original one. With TimeMachine you have to reinstall macOS and import your data back, which can take a lot of time.

 

[macnrolla]

Just some questions in the round:

1. Is it correct that you have to find the sweet spot for your Mac Pro. Not too little but not too much either for the `latebloom` value?
2. What interval do you use when you test new values (+/- 1, 2, 4, 8, etc.)?

 

[hwojtek]

Just some questions in the round:

1. Is it correct that you have to find the sweet spot for your Mac Pro. Not too little but not too much either for the `latebloom` value?

The best results seem to be in the 50-150ms range. However, depending on your CPU clock speed, the number of PCI devices you have, the number of USB devices you have, the number of disks you have, and other configuration items, your value will likely be unique. The only way to determine what works best for you is to experiment

2. What interval do you use when you test new values (+/- 1, 2, 4, 8, etc.)?

Use the successive approximation method. If the minimum is 0 and maximum (or a "certainly working number") is (say) 500, start with the average of the values: 250 (500/2). Keep trying using average of last known working number and last known not working number - remember the latter will change at each unsuccessful try, but this will allow you to find the lowest working number fastest.

 

[dramsey]

EDIT: latebloom=100 /lb_range=20 /lb_debug=on

So far I've had complete success with 11.4 on my 5,1. I'm running 2x 5690s, 96GB of RAM, and my system volume is a 2TB NVME SSD on a Highpoint 7101 card.

(I do wonder, what with timing being an issue, if there are any advantages to having the boot volume on such a card? I had not tried anything over 11.2.3 prior to installing Latebloom so I don't know if it would have worked or not.)

Haven't tried a warm boot yet, but 5+ cold boots without issue so far.

 

[tsialex]

Try changing your dns to google’s, cloudflare’s or 9.9.9.9

I get pretty good download speeds from Apple CDN, which I believe is Akamai, using NextDNS so you can try their service too (it’s free).

I'm using a PiHole with Cloudfare DNS. The real problem is the limited overseas bandwidth/congestion of the maritime cable during day time. I have very good speeds a day or two later when the CDN that Apple uses here caches it, today I've tested downloading 11.2.3 again and got at least 12MB/s.

@tsialex - thanks. I reread the first post and realized I had missed that.

In your testing (post #42) where the values you reported for only cold boot or for either? And if for either would you recommend when testing individual machines for the best values that we only use warm boot to find the best value or cold boot?

Latebloom is a very good workaround for cold boots, but the efficacy for warm boots is not as good. Even doubling the value that I got working for cold boots don't make warm reboots as reliable. This is a problem for software upgrades.

Other people also have KPs with warm reboots, so it's not something that I've found alone.

 

[Stex]

So far I've had complete success with 11.4 on my 5,1. I'm running 2x 5690s, 96GB of RAM, and my system volume is a 2TB NVME SSD on a Highpoint 7101 card.

(I do wonder, what with timing being an issue, if there are any advantages to having the boot volume on such a card? I had not tried anything over 11.2.3 prior to installing Latebloom so I don't know if it would have worked or not.)

Haven't tried a warm boot yet, but 5+ cold boots without issue so far.

Please specify the values you used to make your report valuable to the developer(s).

 

[sfalatko]

I'm using a PiHole with Cloudfare DNS. The real problem is the limited overseas bandwidth/congestion of the maritime cable during day time. I have very good speeds a day or two later when the CDN that Apple uses here caches it, today I've tested downloading 11.2.3 again and got at least 12MB/s.


Latebloom is a very good workaround for cold boots, but the efficacy for warm boots is not as good. Even doubling the value that I got working for cold boots don't make warm reboots as reliable.

Other people also have KPs with warm reboots, so it's not something that I've found alone.

@tsialex - Thanks. I have found the same (though don't really have enough trials to call it other than empirical). I do well at 100 for cold boot but was getting a hang almost every other warm boot at 250.

 

[cdf]

I can confirm the cold boot vs warm boot situation.

In my case, almost every cold boot into macOS 11.4 is successful with 100 ms, and any failures might actually be because of powering back on too quickly after shutting down. Warm booting into recovery is just as reliable.

As for warm booting into the OS, my testing up to now indicates a success rate of around 50%. Increasing the delay (even to 500 ms) doesn't appear to help.

 

[HuRR]

I can confirm the cold boot vs warm boot situation.

In my case, almost every cold boot into macOS 11.4 is successful with 100 ms, and any failures might actually be because of powering back on too quickly after shutting down. Warm booting into recovery is just as reliable.

As for warm booting into the OS, my testing up to now indicates a success rate of around 50%. Increasing the delay (even to 500 ms) doesn't appear to help.

Can you try my parameters, 280, 50, and debug? Curious if that changes anything. I've had success with warm boots. Haven't had a single hang, yet. But who knows for now.

 

[StarPlayrX]

@Syncretic

To make it easier to test what settings are working on various systems, I am wondering if there should be a version of latebloom that phones home.

It could include boot-args, system version, build #, login success or no success (or even a success rate over time) and send that data on each boot. The data can be collected and see what numbers are the most promising and make any adjustments from there. You could just send the data once per week per machine until it's out of beta. Just a thought.

 

[Syncretic]

A Request:

Now that we're starting to see some results, I'd like to start compiling them in an orderly fashion, so that we can begin looking for patterns. As such, I'd ask the following:

Once you've done your testing and settled onlatebloom=/lb_range=/lb_debug= values that work for you, please post those values along with a detailed description of the system as it was tested. (If you're still experimenting, please wait until you're satisfied before posting all this). For system details, I'd like to see the following, in whatever format you choose (I'll be collecting them and reformatting them into a table in Post #2 as time permits, so use whatever format is easiest for you):

• Actual model (e.g. MacPro 3,1; MacPro 4>5,1; etc.). While there should be no difference whatsoever between a 5,1 and a 4,1 that's flashed to 5,1, please specify if it's flashed (e.g. 4>5,1 or 4.5,1 or similar).
  SPECIAL DISPENSATION: While this is not a Hackintosh forum, nor is this a Hackintosh thread, I'll welcome any Hackintosh results here, since those could conceivably shed more light on the issues at hand. If you post Hackintosh results, please provide as much detail as possible about your system (motherboard, overclocking, any special hardware, etc.). Please note that this invitation applies only to Hackintosh results; if the thread devolves into Hackintosh-specific discussions or gets noisy, I will ask the moderators to take action.
• BootROM version (preferably the true version, e.g. 144.0.0.0.0, vs. the OpenCore substituted version)
• OC version (if using OpenCore)
• CPU and count (e.g. 1x X5680, 2x X5675, etc.)
• Memory count, size, and speed (e.g. 4x2GB 1066, 8x8GB 1333, etc.)
• WiFi (e.g. none, factory, BCM94322MC, BCM94360CD, etc.)
• Bluetooth (e.g. none, factory, USB (model info), combo (w/upgraded WiFi), etc.)
• Disks (internal, including optical drives) - physical, location, logical/formats (no need for size of each partition), e.g.
  • Seagate ST3000DM008 3TB, Bay 2, EFI/APFS Mojave
  • SanDisk SDSSDA 240GB SSD, ODD 1, EFI/HFS+ Sierra/NTFS Windows
  • Samsung 970 EVO 1TB NVMe, HighPoint 7101A, EFI/APFS Big Sur
  • (etc.)
    (Please include the name or version of the OS that formatted the partition (if known) - the APFS format has changed over time, and older APFS-capable MacOS versions can't read some newer APFS volumes.
    There's no need for any personal details, serial numbers, partition sizes, disk names, notation of bootable vs. data drives, etc.)
• NVMe devices (cards/location/blades), e.g.
  • kryoM.2 evo, slot 4, 1x WD Blue SN550 1TB
  • Sonnet M.2 FUS-SSD-4X4-E3, slot 3, 4x Toshiba XG6-P 2TB
  • (etc.)
• PCIe devices/location:
  • AMD RX570 4GB, slot 1
  • Sonnet Allegro USB3-4PM-E, slot 4
  • SIIG NN-E38012-S3, slot 2
  • (etc.)
• USB devices (including external storage):
  • Apple wired keyboard
  • Logitech wireless mouse (Unifying interface)
  • 128GB USB3 flash drive
  • External USB2 Blu-Ray burner
  • (etc.)

If you're lacking some details, or can't access the hardware to get them, just include whatever details you can. The more information we have, the more likely we'll be able to find useful patterns (assuming any exist).

So, as an example modeled loosely on my own system, I might post:

---

latebloom=90, lb_range=20, lb_debug=0
MacPro 4,1 -> 5,1
BootROM 144.0.0.0.0
OpenCore 0.6.8, custom config
CPU: 2x X5675
RAM: 8x8GB 1066
No WiFi
Factory Bluetooth
Disks:

• Inland 860270 256GB NVMe, MicroConnectors PCIE-M20803HS slot 3, EFI/APFS Mojave
• SanDisk SDSSA-240G SSD, ODD 2, EFI/APFS Big Sur
• HP S700 120GB SSD, ODD1, EFI/APFS Big Sur
• Inland Professional 256GBM 256GB SSD, Bay 2, EFI/APFS Catalina
• Inland Professional 256GBM 256GB SSD, Bay 1, EFI/APFS Mojave
• HGST 0F22408 4TB HDD, Bay 4, EFI/APFS Mojave
• Seagate ST3000DM008 3TB HDD, Bay 3, EFI/APFS Mojave

NVMe:

• (see first item under Disks)

PCIe:

• AMD RX570 4GB, slot 1
• (NVMe, slot 3)
• Sonnet Allegro PCIE-USB31-X3, slot 4

USB:

• Apple wired keyboard
  • Logitech wired trackball (on built-in Apple keyboard hub)
• KVM switch
  • Wired keyboard
  • Logitech M570 wireless trackball (Unifying interface)
• USB3 hub
  • 128GB USB3 flash drive, Windows exFAT
  • 64GB USB3 flash drive, High Sierra HFS+
  • 32GB USB3 flash drive, Catalina HFS+
• 32GB USB3 flash drive, Windows FAT32
• Sabrent USB3/SATA3 adapter
  • Inland Professional 256GBM 256GB SSD, EFI/APFS High Sierra

---

I may or may not include username attribution in the public table; if you'd rather remain anonymous, just PM me your results, let me know you'd like anonymity, and I'll include your results without attribution.

Thanks in advance. If you're unable or unwilling to get that level of detail, please post whatever you can, and I'll include your results as best I'm able.

 

[Syncretic]

@Syncretic

To make it easier to test what settings are working on various systems, I am wondering if there should be a version of latebloom that phones home.

It could include boot-args, system version, build #, login success or no success (or even a success rate over time) and send that data on each boot. The data can be collected and see what numbers are the most promising and make any adjustments from there. You could just send the data once per week per machine until it's out of beta. Just a thought.

That would have the benefit of accuracy (vs. user perception). However, it presents multiple problems. Latebloom runs in the early boot environment, before the PCI bus is even probed, before the network is brought up, before disk storage is brought online. By the time those services are available, latebloom has finished executing and is dormant. I'd have to create a user-land process and some form of IPC to gather the data from the kext and upload it.

Latebloom never knows if the system really finished booting; its last execution is the last PCI bus probe loop iteration, and latebloom doesn't know which one will be the last, or what happens afterward. I could use NVRAM to store boot progress and statistics, but even that is somewhat problematic due to timing (the NVRAM kext isn't reliably available when latebloom runs, and waiting on the NVRAM kext (or any other service) would almost certainly allow the PCI bus probe loop to run before latebloom got to inject its delays - which rather obviates its usefulness).

Without persistent storage, tracking failed boots is impossible, and tracking successes would involve more complexity than is probably warranted here. I'm more inclined to start collecting results from folks and see where that leads; if some consistent patterns emerge, it might be worthwhile to write a simple profiler that counts PCI devices, performs a cursory analysis of all attached disks, and applies whatever we've learned to produce a "best guess" delay number. We'll see.

All of that aside, I'm generally not fond of software that phones home. There's a level of trust involved in loading software on one's system, especially an unsigned kext, and unless you're willing to sniff your network traffic and decrypt the message, there's no way to know what information is being "phoned home." I'd prefer to avoid the issue altogether by keeping the software mute, keeping all data local, and trusting the users to provide reasonably accurate results.

 

[cdf]

Can you try my parameters, 280, 50, and debug? Curious if that changes anything. I've had success with warm boots. Haven't had a single hang, yet. But who knows for now.

Just tried these parameters. Unfortunately, it resulted in a hang on the first try (and even on the cold boot afterwards). I wonder now whether two machines with the same hardware could even behave similarly. Hopefully we'll see a pattern in the user data.

 

[macnrolla]

Can you try my parameters, 280, 50, and debug? Curious if that changes anything. I've had success with warm boots. Haven't had a single hang, yet. But who knows for now.

Interesting, i can confirm the cold boot vs warm boot situation too. So i changed the values only for warm boot (latebloom=148) and its works better. Back to my "cold value" (latebloom=248) and cold boot works as before. Minor note, these are only initial test figures. This would also explain why the installer ran through with the default values while i first tested latebloom, but problems appear later with cold boot.

@Syncretic Could it be that you need 2 values, one for cold boot (higher) and another for warm boot (lower)? Is it even possible to distinguish the difference between cold and warm boot via code?

 

[HuRR]

latebloom=220 lb_range=20 lb_debug=1

Mac Pro 5,1
BootROM: 144.0.0.0.0\9144.0.7.1.0 (OpenCore 0.7.1, Martin Lo Package)
CPU: X5680
RAM: 3x16GB 1333
OS: 11.5 RC2 (7/19/21)
WiFi/BT: BCM94360CD (purchased from macvidcards a while back)

• Disks (internal, including optical drives) - physical, location, logical/formats (no need for size of each partition), e.g.
  • Apple SSD A45ACXBA9TA (Angelbird SSD), 512.11 GB, EFI/APS Mojave, Bay 1.
  • Samsung SSD 850 EVO 250.06GB SSD, EFI/NTFS Windows, Bay 2.
  • Seagate 1TB ST1000DM003-1CH162, EFI/HFS+ High Sierra, Bay 3
  • PNY CS900 240GB SSD, 240.06 GB, HFS+, Bay 4, Storage
  • SuperDrive Optical Drive, ODD1.
• NVMe devices (cards/location/blades), e.g.
  • HP SSD EX900 500GB, 500.11GB, Slot 2
• PCIe devices/location:
  • AMD Sapphire Pulse 580 8GB with EVGA Powerlink (Running Apple LED Cinema & Acer Monitor), slot 1
  • INATECK KT4004 USB 3.0 PCIe, slot 4
  • HP SSD EX900 500GB, 500.11, Slot 2
• USB devices (including external storage):
  • iClever Ergonomic Mouse
  • Logitech Mx Keys (Unifying interface)
  • ASTRO A40 TR Mix Amp
  • Sabrent USB 3.0 Hub
  • LaCie Rugged USB-C Media
  • Logitech c922 webcam

 

[Macschrauber]

Interesting, i can confirm the cold boot vs warm boot situation too. So i changed the values only for warm boot (latebloom=148) and its works better. Back to my "cold value" (latebloom=248) and cold boot works too. Minor note, these are only initial test figures. This would also explain why the installer ran through with the default values while i first tested latebloom, but problems appear later with cold boot.

@Syncretic Could it be that you need 2 values, one for cold boot (higher) and another for warm boot (lower)? Is it even possible to distinguish the difference between cold and warm boot via code?

you could taka a look at RefindPlus. A bootloader what (aside of other useful things) can run two instances of OpenCore. You could name one cold and one warm.

MyBootMgr: RefindPlus|OpenCore

section and follow the steps one after the otherThis is a guide to using OpenCore and RefindPlus, to multiboot Classic MacPros (3,1 to 5,1) as well as equivalent Xserve (2,1 and 3,1). It involves booting Mac OS from one or more OpenCore instances chain-loaded from RefindPlus and booting other OS...

forums.macrumors.com

 

[macnrolla]

you could taka a look at RefindPlus. A bootloader what (aside of other useful things) can run two instances of OpenCore. You could name one cold and one warm.

MyBootMgr: RefindPlus|OpenCore

section and follow the steps one after the otherThis is a guide to using OpenCore and RefindPlus, to multiboot Classic MacPros (3,1 to 5,1) as well as equivalent Xserve (2,1 and 3,1). It involves booting Mac OS from one or more OpenCore instances chain-loaded from RefindPlus and booting other OS...

forums.macrumors.com

Ok, thats nice. Unfortunately I do not know RefindPlus yet. But so far it is only a theory with the 2 values for cold and warm boot.

 

[HuRR]

Just tried these parameters. Unfortunately, it resulted in a hang on the first try (and even on the cold boot afterwards). I wonder now whether two machines with the same hardware could even behave similarly. Hopefully we'll see a pattern in the user data.

Interesting. I'm currently on 11.5 and when I use the script in this thread to check if latebloom was successful. It's not showing any results. So weird.

 

[dramsey]

latebloom=200 lb_range=20 lb_debug=1

Mac Pro 5,1 (not flashed)
BootROM 9144.0.7.1.0
CPU: 2 x X5690
Memory: 6x16GB DDR3 1333MHz
WiFi/BT: Native, unused

Disks:
• 2TB Sabrent Rocket NVME boot drive, on Highpoint 7101 RAID card
• 2x1TB NVME Samsung 970 in RAID 0, on Highpoint 7101 RAID card

PCIE:
• Slot 1: Highpoint 7101
• Slot 2: AMD RX580 video card
• Slot 3: Empty
• Slot 4: Sonnet USB 3.1 Pro 4-port USB A

Native USB:
• “Brand New Model F Keyboards” F77 wired USB keyboard in native poirt
• Logitech trackball with USB receiver in front USB port

Sonnet USB 3 card:
• OWC USB 3 drive dock
• AUKEY USB 3 hub
• Sony XQD card reader

Note: I had to shut down my system to pull the Sonnet card to get the model. After that, I had two failed boots in a row: the first showed the "Stop" (circle/slash) symbol and a URL along the lines of "apple.com/startup"; the second was just an infinitely paused progress bar. Third time the system booted perfectly.

 

[Syncretic]

Thank you for your submissions!

Mac Pro 5,1
CPU: X5980

Should that perhaps be X5680? I'm not aware of an X5980 CPU...

Mac Pro 5,1 (not flashed)
CPU: 2x3960

Should that perhaps be 2x W3690? I'm not aware of a 3960 CPU, other than the i7-3960X, which I'm pretty sure won't fit in an LGA1366 socket...

 

[HuRR]

Thank you for your submissions!

Should that perhaps be X5680? I'm not aware of an X5980 CPU...


Should that perhaps be 2x W3690? I'm not aware of a 3960 CPU, other than the i7-3960X, which I'm pretty sure won't fit in an LGA1366 socket...

Yup! Good catch. That was a typo! Fixed, also fixed my Boot Rom since I use OpenCore.