MP 1,1-5,1 Mac Pro (2009) upgrade project (Monterey, Win11)

[HunPro]

I've been a long time user here, received a lot of help regarding my 2009 Mac Pro. Lately it's been neglected and I'm planning an upgrade.

I do read a lot before I ask, but eventually I have to ask. Instead of necroing threads, I decided to create one so my questions won't mess up half a dozen others.

The current machine:
- cMP 4,1 flashed to 5,1
- Single X5680 (stays)
- 4x16 GB reg ECC RAM (stays)
- PCI card with NVMe SSD (256 GB), macOS High Sierra (the SSD stays, but plan to have Win 11 on it)
- Radeon 4870 512 GB (boot screen) => will be upgraded to an MSI Radeon RX 570 (4GB), have the dual 6 pin to 8 pin cable already. Both cards are 150W. (new card ordered, on its way)
- Upgraded WiFi and Bluetooth in the past by 3rd party
- 4 port upgraded USB card by 3rd party
- Win7 on a 128 GB SSD in the optical bay (Boot Camp)
- 3 other disks in the caddies, media and Time Machine
- Optical drive

The goal:
- Some kind of Open Core setup in a way that I won't brick the machine
- macOS Monterey on a newer, larger and faster NVMe SSD (there's an empty PCI slot). Forum says Crucial P3 works fine.
- macOS Monterey with hardware acceleration, 4K screen as full HD retina. (MSI RX 570 should do it, forum says and seller says so)
- If I lose WiFi, no big deal. Prefer ethernet.
- If I lose Bluetooth, no big deal. Prefer cabled peripherals.
- A fresh install of Windows 11 on the old NVMe SSD
- Boot screen to choose between macOS Monterey and Windows 11

First step:
- Copy current MacOS High Sierra install from old NVMe SSD to new NVMe SSD.

Reasons:
- Eventually I want the old SSD to house a Windows 11 install
- More speed for MacOS on a new disk
- old SSD always had problems with waking up from hibernation, looking to regain ability to hibernate
- hibernation would benefit from more disk space (up to 64GB RAM to write/read to/from disk)
- would re-enable swap file thanks to greater space and speed

CCC should do the trick. What would happen though, with 2 High Sierra installs on 2 PCI SSDs? Would I get just an extra boot option? (Right now it's macOS and Win7.)

What tells the Mac Pro which disk to load the boot selector from?

If I erase the old disk, would the Mac still show a boot selector loaded from the newer SSD with the copy of High Sierra?

I'm not very confident, last time I fooled around with such things was before 2004 in the PC world. There you would set these things up in the BIOS.

Thanks in advance!

 

[krakman]

Some kind of Open Core setup in a way that I won't brick the machine

OCLP is the easiest solution. you will need to follow the instructions to make a Bootable Usb Installer

macOS Monterey on a newer, larger and faster NVMe SSD (there's an empty PCI slot). Forum says Crucial P3 works fine.

YES I have several 4.1>5.1 Mac Pros running Monterey and Sequoia using 2 TB Crucial P3. they are cheap on Amazon

If I lose WiFi, no big deal. Prefer ethernet.
- If I lose Bluetooth, no big deal. Prefer cabled peripherals.

OCLP has patch for Legacy Wifi and bluetooth

Boot screen to choose between macOS Monterey and Windows 11

OCLP Boot Picker with handle this, AFAIK the forum suggests windows 10 is the best option and requires installation from DVD not USB. search forum for further details

First step:
- Copy current MacOS High Sierra install from old NVMe SSD to new NVMe SSD.

Personally I would use OCLP to make a bootable USB and instal a fresh copy of Monterey on the new NVME and then copy apps, data, etc. you might have a load of old drivers and obsolete apps on the high Sierra OS. might as well start with a virgin OS.

Radeon 4870 512 GB (boot screen)

keep this card in the computer whilst you do installs. you will need it to select boot from OC.

once you have installed Monterey swap the GPUs

If I erase the old disk, would the Mac still show a boot selector loaded from the newer SSD with the copy of High Sierra?

OCLP haas ists own Boot picker. it is possible to flash the AMD card with firmware to enable boot screen. you will have to do it with windows. instructions available somewhere on this forum.

 

[HunPro]

Thanks!

I'm not sure which way to go yet, OCLP is my first choice. From what I gather, the boot volume can stay on a USB drive for a while.

I'm still confused by the SIP stuff, should I disable it in recovery mode. Reading a lot.

As a test, until the new hardware arrives, I'll install Monterey on a SATA SSD with OCLP. Might even go above Monterey. I decided to keep the old High Sierra install on the current NVMe drive, because it's rock solid.

keep this card in the computer whilst you do installs. you will need it to select boot from OC

So I only need the old GPU to pick the OCLP boot drive, once (or when I make my final choice to have it on an internal drive)? Because after that I'll have the OCLP boot picker, which supports newer GPUs.

 

[HunPro]

Update:

1. Decided to go with OCLP
2. Used a 16 GB USB drive to install it on, it also fitted the latest Monterey install
3. Restarted with USB drive in an original Mac Pro USB port in the back, no messing with SIP, no reset
4. Pressing alt went to original boot menu, selected the EFI boot option (the USB drive) and pressed ctrl+enter to make it default
5. OCLP boot picker came up, picked Monterey install

Then I just followed the instructions to install Monterey on an old 128 GB internal SSD (SATA).

So far so good, works well, bit of flicker in transparent areas, but it's still running on the old Radeon 4870.

Now I'll be testing the installation of Sequoia, using a 500 GB platter USB disk as the setup volume, because the OCLP install requires 20 GBs. The USB mouse and keyboard is unsupported during the install, unless it's in a USB 2.0 hub, per OCLP alert, but my cMP has a USB 3.0 card, I hope plugging them into it solves this issue. (Update: it worked! Plug it in old USB during boot for the alt key to register, then plug it into the USB 3.0 card).

Stuck at "An error occurred while preparing the installation, try running this application again" trying to install Sequoia on the previous Monterey disk (erased, reformatted). Will investigate it further tomorrow.

Thanks again for the tip, so far it's amazing, I've been an idiot for not trying this sooner.

 

[HunPro]

Update: Sequoia was wonky.

Update: DO NOT keep High Sierra, it messes with the boot of newer APFS MacOS installs:

https://forums.macrumors.com/threads/manually-configured-opencore-on-the-mac-pro.2207814/page-501?post=31997429#post-31997429