That breakage will only occur is you don't use Catalina Patcher 1.3.1 to create your patched 10.15.4 installer. The 1.3.1 patcher used the trick of setting /Library/Preferences/com.apple.security.libraryvalidation.plist so that its DisableLibraryValidation field is set to "1". The problematic 1.3.2 and 1.3.3 patchers attempted to fix the support for machines with non-Metal graphics that need the legacy video patch by using DisableLibraryValidation.kext. That approach was abandoned when it was realized that approach broke the Music app. The 1.3.4 patcher reverts back to 10.15.3 support only.
So if you have a Metal compatible graphics card, you have to drop back to the Catalina Patcher 1.3.1 release. The resulting installation works better than any prior release on my MacPro 3,1. I would however get rid of 'viral' AAAMouSSE.kext if you aren't using an ATI card. I do that with the following commands immediately after installation.
sudo mount -uw /
cd /Library/Extensions
sudo rm -rf AAAMouSSE.kext
cd /
sudo kextcache -i /
Since you have a RX 580 8GB, you'll need to leave the AAAMouSSE.kext in place. It won't be any less stable than 10.15.3 but I am not a big fan of that hack since, to my understanding, it's effects extend beyond just the ATI driver.
