I had a similar issue with an incompatible VBIOS. My card was used and I had to find a bios on techpowerup.com that I would work.
You Mac is hanging when it is trying to switch to the OS provided drivers.
This is why I asked if it ever worked in your Mac. My guess would be you have a used RX580 that was flashed for mining and you need to reinstall the stock VBIOS.
