Thanks for this. So I have follow these exact steps and somehow I am not having succes. Using the Linux flash method I did however get a lot further then I did before.
what I have done was using open core to boot high Sierra with latest boot rom. Then installed the new gpu and flashed it using windows 7. This did not work and only got black screens and ssh acces.
Now with the Linux method I got much further. I removed windows 7 and then booted from the Linux usb. Using ssh i flashed the card successfully. I used the rx560 2gb ROM. However after this process I still can’t get into Mac OS.
I have two drives and both have different results.
the first drive is a fresh Mac OS high Sierra install which I was booting using open core. After a pram reset I was able to boot this disk after flashing and got an image on my internal and external display. It showed that the message that the iMac was shut down because of a problem and if I want to reopen my windows. But when I press any of the options it freezes. After 30seconds or so the Mac shuts down and restarts automatically. This is the same with the open core loader usb plugged in or not plugged in.
the second ssd is my original high Sierra install I used before this upgrade process. Nothing was done with open core. This one boots and shows the backlight coming on. But then after a minute shuts down and reboots again. But one time I got and apple loading screen with the bar stuck at almost the center position. After that the Mac shut down again and rebooted.
Both cases the iMac does not load into Mac OS and gets stuck. Both are however fully bootable on my macbook.
Can I flash my original bootrom into the iMac. (I have a backup of it) to start fresh.
I find it weird that the internal display is always doing the same as the external and the external is never working like it is suppose to. I really want to first get to the state where the external display is working and the internal display is not. Then I can figure out on my own how to use open core.
I will now reinstall my old card and reinstall high sierra. (I have gotten quite fast in this at the moment

. 10 minutes or less. Then I will install the new card again and see if it wants to boot normally.Is there something else I should also do while I have a display again?
thanks
sieuwe