Try only RX560 installed, with only one monitor connected, then hold Command + R to boot. If the RX560 works, then it’s something wrong with your HS. If not, then either the card doesn’t work (if you never prove it can work), or the HS recovery partition doesn’t work (this is easy to check because you have the 5770).
[automerge]1582102460[/automerge]
For Mojave, it’s worth to try, but better start with spare drive, and a clean installation
[automerge]1582102460[/automerge]
For Mojave, it’s worth to try, but better start with spare drive, and a clean installation