Sorry to burst your bubble, but I have been using my pulse rx580 for over 9 months in sierra without any problems.
It is good news though that hs upgrade seems to have resolved your issue...
Yeah it was the weirdest thing. I even formatted the drive and installed a fresh version of 10.12.6 on it. I hadn't even started to move my files to it when BOOM I got an auto reboot.
I was prompted for a firmware update right before I installed High Sierra though, could the firmware update be why it's not crashing now? Should I try Sierra again or will the firmware be downgraded when Sierra is installed?
It must be the 580 doing something because I never had a problem before it, and it's never been a problem in Windows... even when I ran the benchmark games.
I tried taking the ram out one pair at a time, I pulled all hard drives out except for my boot drive. I put the 5770 back in and it didn't restart (although that could've been a coincidence). I tried a fresh macOS install. I tried resetting the NVRAM and the SMC. I tried running disk repair on the drive and it said it was OK.
I watched my temps and they never really went very high. I have the same exact model computer at work and I was getting the same temps and that one never shuts down on me.
My last thing to try was installing a different OS so I went with High Sierra. I haven't had a shutdown since. Weird right?
By the way... is there a small chance that my PULSE RX 580 has a newer firmware on it than yours and perhaps there's just something about it that Sierra doesn't like?
Before getting the RX 580, I made sure that it would be 100% compatible with Sierra and everyone said it was, but for some reason not mine. Perhaps it's defective, but I can't return it because High Sierra fixed it and Windows doesn't have a problem, so I can't rule out software being the cause of the restarts.