Yeh the 5770 does not play nice with HS. So I ran beta 5 on 10.13.4. Bad idea. After the install, the 5770 was not accelerated. You can tell if the dock isn't translucent. Tried a bunch of things to see if I could get it working including reinstallation of patches, SIP On/off, etc.
Long story short, ended up reinstalling the OS since I didn't have a backup (note to self don't do that again). So I don't know how the GM of 10.13.4 performs on the 5770 but on 10.13.3, it was glitchy. I didn't experience any application glitches but I'd get red and blue redraw errors in Preview, Safari, and other apps. Wake up from sleep is also broken. Moving the window across the screen forced a redraw and fixed the glitches... but if you had to scroll, oftentimes the red/blue would come back.
I ended up just getting a GTX 680 for my 3,1 on eBay for around $200. So far, works well out of the box. Also works with Nvidia's drivers. If you want to squeeze more life into your 3,1 as others have said, I'd recommend going that route. The actual upgrade from 10.13.3 -> 10.13.4 worked fine here. Dare I say this machine works as well as it does on the officially supported El Cap.