I've installed dosdude's 10.5.2 and although I haven't modified AppleGVA.framework, H264 HWAccel seems activated, at least in VideoProc. H264 conversion in other apps, like Compressor, seems to work fine although HWAccel is minimal, which is expected anyway. It is only in hanbrake that any attempt to convert to H264 fails.I am not sure, but H264 HWAccel should be OK, Apple only removed the HEVC HWAccel for Polaris in 10.15.2
If you use Hex edit method, please restore the edited files, and try latest Lilu and WEG.