Finally got an Xserve3,1, some SAS trays, and 6TB drives to go with it - specifically HGST HUS726060ALS640 6TB drives.\
They're SAS-3 (6 Gb/s). I did the 3.3v mask bit and they show up, but only detect as 2.2TB as SATA drives would.
I'm unsure if the tape mod is required for SAS-3 drives - does applying it break compatibility? My RAID controller has a failed battery, but that shouldn't affect drive capacity. The Xserve is a dual 2.26 model running El Capitan currently - attempts to patch it to High Sierra failed after install as if the OS itself hadn't been patched.