Okay, got my Xserve. The large PCI slot is actually exactly 10 inches to the memory. It will probably hold a 10" card, but you probably wouldn't be able to pop open the clips to remove the memory without removing the card. It seems that it would hold a somewhat larger card and probably would hold a WX 7100 Pro. I ended up ordering a PowerColor RX 590. Waiting for that to come up.
But the next question for any of the people who have done video mods on these boxes, what do you do about power? My Xserve has two 750 Watt power supplies. This should be plenty to power a 250 watt video card especially when each PSU is putting out 62.5 AMP's for 12 V. And it supposedly load balances the power between power supplies. It looks like these power supplies are single rail, as it seem there is only one 12V output. The sticker on the PSU says 12V @ 62.5 AMP output, so I would assume it is a single rail which would be ideal. I was looking at the system trying to figure out where to pick up 12V power, there is 'jumper', a 14 wire cable going from the power distribution board to the board to the RAID controller. I took some voltage measurements, but there are 3 12V wires off of it. I was considering splicing into those three wires and combine then together to use to power 6 and 8 pin connectors for a video card. However these are not larger gauge wire, so I am not sure if this could overload the wire or the connector on the power distribution board. It seems silly to use an external power supply for a video card when I have two much higher quality and more powerful power supplies built into the box which are redundant.
I know a 6 pin GPU power cable has three 12V lines and I believe if you had multiple GPU power cables which are normally on different rails in an ATX power supply. Or they are the same 12 v circuit with current overload protection because this is part of the ATX spec. I guess i could in theory solider wires to a power distribution board and pul power off there. Not wanting yet to ruin mine, some of them are similarly priced on ebay as to buying a new supply, but I found some 2008 ones for $10. They look the same, have a different part number, not sure if they will work or not as a test to try to pull power off the internal power supplies for a graphics card. Anyone have thoughts on this?