To be honest, I tried putting Snow Leopard server from an original snow leopard DVD and it too is having some weird issues during the server setup. It was too late on Friday to start messing around, but my current theory is this particular Xserve maybe has a hardware problem of some sort. Perhaps it is related to the on-board networking, as there is no third part hardware connected. However, it had 10.5 on it previously with no issues. So I am going to look into it this week.
But I have also had bad experiences with Lion on my own 2010 mac mini server. The 320M graphics drivers are rubbish in Lion, and keep causing all sorts of weird graphics corruption particularly in Safari (doesn't happen in snow leopard!). I also cannot create a software raid set which I can install Lion on (works in SL), as the installer just hangs. I had the same issue on a 2010 Mac Pro with a hanging installer on a partitioned HDD.
And in fact, having re-installed Snow Leopard on the second hard drive in my mac mini it seems that the performance of SL could in fact be considered superior to Lion.
The only Mac I haven't had issues with Lion on is an Early 2008 MacBook Pro, although there was a nasty trackpad bug where you couldn't enable secondary click for this model - that was fixed in 10.7.1