I am curious to see if Apple either a) licenses Mac OS X Server to run directly on third party hardware,
Unlikely because this flys against there stated goals ( to deliver systems . not just software or not just hardware ).
or b) licenses it to run in VMware.
Do not need one. You can already run Mac OS X Server in Fusion ( so runs in one VMWare product).
You can run Mac OS X Server on raw hardware :
http://www.parallels.com/products/server/mac/baremetal/
just not from VMWare. It is purely a matter of VMWare ( and Citrix/Xensource ) deploying a hypervisor that doesn't allow Mac OS X Server guests to do live migration to other machines and validating the guest is about to start on a "Apple labled" box. for example if VMWare just got ESXi for Mac Pro and just didn't allow it to do the "advanced management" features from a Mac Pro.
Even without motion/live-migration a hardware hypervisor is useful. People often create cluster pods of similar virtualization boxes anyway. Apple would buy into a strict system that only allowed Apple-to-Apple migration would work for them also. There are limits to live migration anyway so this isn't a show stopper issue.
Part of the problem is a catch-22. VMWare isn't going to do a custom ESXi if there are no buyers. If not enough folks buy Mac Pro Servers then they may not ( unles perhaps Apple pays them for a custom ESXi embedded. ). The other half of the catch-22 is that folks aren't going to think to ask for a ESXi unless on Mac Pros if are fanatically fixated on the XServe.
the other problem is that most of the virtualization vendors give away just single box virtualization for free. ESXi and the entry level Xenserver are free. They change money only for the fancy stuff like live migration etc. Some of that is what Apple disallows. Perhaps a model where the virtualization vendor gets paid for every Mac Pro Server would solve that, but the fact that all of the commerical Linux distributions come with free (gratis ) KVM virtualization it is hard to charge money just for the basics. That is the catch 22 for Apple .... they can't leave out virtualization either. All the serious server OS have it too, for free (either in bundled hardware for AIX / PowerVM boxes or in the OS like Solaris Zones and Linux KVM ).
There remains the possibility of c) a revised Mac Pro that is more suitable for racking horizontally, I guess.
The handles on the Mac Pro make it about 2 inches taller. Minus 2 inches it would be 19". It wouldn't be hard to put the Mac Pro on a case diet and trade off perhaps a 2 inches from that 19" in height for 2 inches in depth. They could use exactly the same motherboard and most of the internals to do that (maybe sub in two smaller 700W supplies.) . Just like with the mini; small case mod ... ta da sever veresion.
And who knows what they have planned for that massive data center in North Carolina?
regular back end servers that other top 10 web data centers run.
Apple can run parts of their internal operations on Mac OS X Server. (servers for desktops , local DNS, Open Direcdtory ) But that doesn't mean you would want to run a web farm with them. There isn't really any real cost advantage in that since many of the other candidates (Linux , Solaris , etc) have been fined tuned (and apps tuned: java app servers ) to run that kind of load. Mac OS X Server isn't a big cluster ( computational , web ) OS.
Guess what HP, IBM, Oracle ... they've got Mac sprinked around several places inside the company. Nobody big runs 100% their own "dog food" . Microsoft maybe but that only because they only make half the equation (just the OS).