So when you say media server, what will be playing the videos, musics, pics, etc ?
AppleTV, other Macs, Xbox, etc.
I have been using a Mini as a media server since '09 and I primarily used it as a Plex server.
Now that I have AppleTV's on each TV I have since converted all of my files to mp4/m4a and iTunes handles all of my movies, tv shows and music.
I can easily have multiple TV's streaming at the same time without hiccups, I also have all of my aTV's hardwired CAT6.
My mini's starting in '10 have been headless and I will also suggest picking up an HDMI dongle if you get a unibody Mini. This makes the mini think a monitor is connected and if the GPU is supposed to do any graphics crunching it now works like it should.
Headless mini's, when remoting in, sometimes through up black screens or black spots and it took me a few years to figure out what was causing this, I'm not sure of the intricate details behind it but I know I enjoy a headless mini much more with a video dongle connected to it. The price of the dongle more than makes up for the irritation factor of not having one and I've had that same issue with a 2010, 2011 and 2012 min.
I know I mentioned hardwiring earlier and I am a believer in, just go run some ethernet to anything that will remain in one spot and needs internet. Most people wish they did instead of relying on WiFi. I have 2 802.11AC Airports bridged in my house and gigabit is still quicker. Nothing worse than having a lot of company over and you Que a movie up and for some reason at that particular moment you are all staring at a buffer/loading wheel for 3 minutes. It won't happen until there are people there
I so have OS X server installed but it is more of a want than a need. OS X is more than capable of providing content without the server app, just like Windows 7 can be a media server without having Windows Server 2012 R2 w/ update.
OS X Server does have some extra neat functionality it can add but that's pretty much it for most home users, if someone needs server they already know they need server.
But, like I said, I like all of my iDevices and what not to all get along so I use iTunes. Don't ask why but I am still a PlexPass "lifelong" member with it installed on my mini, I guess just in case I want to check out a new function it is already there.
Since I do use strictly iTunes I also researched and I picked up some good media converting apps and apps to add metadata to my videos.
iDentify is usually my goto for adding the "cover art" and basically all of the other info to a video file so that it appears as if you bought and downloaded that video from iTunes.... basically makes your video library look pretty.
iFlicks is another good app that does the same + does video conversion.
MKV2M4V is my goto super simple app that does what its name implies, it takes an mkv file and changes it to an MP4/M4V and quick.
Also install Perian on your machine which includes pretty much every video codec imaginable so that Quicktime can play any video file, you don't need multiple pieces of software to play a certain filetype. AVI, Divx, Xvid, etc.