I really don't think that much horsepower is needed for a media server. Like I said, I don't think I've ever come across a video that a Raspberry Pi can't handle.
Lots of horsepower is only really necessary if you need to transcode stuff to be played over WAN.
For most home media servers, they're basically file servers. Extreme CPU oomph isn't really needed unless you're doing your transcoding on it as well, and even then, a Mini can handle that like a champ. My base 2012 with a 2.5 i5 chews through encoding with no issues.