No, it's not hard to replace the HDD with an SSD provided you have the right tools. The main requirement are the 2 Torx screwdrivers. There are lots of videos showing you how to do this. Check out:
https://www.ifixit.com/Guide/Installing+Mac+Mini+Mid+2011+Dual+Hard+Drive+Kit/6634
https://www.youtube.com/watch?annot...&feature=iv&src_vid=13wcKCYo_Xw&v=cvmuQIZPFMk
https://www.youtube.com/watch?v=gBongunj4pc
I recommend watching the videos through to the end a couple of times so you understand what to expect before you start work. Get some small containers for all the screws, and use a camera to take photos to remember where things went before you disassemble!
It should take no more than 15-30 minutes. If you have a single drive Mini you might want to consider adding a second drive (there is space inside the case) by getting the additional SATA cable (the Server version of the Mini already has the cables included).
My 2011 quad-core mini now has 2 SSDs and is pretty snappy.
John