Try out DriveDX to see if there's any problems with your HD, I remember having other Mac Mini's with poor quality drives, one with noise, one was pretty much like yours.
I had a terrible experience with Yosemite on my 2012 Dual Mac Mini, it was slow and laggy and frustrating, so I went back to Mavericks.
Maybe it's some bugs they need to fix or it needs an SSD to boost things up.
Can't wait to get an SSD in mine to be honest, I hate moving drives now.
On a bang for the buck basis the new Crucial MX500 looks like a great upgrade. ~$200 for 512GB.