Coming from an M1 MacBook Pro, or even an iPad, both of these machines are slow in many aspects* . With many applications often using single core, the old Xeon systems benchmarking mid 300s to 500s in Geekbench as compared to a MacBook M1 in the 2,000s, both machines can feel slow / throttled when using basic applications.
*As a collector I fully understand there is far more to these systems than simple benchmarks!
When you start to use applications that are Metal + SMP aware, the systems start to come into their own. When you start to fill them with large drives, PCIE cards, and turn on (or install) server services, these will do far more (and more attractively) than a newer Mac mini or laptop can within their own singular case. I honestly think of my 5,1 and 3,1 as being tractors of the house. They are stable, reliable, built well, and in some cases can be a bit slow but will always get the job done.
While I love my 3,1 machine, the more future forward SSE instructions, ability to support newer GPU models, and Hyper-threading really make the 5,1 a wise choice if you can only have one. I do feel fortunate to have both, having acquired my 4,1 (flashed 5,1) 8 years ago and was given an 8 core 3,1 last fall.
The 3,1 is a bit more media consumption but is also a backup to my 5,1 whenever I am in an upgrade cycle. Both computers are used (in turn) to run 6 - 7 server services I use in my home and away.