So, I thought long and hard about the HTC One vs the S4 when I switched carriers a few weeks ago, and ended up going for the S4. Aesthetically it's less appealing, but functionally I'm pretty convinced it's a better phone (at least for me).
The replaceable battery and SD card slot are nice, but in practice they weren't that significant. With Google removing support for putting things on an SD card my 32G card is almost empty. I use Google Play Music a lot, and I can't get music from it onto the card (if I could I'm sure I'd be using at least 10G, but as it stands I just stream a lot). I don't watch video on my phone (Nexus 7 is much better for that), so I don't load video onto the card. T-Mobile doesn't have the update to move apps to the SD card, so I can't do that yet (and I'm not sure I will, since I don't have a lot of apps I use infrequently I'd be willing to take the speed hit for using SD storage instead of internal flash on). YMMV, but for me the SD card is something I just don't use a lot. Battery life is good enough that I don't have to worry about swapping batteries, and ii'll probably be a LONG time until I have to worry about battery degradation.
The S4 felt a lot more comfortable in my hand, though. Keep in mind I preferred using a bumper on y iPhone 4S because the hard edges bothered be there as well, but I overall didn't like the feel of the HTC One. The One was solid and well built, but as a device I was going to be interacting with a lot the S4 was more comfortable.
I don't use a case on any of my phones, and although I'm not really worried about the Gorilla Glass 3 scratching I was a little concerned about the back of the HTC One showing wear. With the S4 I can just replace the entire back if it becomes unsightly, which is a win in my book.
The S4 also supports wireless charging with the optional battery back, which I've really learned to appreciate after my Pre2/Nexus4/new Nexus 7. It's a little hard for me to consider a device that doesn't have it, really. If you order direct from Samsung and use the 50% promo code going around the new back will cost you about $20, and although it makes the phone a little thicker the more rounded shape and weight balance is a little better, IMO. It also adds a slight gap between the plane of the back and the lens of the camera, so if I you set the phone down on it's back you can't scratch the camera lens on a table.
Software wise I'm more confident in Samsung continuing to update their devices than HTC. Although 4.1-4.3 are all fairly similar when something significant comes down the pipe there's a better chance the S4 will get it than the HTC One based on previous Samsung vs. HTC update timing.. Take that with a grain of salt, though.
Most of the S4 software features are gimmicky. There are a couple that are genuinely useful, though. I'd rather spend 10 minutes trying out a dozen features and finding two that really help me out than not having the option, though. Smart Stay and (surprisingly) S Voice I use on a daily basis. Again, YMMV.