I'm not sure about the HTC One... but nearly a month ago my younger brother bought the GS4 when it was time for an upgrade, and he returned it 6 days later because he was sick of the lag. He liked almost everything about it, but the morning of his graduation he drove to the at&t store to exchange. So the GS4 does have some lag. It definitely wasn't as bad as my Nexus 7, though.
I have a GS4 as well and they have thus sent out a patch to fix whatever lag there was. I was fortunate enough to buy mine with the patch already installed and never experienced any lag. Trust me, it's a non issue with the S4 and I'm running mines stock with just a few cosmetic mods but nothing on the performance side.