One of the reasons I ditched Android a long time ago, and it seems that it hasnt change at all. Its just a bloated, sluggish OS. No real way around it. I have never had a slow iPhone, its always as responsive as the day I get it.
Some will say they have never had this particular problem with Android, but I am not sure I believe them.
This is so dependent on WHICH Android phone someone has.
If we're talking about a Samsung, then yes, bloat is definitely an issue and does slow things down (and in turn, cause the stuttering and lag some people complain about). While I've got the S4 (and the Note 2, S3, and S2 before that), TouchWiz is just awful. The first thing I do with a Samsung phone is root and install a stock, debloated TouchWiz ROM. It's amazing what a difference it makes. And that doesn't even include all the bloat that the carriers add. They just make it worse by putting crap on there that's just a duplicate of stock functions (AT&T's navigation app and Smart Wifi come immediately to mind).
Some will say that LG is basically cloning Samsung phones, so the same bloat and lag exist, but my experience is that LG phones (starting with the Optimus G) are fast as lightning and have zero lag. I'm not sure what LG figured out that Samsung hasn't, but their "skin", while looking a lot like TouchWiz, sure doesn't function like it. I really like what I'm seeing out of LG over the last year.
The HTC One, while very nice, did have some noticeable lag present. Sense 5 is very toned down from previous versions, but again, it's resource-heavy and again, once the carriers put their crap on there, you get lag. As soon as I rooted and unlocked it and put a stock Android ROM on there...the lag was gone. But I couldn't deal with the crap camera. It was just awful.
And you'll hear people talk about the "Nexus experience". I can tell you, it DOES make a difference. While you may not have all the bells and whistles of a Samsung phone (honestly, who uses all that stuff anyway), the trade off is that you get a fast, lag free experience. I got to play around with the Moto X finally, and man...that thing is a joy to use. It's so fast...and I couldn't make it stutter or lag even when trying. Removing all that bloat and focusing on the user experience really makes a difference.
The only reason I passed on the Nexus 4 (and opted for the Optimus G instead) was because of the lack of LTE. Since the next Nexus will have it, I'm going all-in on the Nexus 5. I really think it'll be the perfect Android phone (other than the lack of SD support...but I'm slowly weaning myself off of it).