It's huge to the experience. I've been using responsive Apple products for the majority of my tech life, and I couldn't imagine going to something with noticeably more latency. It would never be the sole reason, but it's definitely one of them.
Over a year ago, I compared an Android device side-by-side with an iOS device. It was my HTC Desire Z and my father-in-law's iPhone 4. I definitely noticed that scrolling was smother on the iPhone 4. Now, my SGS3 has closed the gap and it is only running ICS. My friend's Galaxy Nexus had JB and was a bit smoother.
If you are used to the iPhones, that extra bit of latency may be a turn off. I've gone the other way with smoother and smoother devices from a Nokia N97 to my SGS3, so any latency you may notice certainly doesn't bother me. The extra smoothness of the iPhone would not be enough to make me want an iPhone considering I would have to give up a lot more for what seems to me to be a very minor improvement.
I want an OS that I can feel connected to. Over the years, iOS just seems there. It gets the job done but I don't feel that "connection".
I think the best example in my case would be texting. Android has swype(I believe its called) that makes texting seem so streamlined and smooth, while with iOS, we've pretty much had the same bland experience with the idiotic autocorrect.
If you're an Android user, I love to hear your input on this. I've been considering jumping the fence.
I use an alternate keyboard for my phone. I personally like Thumb Keyboard over the Swype keyboards. I personally do not like auto-correct or text prediction. I find that it bothers me when I type something (usually an acronym I make up on the fly) with prediction and the phone is totally wrong on what it thinks I mean to type. It is expecially bothersome with Swype.
There are a number of alternate keyboards available for Android. Thumb even has different keyboard layout to choose from. I choose a split keyboard in landscape mode because I have such a large phone (SGS3). It makes reaching letters in the middle of the keyboard easier. Also, for the portrait keyboard, I use the split/stacked keyboard. The left-side keys are above the right side keys so that each individual key is wider. I have trouble with portrait keyboards because they are so narrow. There are several different landscape and portrait layout available to choose from. You can also scale them to be smaller and sometimes higher.
=====
On another note, I don't think that iOS is necessarily significantly less buggy than Android. From my experience, some iOS devices do slow down over time and have issues such as crashing, just like Android devices. iOS doesn't always just work just like Android.
One significant difference between iOS and Android is how each OS handles a crash of some sort. When an iOS app crashes or force closes, iOS does its best to hide the fact that there was a problem with the app. You just go back to the home screen just as if you pressed the home key. It makes some users think that it is the user who did something wrong and that it wasn't a problem with the app.
Android on the other hand will actually display a message saying that there was a problem with the app. Then the user gets the opportunity to add more information on what happened prior to the app crashing. The user can then send this information as well as other system info to the developer, so that the developer has some information to perhaps update the app to fix the issue.