The 4 had a single core processor. The 5 has a better, faster, dual core processor. That alone should make quite a difference from the 4 to 5. Even more so if they're on a network which supports LTE.
I think when they said "honestly there isn't any difference", I think they are referring to the fact that the iPhone 4 can do 97% of the things the iPhone 5 can, maybe with the exception of things like Siri, 3D maps & panorama. Most people don't care abt that few seconds difference in speed, they care about what the phone can do for them.