Same boat. I've had iPhones since the 3G and apart from the screen size I was quite happy.
It really depends on what you want to use your phone for.
The S4 is good, but the main selling point is the screen. Ignore all the features Samsung are trying to push because they suck.
Ultimately it will depend on whether you can live with Android. It's a flexible OS and will feel fresh at first. At times I feel like it's a level above iOS in what it can do and what it offers the user. However using it does mean either abandoning iCloud and all it offers or finding workarounds which can be a bit of a headache.
Apps wise, it will be a big change too. Generally the Play store isn't nearly as bad as some would have you believe, but generally iOS does offer more and shared apps seem to be updated first on there.
Although iOS feels stale, it gets a lot right. You just set it up and go, it takes 10 minutes at most. A week later I'm still trying to refine my Android experience to a point where it's acceptable.
My carriers given me another week to decide whether I want to keep the S4. I'm about 50:50 at the moment.
I am in EXACTLY the same boat as you are...iCloud, Siri and my app library are the only things holding me back from switching to a GS4 or a HTC one right now...
I'm waiting for WWDC '13 to see what iOS7 will be like... If it is ANYTHING like the iOS5 to iOS6 update, I'll be switching regardless of what I'll have to leave behind...