I have both currently, and I actively use both on a daily basis (also had almost every previous iPhone, as well as all of the previous Galaxy Notes as well).
I'm literally split down the middle on my preferences between them.
On occasion, my iPhone 6+ will "lag" when rotating from portrait to landscape and vice versa, and other times it will rotate far too quickly for my own preferences. I haven't correlated this with whatever else the phone may be doing at that point in time (ie, iTunes updating apps in the background, for example).
My Note 4 will occasionally lag when switching screens ONLY when first waking up. Once it's "awake" and I've flipped across a couple of my home screens, I've never experienced any lag with my Note 4. And it's bone stock (Touchwiz and all...), with several running widgets (Touchdown, HD Widgets, Pure Grid calendar widget, GMail, Flipboard, CNN, BBC, and PlayerPro).
So in my experience, lag isn't an issue on the Note 4, and I'll readily admit there was noticeable lag on the Note / 2 / 3 (I finally switched to Nova launcher to alleviate some of the lag).
I have a pretty significant investment in the Apple ecosystem (several movies, tons of music, all in iTunes), so the ability to sync via iTunes was critical. iSyncR takes care of the music aspect of this on my Note 4, though the iPhone 6+ is pretty much seamless. That being said, the big advantage for the Note 4 is the ability to load content onto the phone without having to go through iTunes.
Video playback on both is pretty good....no clear choice between them for me. I don't play a ton of games (Solitaire and chess mostly), so I'm not stressing out either one very much.
I do like the iMessages integration into my rMBP, and the ecosystem around Apple is pretty hard to argue with.
That being said, there are some apps / tools for my Note 4 that I can't get anywhere else (WiFi Analyzer for one). I like being able to walk around my house, and I can check WiFi signal and interference at any point in my house with my Note 4. The iPhone is useless for this, since Apple won't allow 3rd party apps to access low-level hardware.
Several friends have asked me which I'd choose if I had to pick only one....and I've honestly said that right now I couldn't choose. If Apple launches ApplePay in Canada, that might push me clearly in that direction, but until then there is no advantage.
I agree on the physical aspects of the phones....I think the iPhone is a bit better looking, but I'm also highly reluctant to use it naked. The Note 4 on the other hand, I just toss into my suit jacket without thinking twice about it.
Quick off-topic recommendation...the best case that I've found for the iPhone is the SurfacePad from Twelvesouth. Enough protection to at least protect the phone from most things, without any significant bulk to it. I don't work for Twelvesouth...just happen to love their products.
Ironically, one of the best "cases" I ever had was the original Galaxy Note Flip Cover. I despise the new ones...but the original was as good as they got. Not sure why Samsung screwed that up so badly.
Patrick