I have the Note4 and Android 5.1.1, the fact that you noticed lag is not necessary related to Android but it can be the apps that you're using. I noticed lag on my phone too and did some reading. Some folks did experiments with the Facebook and messenger app. When you disable those 2 apps, the phone is noticeably faster. If you install them (but don't open them) the phone gets slower. Switching from camera to the photo apps with Facebook installed created a delay. I've heard there was a similar issue on iOS. From that point I did a shortcut to the Facebook web page and I left the apps disabled.
I noticed issues with other apps too. Lately, I also noticed issues with the Plex app. Even without using the app, it was draining my battery like crazy. I had to FORCE STOP the application each time I use it. I also experience the same thing with MIPC.
Each time I'm done using my phone I bring all the tasks and kill them.
It's really annoying to have to do micro-management like this.
What I dislike about my current phone and the reason I may switch back to iPhone is the time it takes to get things done.
Simple task: send a picture
Real actions: open camera, wait, take a picture, click to open photos, 2 sec wait, at that point I see the photo but If I try to send it, it will says it's too big. So I click to EDIT the picture and click SAVE, even without doing anything, the file gets smaller and can then share it with someone. If I choose to really edit the picture, like cropping or fine tune colours, then when I save, It gives me a duplicate picture (which get save in the Studio folder) because it doesn't want to overwrite the original. Now I send the new picture but I'm left with duplicates, that I need to manually delete. If you got photo backup turned on, the duplicates pictures are sent. It's a real mess.
On iOS, it's just take picture & send.
Simple task: read emails
Real actions : I need to have 2 apps, one for GMAIL,yahoo,msn the other one for Exchange.
I hope android 6 will fix the issues I'm having. If yes, I will probably get a S7EDGE (or Note6). Otherwise it will be an iPhone.