IMO, the two real reasons to seriously consider a Note over a different, not-quite-as-big Android phone are:
1. You want the stylus integration and/or
2. You want a mini-tablet instead of a real pocket phone.
If those are features/benefits you care about, especially #1, then the Note is a no-brainer for your shortlist, and probably worth taking a chance on those two points alone.
I suspect most people don't care too much about the pen (though it's the main reason I bought one). I also suspect a lot of people are of the "I think I can deal with the size" mindset instead of the "bring on the phablet!" one. These are the people that I'd caution not to get swept up in the novelty of the device.
Any phone is going to annoy you at some point for something. With this phone, you'll be able to console yourself with, "This is what I needed to get a real, actual pen" and "This is big enough to make surfing/email/twitter/reading feel more like I'm on a tablet than I'm on a phone." If those aren't compelling statements for you, and aren't going to remind you what's great about your phone, then this might not be the phone for you.