I would normally agree, but the Note 4 opens apps so fast (especially with animations disabled) that having 32 vs 64 bit doesn't make a big difference. iPhone 64 bit apps... From real estate to google earth to photo editing, does 64 bit really make a noticeable difference? I think games might have one but I don't play games much on my Note.
More of a future proofing thing and taking advantage of the native 64bit android L. The potential depends on the application as well. 32bit is disappointing. If I get one it will be the Xenos version.