Ok so the consensus is: You won't really notice anything but it's nice to have 4GB, because there is paging that could cause lags (even though it's not noticable-mostly)
I bought the 2GB. I really thought about this, but having come from a 2GB Air I never noticed anything except how much everyone talked about wanting 4GB, which made me feel like I maybe should have 4GB, even though........whatever