It looks like you have to buy pairs. Which would mean you have 2x2GB right now. So those will have to be tossed out.
http://eshop.macsales.com/shop/memory/iMac/DDR3_21.5_27
That's the cheapest I found but I also didn't look terribly hard. Assuming that's your iMac, it'll run you about $417. They do buy old RAM though so it looks like if you send it to them they'll give you $50 back. Not sure how much your 2x2GB would go for on the market though so you might consider eBaying that.