Could be a host of things. How many tabs are open? What sites are up? Any add-ons/extensions? How many, and do they have the potential to eat up memory? How big is your history file (eg. never have cleared it)?
For a baseline, I use Firefox maybe twice a week (I only use it for certain sites, ditto Chrome, bulk is in Safari), so, small history, 3 extensions, no add-ons that run automatically (all are set to "Ask"). With a blank tab, Firefox starts at about 360MB of memory used, for me. Set the tab to an iHeartRadio station, and a second tab on one of my e-mail account's web interface, usage goes up to 580MB. And don't forget, that each tab will consume some additional memory that is not showing in the screen grab provided: each tab has a separate process that is eating memory (in my example, an additional 60MB and 170.7MB).
So, that said: clear cookies, cache, history. Maybe cut down number of open tabs (especially to sites that are updating itself on a regular basis). Get rid of some extensions/add-ons.