Getting back on topic, I've found the version of Safari in Mountain Lion to be excessively buggy. I was able to pretty much fix the IconServicesAgent problem in Mavericks by adding a few non-Mavericks volumes into the fstab file which prevents them from mounting then adding a script to kill IconServicesAgent and allow it to restart.
It seems to me IconServicesAgent makes whatever errors it's making at boot time when it apparently goes out and interrogates all OSes on your drive (I have a multi-boot system). After that's done and the "bad" disks are unmounted and it's allowed to restart, memory use is way down in Mavericks.
I may switch back to Mavericks just because Safari has a few too many kinks in it on ML.
Like Yosemite or not, Icon Services Agent problems in it and El Capitan are (or have been) fixed. A lot of bugs are fixed in Yosemite, but IMHO it still looks stupid. There a subtle 3D effects in El Capitan that, at least to me, make a huge difference. That and the font changes. I can actually see using El Capitan (assuming it's reasonably bug free). I'm hoping El Capitan will be to Yosemite as Snow Leopard was to Leopard.