ok..... understand that i am coming at this answer from the standpoint of someone who teaches undergraduate- and graduate-level graphic design and who is a graphic designer as a (much more than) full-time profession.
if you just want to be able to make better icons for a website, well, i guess a photoshop/illustrator class is worth your time. if you want to make better looking websites, then maybe a UI class is worth your time. in other words, if you are looking to improve a specific skill then a class/mentoring/internship/whatever in that specific skill set will not hurt.
it sounds like you understand that this is a very limited approach to learning how to be a better designer in general. with that in mind i would say you need to start from the basics of form, color, type, layout, hierarchy, meaning, etc.. and build up the background before you can start to make really good work. you can do this by doing some stuff online, by reading some specific books, etc.. but ultimately the best way to really learn how to be a critical visual thinker and communicator is to be in a room with a bunch of other people trying to do the same thing and having an involved dialog about the work. the easiest way to do this (tho not cheapest) is to take classes, and specifically is to take a number of classes that use a sequential methodology to teaching you graphic design. this will take time and depending on your situation, money.
having said that there are ways of doing some learning on your own time (books, etc..) doing exercises and getting critique and dialog on them via some web forums; none of which i personally have found are worth the time, but then again i went to a design school so i have had the group critique experience.
good luck!