Number of vulnerabilities is a very bad metric. You can have 50 different DoS type vulnerabilities that just result in a crash no matter what the attacker tries, and you can have 1 very bad vulnerability that can cause code execution. The fact is Internet Explorer has had a very bad run of code execution vulnerabilities where simply navigating to malicious sites result in installation of malware on your computer.
Also other factors are time between exploit and patches. Microsoft has a very bad sheet here where some vulnerabilities with exploits in the wild have been left unpatched for many months. The release cycles have been long, leaving users open to known attacks.
Then there's the most important factor : Standards compliance. Let's face it, anyone still using Internet Explorer at this point is personally responsible for holding back the web and preventing users of alternative platforms from using it. Microsoft's goal with Internet Explorer until about version 7 was basically to close down the web to their platform and thus cement their monopoly over desktop technologies. Poor respect for standards, a lot of extensions and especially dev tools that promoted the use of these non-standard extensions.
The web was designed as a platform agnostic medium. Everyone should participate in making sure that no corporations gains controls over displaying web content, be it Adobe through the use of proprietary plugins or Microsoft through the use of non-standard technologies for formatting. Some might argue that this just isn't important, but with the growing number of devices that don't have access to Trident (Internet Explorer's rendering engine) and the proliferation of mobile web devices, this is becoming even more important. Microsoft needs to learn to play ball or go home, not the opposite.
Oh and I guess performance too, if that matters to you. Internet Explorer gets dominated here. Internet Explorer 9 you say ? Let's look at that when it actually ships vs all other things in development...
I use Chrome these days, a return to my KHTML roots (yes kids, I used WebKit before it was the cool thing, back when it wasn't even a glimmer in Apple's eye).