If an icon in the dock with a badge is distracting or taking up too much space, then I would suggest the problem is with you, not the OS/app.
Well that's helpful is it?
There are other ways (using
ifttt.com comes to mind).
My first thought was IFTTT too, I browsed the site for a while but I've never actually used any of them before. From what I understand, it doesn't seem to have support with any OS X notification systems, unlike Android. It might be my lack of experience with IFTTT but I didn't see anything that would seem to serve this purpose. And – is it ultimately a mobile app?
I thought of Growl too but while it still exists, it's been a very long time since I've seen any mentions of it or recent applications.
And Bonjour, but I have never used it either so I don't know if it's a system that can be harnessed for this sort of thing. Triggering a push notification to yourself seems like such a simple thought, but it turns out to be surprisingly elusive!
Safari is highly likely to be open, so even getting it triggered via Safari would be acceptable.
For your purposes I'd drag Mail off the dock. I would expect that Cmd-Tab access would still be quick enough. You can also shrink the dock to minimum height, and/or turn on dock hiding.
Hiding unfortunately isn't an option, as I can't have elements blinking on and off...
My dock layout relies on icons that contain text, so they don't take the arbitrary amount of scaling too well.
Okay I tested now, using Dock Dodger. When I turn Mail into a dockless app, then relaunch it, Mail quits unexpectedly. If I try to launch it again, I don't even see its icon briefly flash in the dock as on the first time, it just seems like nothing happens from there on. If I restore its dock icon, its functionality gets restored to normal as well. Not sure why it doesn't work with Mail, Dock Dodger has worked with other apps I've used it for, although they have been third party apps, I think. As I think Mail fails to launch itself as a dockless app at all, that must be the reason why I can't see it in the cmd+tab view.
Can always setup filters on the Gmail side of things. That is how I've setup my environment. Most emails get sent to some other folder/label via a filter (and possibly marked as "Read"), and only the most important stuff gets to the inbox and alerts.
These seem to be all things that work when Mail is open. I tested Mail's rules by checking what it does if you set a filter so you only get a desktop notification if you get mail from a specific address. It's unseen until I launch Mail, but now that's exactly what I do anyway: launch Mail to see if the new unread message that I've been expecting has appeared to the top.