Flash is terribly inefficient on OS X, even some of the Intel macs newer than your G5 peg the CPU trying to run Flash. Someone said force Youtube to HTML5. I didn't know you could do that, but if you can, I get it would help a lot.
For most applications, additional RAM will either do nothing at all or will help tremendously. If you are actually running out of RAM, then it will help a lot. If you aren't currently running out of RAM, then it will not speed up your computer by even a tiny fraction. So before you bother to spend money there, watch your memory usage.
I don't think updating your G5's video card will help Youtube at all. Also the best video card you can get for it is really old and outdated.
For the money you would spend on video and memory upgrades you might be better off selling the G5+monitor, adding your upgrade money, and get a blazing fast new iMac.