It's actually pretty easy.
Turn off Mac App Store auto-updates so it doesn't undo the downgrade.
Download the standalone iTunes 11.4 installer from Apple. iTunes 12 beta is just for Yosemite, so the regular iTunes 11.4 is still the current release.
Move your current iTunes folder out of the Music folder, otherwise anything you do will be automagically undone by the OS (clever, but annoying). I put mine on the desktop.
Delete iTunes. Not sure you can just trash it - I used AppZapper. You can probably do a sudo-something from Terminal too.
Install iTunes 11.4 from your downloaded installer.
The tricky bit is next - go into Time Machine and restore your old, pre-Yosemite iTunes music folder. I have six months of Time Machine back-up available so I looked for the last folder I had before I upgraded from Mavericks to Yosemite, as I knew this was the last iTunes 11.4 folder I would have had.
Once that library is restored, if you open iTunes 11.4 it'll recognise it and you'll be back to where you were, pre-Yosemite.
If you leave the new iTunes folder where it is, iTunes 12 beta will reappear, overwriting iTunes 11.4, no matter how many times you try and run the 11.4 installer. It's pretty neat - some kind of self-repairing install - but damn annoying. Took me a few goes to figure out what was going on, but moving the iTunes 12 beta/Yosemite folder out of Music did the trick.
Now I have Yosemite beta 3 running with iTunes 11.4.