It's called 64bit dude, welcome to technology. 32bit apps can't access anymore than 3-4GB of memory before it starts paging out and doing a larger swap on your HDD, whereas a 64bit can access something like 16 TERABYTES of RAM before needing to do that.
SO, if you've got 12GB in your machine, Aperture will happily just keep using it up. In all honesty, it's not using up anymore memory than it did before, it's just not having to page in/page out to the HDD to go over 4GB.
I've noticed that my available memory is lower than normal, but my swap hasn't been over 64MB since I got Aperture 3.
Be happy it's a memory 'hog', it's keeping your computer running faster!
PS Why on earth would you reboot it into 32bit mode unless your plugins require it? That's just retarded.