It really makes me wonder if I didn't make a (huge) mistake not waiting for the 64GB at this point.
Have you recently watched a film from iTunes? Maybe it is still on the SSD as it downloads temporarily and if there's space, maybe it stays on the SSD for a few days.
I think I can explain this based off of my experience.
A few months ago, there was a thread saying the 64GB ATV was pointless because you would never be able to fill up a 32GB ATV.
While I do not have a 32GB ATV, I wanted to test this theory using my 64GB ATV4.
I already had a lot of apps save on the 64GB, and the "Storage" app I had on there said there was only 15ish GBs left on the disk.
I then downloaded every free game I could, and I noticed that the available space got smaller and smaller, all the way down to 900MB, but then the system would free space, which I assumed was cached videos, app data, and maybe the aerial screen savers.
So, I kept on downloading, and every time I would come close to running out of space, the system would free up more space.
You might think this means that the 32GB would be okay then, and the 64Gb would not be worth it, but I have noticed that after downloading a lot of apps, if I use a lot of different apps in a short period of time, the ATV4 slows down to a crawl.
I assume this would be the case for a 32GB, just much worse. Actually, I probably have too many apps on my 64GB on the 32GB.
NOW, all that said, if you are using the 32GB under normal use, and not what I was doing, you will probably be fine. If you have a crap load of apps saved and you notice a slowdown after using a bunch of different ones, then I would delete a few.
BTW, I don't think that restarts removes the cached data.
My first iPhone was 16GB and I fought with it every day for 4 years! Always having to remove playlists, apps, photos - I never want to do that again
I felt the same way, I had an 8GB on the oringal iPhone, and that didn't seem like too little, but after that I have a 16GB 4s, and then a 16GB 5s, and dealing with storage issues was a constant battle.
Thinking just like you, I then got a 128GB 6s Plus, and now a 128GB SE and I no longer have storage issues.