I'm gonna be really curious to see what you think coming from the 12.9. I upgraded from 12.9 to the new 12.9, and am really happy with the decision, but still questioning myself sometimes as to whether or not I should go smaller strictly for the portability factor. I just don't know if I could give up my screen at this point. I'm planning to head to the Apple Store Monday or Tuesday to check out the 10.5 and see what I think while I'm still in the return window with AT&T.
Honestly, I had this same struggle when I bought my 12.9 (9.7 pros had just been released), and ultimately decided to go big or go home. I LOVE the 12.9, but I know I'd like something a little lighter and easier to haul around as well. I'd consider just doing a 2017 iPad for hauling around, but I know that would be really tough to use out and about when I have this amazingly smooth pad at home. First world probs...
You play games and watch lots of videos? Stick with the 12.9, I tried the 10.5 coming from the 12.9 and loved the size and the way it was easy to handle BUT if what I mentioned is what you do lots of on your iPad then just stay with the big one. I am not keeping the 10.5