Well my opinion isn't the one that counts but I think the 42mm looks great. If I were in your position, I'd think about screen size-- as in how much screen you want to view. Do you ever wish you could see more screen in your 42? If you're content with how much content you can view on your 42, then maybe the 40 will be a better choice.
Something else to think about is that your current bands for 42 will fit a 44. If you go to a 40, you'll need to get new ones.
Not trying to sway your opinion one way or another but hopefully giving you things to think about
Let us know what you end up going with and I'd love to read feedback on how you like the Series 4. I've really, really enjoyed Series 4 and even coming from Series 2 I'd consider it a significant and worthwhile upgrade.
Here are a few shots of my 42 and 44 side by side just so you can get an idea of how much difference there is in the size of screen.