Even just for "cruising around" a bigger engine can have benefits.
More cylinders usually idle smoother. There are certain designs that are inherently more balanced and smoother running than others(some Italian makes, for example, have been known to use flat plane crank V8s vs. the typical street use cross plane as they rev better, but idle worse), but that's just a general rule.
Not only that, but GENERALLY a more powerful engine can be geared such that it's running at lower RPMs when cruising at speed, and that gives an overall more comfortable/enjoyable ride.
A lot of good old American V8s had nice, flat torque curves from 2K to 4K rpms or so, which didn't necessarily give great performance but let them cruise without feeling strained. That's part of what makes something like a Town Car so comfortable in even at 70-80mph-the engine is running at 2K or so and you just don't feel it.
I get the feeling that you're trying to justify the fact that you have a base model RX350 or something like that(not that that's a bad vehicle by any means, but let's be realistic about where it sits in the market) and want to continue to pretend to be a high roller.