It's actually not like that. You have to press and hold down for a while. Then you can swipe. Then you have to press again to dismiss. If you're old enough to need glasses for reading, there's the extra step of putting on glasses. This is an accessibility issue.
The reason this is an issue is that some folks use a bunch of complications. They don't all fit on one face, so we group them. I typically use 3 faces -- health, work, and workout. Being able to quickly swipe between the three makes my watch so much more useable.
I know that some folks don't care about this. I would be fine if it was a setting in the iPhone's watch app.