Precisely. The worries appear to be based around the arguments like "but they used to boast with 20% better performance in the past!", and while I sympathise with those posters, these are not good arguments. Times change, as do design goals.
For example, Apple has very little incentive to make the new phone CPU 20% faster (they are already years ahead of the competition), but they have incentive to make Mac chips faster. Maybe this new CPU is designed with higher power draw and higher clocks in mind (unlike A14/A15 that pretty much peaked at 5W per core max). Then Apple would need to clock it more conservatively in the iPhone, so that they can meet their power targets.
Of course, what I wrote above is just a speculation, and might as well be baseless, but I hope it illustrates that there can be plenty explanations for only modest performance improvements in A17, not all of them equally gloomy. We will know more when we have more data bout the chip (performance, clocks, features, etc.). Then we will have at least something to speculate about Apple's inevitable impending doom.