The performance increase from Intel to Apple silicon was staggering. The performance increases within the M-class chips far less so. Far less than what Apple has been toting in all those synthetic scores they show annually. In the real world, M1 is very comparable to M4, especially when looking at raw core counts between the models (an M1 Max will still outperform an M4 Pro in the real world).
People seem convinced an M4 is light years ahead of the M1 because Apple creates that type of fervour to push sales.