I have Swift code that contains things that only run on ARM. It's not magic. Swift does not run in a virtual machine. It's not Java's compile once, run anywhere. It's the same as Obj-C, even sits on top of the Obj-C runtime..
I miss that, objective-c binaries don't run on top some runtime (as does visual basic), it link its executable binary to objective-c binaries or libraries as you like to name.
Exactly the same on swift once you compile some library for ARM your binary it's arm specific, once you compile the same library (as my Delaunay triangulation) for x86 you don't rewritten your library you only selected an different output from Swift compiler.
I suggest you to read on heterogeneous systems architecture, you desperately needs it's concepts.
A final comment, supporting an new architecture (cpu/gpu) while now it's easier than ever always has some cost, even if some app only need a simple re-compilation it needs new testing and distribution efforts which adds costs, most software vendors when on this, also use these events as justification for some improvements (and to bill it's customers), but isn't something as dramatic as to claim it's the world's ends.
Also we see it more often than you think, Intel is deploying AVX 2.0 by first on the Mac Pro, it's really improves vector related stuff by an order, you think implement AVX 2 will have no user costs? (involves most cases just a compiler switch) no way, you'll pay for it somehow.