The speed will likely be acceptable, but the latency will be more noticeable. Again, are you doing this because you're worried about causing wear and tear on your SSD? If so, please read that thread I mentioned above. Apple's SSDs are very robust.
I'm a developer and a photographer. I've pushed the original 8GB M1 MBP way past any reasonable expectations and it performed very well. It performed well because the way the swap was handled was excellent.
Even though I was under severe memory pressure almost constantly, there was hardly any real world impact. I don't think you would be able to say the same if you had the swap go to an external drive.