How much memory do you have in that iMac?
Is this a 27 or 21.5-inch screen?
I think you have control over the storage drive that you use, and how much memory is installed.
So, the answer to your question mostly depends on the tasks that you normally do, and the specific apps that you use to do those tasks. After an SSD, upgrading the memory will not have as much effect on performance, but still can be noticeable.
If you are doing work that depends on the CPU, such as encoding video, then you will mostly be limited to the performance of the CPU. In that case, the most increase will happen when you upgrade to a more powerful CPU -- and there's no way to do that practically without replacing your logic board (IMHO, probably not worth doing on a 6-year old Mac).