There is at least one application that I know of that makes for long backup times. It's AppSniper. There is an option in its settings to clear the icon cache. If you do that before backing up, your times will be much shorter.
I'm guessing that you have an application installed that has a similar nasty way of backing up.
If you want to take the time to track it down, this would make the job a little faster. Delete half of your applications. Check backup times. If they are slow, your culprit is still on the iPhone. Times are slow, delete half of the remaining ones, and so on. If, after deleting the first half, the times are fast, delete the remaining applications and reinstall half of the group you first deleted. Check backup times.
Once you narrow your group containing the suspect down to a manageable few, install them one at a time until you find the one giving you the problem. This will be much faster than installing and uninstalling all of your applications one at a time.
Then only you can decide whether that application is important enough for your to hang on to it.
If you do this, I would be interested to know which application is causing your troubles. Others might be too.