I think you have to shift your way of adjusting the airpods while in your ear.
Take your middle finger and push the airpods in your ear while using your pointer and thumb to stretch your ear lobe away from the airpod. This will not trigger the double tap feature. This is what I do to adjust during the day (I use one airpod at a time all day at work everyday).