In my experience it is based on whichever uses audio or that you have audio enabled. For instance, when I am just scrolling Reddit in the app Apollo the videos I have set to play automatically, but no audio unless I click the unmute and because of this I can have my airpods connected to my iPad and say have Netflix on no problem, then when/if I click on unmute the audio in my Reddit video the audeo gets swapped over to the new source the iPhone. I have ran into instances where it is annoying and I have the audio switching automatically turned off like on my Apple TV, when my little one is sleeping nearby I just use my AirPods, but since that is connected to the living room TV my audio would keep wanting to go to the TV or the alert that my AirPods are nearby would show up so I turned that off and the autoswitching so for that device I do it manually