I had this issue. It saved a temporary number while my number was being ported across from my new mobile provider. I couldn’t delete it from my Apple ID. I called Apple support and they were stumped but said they would investigate it.
A month later I noticed it had disappeared so I guess they’d found a way although they never came back to me.