That isn’t changing color, that’s picking up color from something you wear regularly. Denim at a guess.
Magic Eraser will get it off, though it could make the band slightly shiny if you have to rub too hard. My original white from 2015 got the same way along the edges and it’s got a sheen now. I’ve looked at ways to make it dull again and a gentle scuff with 1,200 grit paper and a light wipe with a Caped Cod cloth seems to be likely to work, I just haven’t had a chance to buy sandpaper in that grit since the hardware store reopened here.