Adds up to what? Sure, you can say "I wasted x minutes last year" because they're numbers that can be totaled. But beyond a total of that time, what else can you do with it? You can't save those wasted milliseconds and enjoy them all at once. It doesn't work that way. It's not really "useful" time. But just for kicks, let's say all that wasted time in a day adds up to, I'll be generous, 3 seconds. So, right before bed, you now have three additional seconds thanks to a fully optimized OS. What are you going to do with those extra three seconds that you couldn't do before? How would those extra seconds make a noticeable difference in your life?
I'm not trying to say those fractions of a second aren't annoying. They don't bug me, but I've been around long enough to know they actually do bother some people. I'm just saying "it all adds up" really doesn't make a lot of sense in the context that it's being used. It's just a phrase people like to throw around because they think it strengthens their point. But it really doesn't.