I agree. I have 2 EVs (1 for 2 years, the other for 1.5 years), I have fast charged 3 times total. Even if I was doing road trips, it would be 1-4 trips a year.
I do care about L2 charging times because I have 1 connector and 2 EVs with no access to free chargers at work. So when I get home, I back into my garage with my TMY, then run the cable to my wife's TM3. Once hers is finished I move it over and leave it connected to my TMY overnight. I need to be able to get her car back to 80% before I go to bed. So, for me, L2 charging plays a role in my daily life. If I had 1 EV, or 2 connectors, I wouldn't care at all. Had I known I was going to go full EV, I would have gone the extra step and done a 60-amp setup, instead of the 50-amp. We easily make do with the 50-amp setup, but only because my wife uses a maximum of 30% of her TM3's battery daily (in the winter, usually she uses 15%-20%).
I admit, my situation is not common yet, but full EV households are going to become a regular thing, and this will be an issue as having 2 connectors is expensive (you can load share, but it adds a big expense to a home setup).
Simply tossing a bigger battery into an EV doesn't solve this problem, it's all about efficiency.