Not sure if it's been said, but Rogan is a funny guy with an interesting background. He has that "regular guy" appeal who realized he was curious about a lot of things and likes to talk about it. I like him when he interviews celebrities, comedians and athletes so I'm fine with him getting out from under Spotify. But when he starts talking with serious intellectuals, experts or self professed experts he seems out of his element. So I don't listen to those interviews.
I think an issue folks have is that now that he has become a cultural voice (and he clearly has), expectations for what he does have also changed. It's one thing to hang out in your basement and talk s**t with your buddies, which is how the podcast can feel. That was and is fun. Some things he says and believes are ridiculous and his biases are pretty evident but with a smaller audience those beliefs and biases can be dismissed as cringy and maybe even funny because he's such a good guy. But now he's gotten a huge platform and those things that you could once dismiss are now potentially a problem. This issue gets amplified when the guests he chooses are also questionable. He started out as an entertainer and he is now more than that- whether he wants to admit it or not.