Lotta great stuff in your OP, but I quoted what resonated with me most. I don't know why it is that we think somebody's proficiency in a given field makes them the most proficient educator about their… proficiency. It's entirely possible that one's sporadic creativity that earned them their fame also makes for inconsistent and unhelpful self-examination.
In addition to the observer effect, I also wonder if a bit of survivor bias plays into the low quality of some bios; often the famous falsely attribute their success to things that many unsuccessful people have tried time and time again. I don't know whether it's because of a desperate desire to come up with an ad-hoc justification for their success or not; maybe people need to feel as though they deserve to have won their circumstantial lottery.
Sometimes I feel as though asking smart/creative/charismatic people about their intelligence/creativity/charisma is as futile as asking a model how they have such good bone structure. "At what point in your career did you decide to have such striking cheekbones?" I think that if most autobiographies were honest, they'd be much shorter and more similar. Something like, "I was born, lived, and eventually made things that people really liked for some reason. When examined through the lens of certain academic theories, my work has been found to have merit, which is really cool because I wasn't even thinking about that stuff when I made my stuff. I also wrote this book.
The end."My favorite biographies have been ones that integrate an element of narrative structure. I really liked Alison Bechdel's graphic memoirs Fun Home and Are You My Mother?. They were self aware, poignant, and explored aspects of her life through things like psychological and feminist theory—while acknowledging how goofy it is to do so. Despite that self-awareness, at the end of each memoir, I felt as though I had a much better idea of why she is who she is. I also liked Aaron Sorkin's Steve Jobs biopic. Distilling Jobs' life into a three act exploration of the relationship between him and his daughter revealed more about him than most biographers manage with hundreds of pages.
Three last odd thoughts: reading bios reminds me of reading the "self-summaries" on dating sites. Dunno if you've ever done this, but I find that most say something like, "I'm bad at describing myself," and any other detail they add applies to most people, like, "I'm shy at first but talkative once I've warmed up to you. I like to eat food, to sleep, and to travel." It's as though our biggest, most deeply felt traits are universally shared. If you drew a Venn diagram perfectly illustrating every person's attributes, you'd probably notice the overlapping section in the center would overwhelm every little difference. Nice for a sense of unity, terrible for interesting insightful biographies.
I also hate it when people take note of a famous person's few unique attributes and assume that adopting those traits will ensure their success follows. If you want to be successful like Steve Jobs or, say, John Lennon, you can be sure as hell that wearing round glasses, neglecting your children, and being seen as a nice hippy despite having a volatile personality aren't going to instill in you their technological or artistic visions. I imagine the truth is too boring to sell; people are successful because of equal parts luck and rigor.
We might not all be naturally gifted (lucky) like Nikola Tesla, but if we aspire to greatness, we're more likely to get there through Edison-ian iteration and rigor. If tiny iteration and unrelenting trials were good enough to evolve apes into interplanetary-traveling destroyers of worlds who communicate with devices made outta electrified sand, it should be good enough to help in our crafts.
But whaddo I know?
