I've owned both the 1st generation and now the 3rd generation. I think it's market is both HUGE and Apple's to take (but they need to get with it). I very much enjoy what

TV can do and think it could be the next "leg of the table" if Apple would give it some serious focus (which should be easy to do for the largest company in the world (which means they could put a lot of focus on this without taking focus away from the small mix of other products Apple produces)).
Those are key "ifs" though. Relative to the question, I find it frustrating that:
- the latest & greatest has the stuttering problem on playback (Apple, you've been at this for 5+ years now; smooth video playback should be in the bag now; there should be no more software releases where smooth video playback is lost. Hire a guy or two to test software updates to insure this; you can afford them).
- features that worked well in prior versions of the UI are removed (for example, how does it make sense to take away the "show" folders so that you could put all of the Star Wars movies in one "Star Wars" folder, all of the Harry Potter movies in one "Harry Potter" folder, etc. That worked very well between about 2009 and 2011. Then, that feature got killed for a "new" version of the UI (so now every movie- including those in a natural group like Star Wars, Star Trek, LOTR, Alien, Rocky, etc- are all individual line items resulting in a longer list).
- Poster art for TV shows used to be able to be unique for each episode. Now it's one poster for all episodes in a season. It worked great (for years!) with the former. How does taking that away so that the same poster shows for all shows make for a better UI?
- the last version of the gen 1 UI had our own movies under "movies", our own TV shows under "TV shows", etc. That was perfect, intuitive, logical, etc. The newer UI puts all of our own media under "Computers". Isn't it more intuitive (Apple) to find our movies under "Movies" and our own TV shows under "TV shows"? What happened to "just works" thinking with this product?
- even gen 3 is still not fully toe-to-toe with the "bag of hurt". If you (Apple) want to stick with the case that BD is NOT the way to go, how about giving us a full replacement that matches up on both video and audio quality... even if what can go into the iTunes store for now is not quite BD-level stuff. For example, Dolby Digital- while certainly better than no surround at all- has been a standard since about 1992. There's been HUGE progress in movie audio standards since then but nothing better than DD is built into this box's software.
Those are all within Apple's control and most of those are features that they engineered in and then took away (not for the better IMO).
Then, there is the need for other features that Apple seems reluctant to develop (for this "hobby"). No problem, just launch an app store and let the third party developers build out the missing pieces. An app store worked so incredibly well at making the other iDevices very appealing products to own. Learn (Apple) from your massive success. An app store for this

TV would likely pop it's popularity in a big way.
Then, there's the camp that misses the local storage of gen 1 (the version that had a decent amount of hard drive storage on board). I grasp why they wanted to remove an internal hard drive. However, the latest version comes with a USB port. Why not normalize that USB port so that those that want local storage could attach whatever size storage they want to it? Gen 1 jailbreakers brought this functionality to Gen 1 (so that those interested could attach massive local storage if desired). So it could be done there with just some different software. Do it here. Then, those happy with "stream everything from iCloud" can still get exactly what they want and those missing/wanting local storage could optionally get what they want too.
If they normalize that USB port AND launch an app store, then pockets of people who would like it to double as a DVR can get what they want (Elgato hardware in the USB port and Elgato apps in the app store). Pockets of people who would like it to also be their BD player could get what they want (a BD player attachment hooked to the USB port and a BD player app in the app store). Etc. In short, there are pools of people that would like it to be a "2 birds with one stone" product. If Apple doesn't want to go there, leave those solutions for the innovative third party developers.
Speaking of jailbreaking, stop limiting the device. Stop making it a "wait for the jailbreak so that it can really do what we want" box. Just do it. Make the jailbreak obsolete by either giving it that functionality OR letting the app store submissions cover those bases. Lots of other, competing boxes have all kinds of added features like these so it's not an issue of can it be done. It's just a choice to not do it (thus building the desire for jailbreak software). Should a WD or Roku box be able to play just about anything you throw at them but an

TV be so narrowly limited? Utility breeds market demand.
It seems like Apple is overly fixated on trying to make this an iTunes content-centric device at the expense of a "just works", intuitive UI. If Apple wants us to rent/buy through iTunes, win that battle on content & cost in the store (not in limiting a UI or overly biasing a UI to try to push iTunes too hard).
All that said, I love the little box. I think it could be a fantastic product for Apple. The question is about "really taking off" which I interpret to mean "with people who are not techies and/or overly enamored with Apple". I think stuff like the above is how to turn on that much larger crowd. The hardware is close now. The UI needs to go back and resurrect some jettisoned features from the past, adopt the well proven lift that comes with an app store, and basically eliminate the "wait for the jailbreak" by making it possible for the bulk of those jailbreak features to be realized in an app store. Then it can thoroughly own the space.
2 more very narrow requests I'd love to see: 1) add analog stereo audio out (gen 1 had this) so that those of us with Zone 2 receivers can use the Zone 2 feature (which usually requires an analog audio feed) and 2) how about making it also work native with 24fps and 60fps (the former being what just about all movies use and the latter being increasingly available in consumer level camcorders: 1080p 60fps, yielding "full HD" with "buttery smooth" motion) instead of locking it in at 30fps for everything?