It isn't really the size of the library but the number of elements in it. iTunes doesn't load the library per say, it loads the index file that tells it where to find all of the data. Thus if you have a library with 5TB but it's 1000 movies, it isn't that bad. But if you have a 250GB audio library with 50,000 songs in it; it's gonna suck.
Via wifi, it does take a LONG time to load a large library (lots of elements, say 50,000+). I have moved to all wired except for the laptops in the house and that helps a lot. But I can't load that huge music library anymore via wifi, it's just a pain in the butt. But then again, since I went to a personal library, I RARELY look at the other one anymore anyway -- I may just delete it.