Well, to the direct question, 100% reliable on the PS3 network connection. It has a gigabit which is nice if your connecting it to a GB router/switch.
Now, I've been using XBMC on 2 xboxs for almost 2 years, and have had beautiful performance with it. There are a few things I would ask about your setup...
First off, what is your network setup to each device? Wired, wireless..?
Secondly, what kind of router/switch do you have and what speed?
Finally, keep in mind that the venerable xbox is pretty old hardware (a 733 mhz celeron: intels budget proc). It is absolutely not capable of handling any HD content with consistent results, and I've had trouble with x.264 encoding if the bit rate is too high. For my usage, I've either ripped to an ISO file which is FLAWLESS performance wise on XBMC (since it's an image of a DVD with compression/bitrates the hardware can handle since it can play DVD's)... and MP4 compression using handbrake. I encode at whatever the bit rate falls at specifying a 2gb file max (2048mb). This is the max size give or take that the XBMC can handle without losing the ability to fast forward / rewind while watching. Bigger than 2gb and things get a little wiggy. Again, MP4 video compression, not x.264. Now x.264 will play, but the bit rate has to be lower than what your mac or an apple TV can handle. I spent a ton of time doing side by side comparisons of MP4 vs x.264 at the highest bitrate that would play on XBMC and found that MP4 was better in quality (given the bitrates that were available).
My Mac is wireless, gets a constant 270mbps connection to my router (802.11n) and my XBMC and server where my video is stored are both wired (server at GB, xbox 100bt of course).
Hope that helps... like I said, give some consideration to your network setup, if your trying to do streaming to the xbmc wireless (esp. on a less than high performance wireless connection) you will have problems.
Also on the protocol front, go with SMB; It's the most reliable by far. FTP into your XBMC and pull off sources.xml and edit to your liking. You can take adavantage of the <DEFAULT> </DEFAULT> tag and make xbmc hop right on your share when you go into Videos...
Please hold the blow torches but... I will add that I've not come across any set top media extender type device that even comes close to the interface and functionality of XBMC. I'm SO looking forward to when the mac port is complete and ready for prime time. A MINI will replace my fugly xbox on that day!!!