iTunes on iMac, media on Linux server via SMB - slow imports, changes and stalls

[abernardi]

I have an unRAID server (If you're unfamiliar with unRAID, you can check it out at lime-technology.com) which is a Linux based server. I'm getting terrible performance with iTunes on my iMac. The music playback is fine. But if I import a song file or change any metadata, it has problems. This isn't new, I've read many, many posts complaining about this and other problems concerning SMB shares and file systems. But I think I may be onto something and would like some advice.

This is what I've found so far. Snow Leopard, OS 10.6.x, has a problem with SMB and/or any file system other than HFS+. HFS+ is the common files system for OS X.

There are hundreds of posts on the web complaining about temporary freezes, slowdowns, sometimes crashes on the Mac using SMB shares and other file systems. I have yet to see Apple acknowledge that there is a problem, but there are a few suggestions out there that seem to have helped people. I'll be trying out some of these this week and post the results here.

From my own observations, specifically with iTunes, I've noticed some strange behavior. Here's my setup. I have an iMac running 10.6.4 (the latest update), the latest iTunes 10. The iTunes app is located on the local internal HD, I have my iTunes library files and all my media on disk 1 of my unRAID server. I'm hard wired to the server via a gigabit router. I have iTunes set to keep the media organized and copy the media files to the server.

When importing a song file into iTunes, the file initially seems to copy at a good rate, the file appears in it's proper place and you can see the size increasing as it copies. Then it stalls for a while, then the name changes to a long hex number for a few seconds and then reverts back to its' normal state. The process can take 30-40 seconds, even with a small song file. I looked at the console and I got the following error message:

11/8/10 12:24:28 AM /Applications/iTunes.app/Contents/MacOS/iTunes[2820] Possible unresolved transaction race -103/(Master Music Library/John Lennon/Rock 'N' Roll,(null))

The same thing happens when I make a change in the metadata. It takes a LONG time to update the file. For example, I added onto the name of an album and for each song it changed I got:

11/8/10 12:37:36 AM /Applications/iTunes.app/Contents/MacOS/iTunes[2820] Possible unresolved transaction race -103/(Master Music Library/John Lennon/Signature Box Set 5 - Walls And Bridges,(null))

This happens for each file imported or changed even within iTunes. It doesn't seem to matter if the source file is on the Mac (for import from, let's say) or the server (for metadata changes, let's say)

I followed the first path in the error message: /Applications/iTunes.app/Contents/MacOS/iTunes[2820].

In the /Applications/iTunes.app/Contents/MacOS/ folder I find the following files:

iTunes
libgnsdk_musicid.1.7.1.dylib
libgnsdk_sdkmanager.1.7.1.dylib
libgnsdk_submit.1.7.1.dylib

where iTunes is a Unix Executable File.

I did some more research and have found some posts about file systems other than HFS+ having trouble with app packages. I believe that all OS X apps are actually packages, like folders, but appearing as a single file. Other file systems don't seem to know what to do with these. There was some talk of "flattening" these app packages, but I haven't gone forward with that yet because I'm not sure I understand it enough yet.

I've also noticed that in the getinfo box under sharing & permissions I usually get "You can read and write", but with apps I get "You have custom access". This was something i had noticed before and thought it was a problem on the server, but I didn't realize it until now, I'm getting the custom access on every app. So I think that's probably normal.

I'm wondering if I'm having a permissions problem with the iTunes Unix Executable File. Right now the permissions are set as follows:

system: Read & Write
admin: Read & Write
everyone: Read only

Maybe it's trying to write to that file or another one and doesn't have the right permissions since the song file is now on the server and that's causing the stalls? I'm over my head here. Anyone have any ideas?

 

[Krevnik]

I think you are going down the wrong rabbit hole. The path from the logs just tells you the binary logging the message, not anything about the source of the error, per se.

The reality is that network latency is pretty terrible. Say you have a ping of 10-20ms, that is added to each and every small piece of I/O done to the SMB share. When an app is trying to read/write a file on disk, there's usually some level of smarts about writing things out in small blocks. You don't read the whole file into RAM, then back out to disk unless the app really requires it.

iTunes has a few special files that drive the library which are likely the source of the stalls you are seeing. There is a SQLite DB that holds all the metadata and information on where the files live. There is also an XML file that needs to be updated. SQLite likes to do a lot of small transactions on a large file, which will spew a lot of small I/O requests. All of these will have that latency added to each and slow down the whole process. The XML file can also get pretty large, and depending on how that is updated, it can take a bit of time to update over the network.

That said, I'd only be storing the library itself on the server, not iTunes.app... the application doesn't really need to be there and requires system files that are installed along with the application itself which aren't exactly portable. You don't really gain anything by it.

SQLite is also not really designed to be hosted over a network like this. It works because you only have the one client accessing it, but the perf is a lot lower than on a local drive.

 

[steviem]

I guess one approach would be to try installing netatalk on the unraid box and to see if connecting using afp would make any difference...

http://lime-technology.com/forum/index.php?topic=3824

 

[Krevnik]

I guess one approach would be to try installing netatalk on the unraid box and to see if connecting using afp would make any difference...

http://lime-technology.com/forum/index.php?topic=3824

That's a good point. The specific implementation of netatalk could be a fair amount slower than SMB though. AFP support outside of Apple has been rather meh, and hasn't gotten the optimizations that SMB has in the Linux community. The ReadyNAS guys have done some good work there, so it is completely possible that work has made it's way to these folks. Worth a shot to see if it improves things, but I'm thinking it won't make it feel much faster if it did have any impact at all.

 

[abernardi]

Say you have a ping of 10-20ms,

The ping is averaging 0.2 - 0.6ms.

SQLite likes to do a lot of small transactions on a large file, which will spew a lot of small I/O requests.

That could be the problem. But I've tried putting the library on the local drive and just the media on the server and had identical results. However, the library is huge, over 750GB, so updating the SQLite and XML files could take time. The weird thing is that I only noticed this problem after I upgraded iTunes to 10. It's possible it was there before, but I don't think so. I tried rolling back to iTunes 9.2.1, but that didn't help, so I went back to 10

 

[abernardi]

I guess one approach would be to try installing netatalk on the unraid box and to see if connecting using afp would make any difference...

http://lime-technology.com/forum/index.php?topic=3824

Yes, I'm familiar with that thread. I think the consensus is that it's more trouble than it's worth. And also, I think there will be AFP support in the upcoming unRAID 5, which should be a huge upgrade!

What do you guys think about NFS? unRAID can do that.

 

[Krevnik]

Changing the file server will only do so much. What type of network connection is this all hooked up through? I'm assuming GigE, but if it is something else, that is a big chunk of the perf problem right there.

 

[abernardi]

What type of network connection is this all hooked up through? I'm assuming GigE

Yes, I'm on a gigabit ethernet switch using new CAT6 cables. Though the cable to the server is 50ft. But that shouldn't be a problem, it's not affecting other types of data throughput.
Someone on another forum pointed out that importing probably isn't an issue, that it's probably the processing of the metadata because that happens in both cases, wether you import a song, or change the metadata.