AFP rendering Mac servers unusable

[fragra]

We've had problems for months now, big problems with our MacOSX 10.5x Server in connection with server based user folders.

Apple has been no help. Complaints are piling up worldwide and the threads are becoming longer and longer. Even the last update 10.5.6 doesnt address the issue, though apple said it would.

Here's our problem briefly. On our current XServe systems, which are made available over the AFP protocol server-based user folders, the AFP service is completly using the CPU. On our system with 8 cores the CPU burden is up to 750%. Work by all logged on users is no longer possible because all applications no longer react and the computer hangs up eventually. The only thing left is to restart the server, but in most cases this does not help because the problem returns after a new start. The AFP service takes the whole CPU. For a company like ours with over 100 server-based workstations this is inacceptable.

Administrators around the world are frustrated and clueless. Apple says nothing. There is no feedback, help from Apple Enterprise support is like a trying to poke-through fog. Many now believe that there is such a serious problem with the AFP proctocol that even Apple developers are helpless. Please this thread at Apple: http://discussions.apple.com/thread.jspa?messageID=8798468 The problem is evidently so bad that companies are experiences damages from down time. And there is also a new website, that perhaps you would want to publicize with a notification. http://www.go2xserve.com

We administrators no longer know where to turn. Apple is not being helpful and provides no feedback or support of any kind. Thanks much for any help you can provide.

Yours truly,

 

[mikedavis]

fragra,

are you 10.5 servers updated from 10.4? or, are they fresh installs of 10.5?

 

[fragra]

fragra,

are you 10.5 servers updated from 10.4? or, are they fresh installs of 10.5?

in April 2008 we tried to update from 10.4 to 10.5.3 but unpredictably apple had a bug in the afp protocol, so the managed useres disconnected after a few minutes automatically.

we installed totally new with 10.5.4, at the moment we are at 10.5.6. it looks like each version had the same problem!

 

[Sayer]

So you are running apps over AFP on over 100 desktops? Have you thought maybe you should restructure instead of trying to fit a square block through a round hole?

Why not net-boot from a single pre-made image and use AFP for sharing files, not apps and I presume the data the apps are working with?

Why does the app *have* be run from one server? Sounds like a poor app choice, or skimping on buying a proper client/server solution.

 

[foidulus]

So you are running apps over AFP on over 100 desktops? Have you thought maybe you should restructure instead of trying to fit a square block through a round hole?

Why not net-boot from a single pre-made image and use AFP for sharing files, not apps and I presume the data the apps are working with?

Why does the app *have* be run from one server? Sounds like a poor app choice, or skimping on buying a proper client/server solution.

Because stuff like this worked fine in Tiger? It's harder and harder to be an Apple apologist after the non-stop bug-fest that is Leopard, however Apple refuses to admit their is a problem and as Leopard goes on, the bugs become more numerous in my opinion.

We haven't switched any of our servers over to Leopard, and I am incredibly hesitant about doing so, but since Apple forces you to use Leopard with new hardware, we will have to make the plunge soon.

 

[corbywan]

I must have missed it. The the OP say apps? I thought it was just home folders on the server for documents.

 

[fragra]

So you are running apps over AFP on over 100 desktops? Have you thought maybe you should restructure instead of trying to fit a square block through a round hole?

Why not net-boot from a single pre-made image and use AFP for sharing files, not apps and I presume the data the apps are working with?

Why does the app *have* be run from one server? Sounds like a poor app choice, or skimping on buying a proper client/server solution.

it is "just" home folders on the server for documents, no apps!

 

[SC68Cal]

I'm surprised that you were even able to get close to that kind of usage with AFP shares. Sadly, you're going to be waiting in vain for a fix from Apple. Your best bet is to switch over to SMB/CIFS which can and does handle those kinds of loads. Apple never tests anything that they do on large enterprise deployments because they actively avoid enterprise solutions and their company culture does not put a priority on addressing enterprise needs beyond putting up a nice PR web page about VT when they bought 200+ Xserves for a computing grid.

So many admins got screwed over when they put out OS X Server 10.5 and everyone's Active Directory got screwed up. Did they ever test it? Apparently not! It took them until the point two update to get their stuff together and send everyone a fix. I know myself - after having a colleague wrestle with trying to keep his server and client deployments on the same release that Apple wants to sell everyone a shiny product, take the money, and run.

From the way Apple conducts themselves - it's almost like they're laughing behind your back when you purchase an XServer.

So you are running apps over AFP on over 100 desktops? Have you thought maybe you should restructure instead of trying to fit a square block through a round hole?

100 clients for simple file services is not a square peg in a round hole. That's Servers 101 buddy. Have you ever administered anything bigger than two computers attached with a crossover cable?

 

[Les Kern]

I'm surprised that you were even able to get close to that kind of usage with AFP shares. Sadly, you're going to be waiting in vain for a fix from Apple.

A while back I was having some terrific issues with laptops linking to my OD servers... response was sluggish, lots of beach balls. I have an OD master with replicas, about 20 Xserves 10 with home directories and 2200 clients. Eventually it was traced to my HP AP's and switches. Once that was identified things are a lot better. Sure, there are occasional OD issues like some of the 10.5/Legacy settings I have yet to solve, but during all that I have never, ever seen numbers as described here. And Apple technical support has been nothing but helpful along the way.
So you say you're surprised at the AFP usage rates. Why? My servers are hammered daily with little problem. You say we will "wait in vain for Apple to fix this". Why's that? Do you have inside information you'd like to share? If not, this is nothing more than bashing. I have to assume you have first-hand knowledge?
Being no fan-boy, I'd be the first to scream at Apple if my mission critical systems take a hit because of their gear, but I pick my battles and TRY make SURE it's nothing my limited knowledge has missed. And, since I have a brain that is sometimes the size of a pea, missed things I have, and screamed I have. But in the end it's corrected, with yours truly looking like a dolt.
This being said, I will be investigating this issue more... perhaps there is something to it. Always best to keep an eye on everything.
It's what we do.

 

[Les Kern]

We've had problems for months now, big problems with our MacOSX 10.5x Server in connection with server based user folders.

Sometimes when the issue is that serious I like to schedule a visit by an Apple SE. They are pretty pricey, but make no mistake... they are to a person brilliant. Some of the projects I've had them assist me on have been incredibly complicated, and in every case they finish the job quickly and completely. It's obvious that your problems are affecting not only the productivity of your users, but it quite probably is destroying the goodwill we try so hard to keep healthy... and what they charge I would consider chump-change if there is resolution.
Having an SE on site is a lot different that talking to some nebulous individual hundreds of miles away, concerned or not.

 

[fragra]

Having an SE on site is a lot different that talking to some nebulous individual hundreds of miles away, concerned or not.

yeah but i´m not old and slow, it´s jut a mac! it´s bsd, we can do it.
we are in a very close connection with an applecare technician but he is helpless...
even the "new" appleshare version he gave us didn´t make it.

There are a number of possible workarounds for this, whose effectiveness range, until now we have no solution.
The top 10 are:
1. Setting AFP wan threshold
2. Turning off spotlight on server by marking AFP volume as private
3. Turned off Time Machine even though no backup volumes where defined
4. Set kern.maxfiles=200000 and kern.maxfilesperproc=5000 sysctl’s
5. Turned DS_Store off on both clients and servers
6. Turned off smb (windows samba) file server
7. Disabled auto-disconnect in AFP after idle time.
8. Removed spotlight indexing on all afp volumes and deleted .SpotLight-V100 directories on AFP volumes.
9. Verified that Host Cache Flushing is disabled on external RAID array.
10. Set the following default: defaults write com.apple.desktopservices DSDontWriteNetworkStores true. Set as preference for all groups.
11. Disabled kerberos for AFP authentication.
12. Changed the fibre topology to Point to Point for all 4 fiber connections to the Promise VTrak array.
13. Stop spotlight indexing by using the command: touch /Volumes/Sharename/.metadata_never_index
14. Renamed odpac.bundle in /System/Library/KerberosPlugins/KerberosAuthDataPlugins/ to odpac.bundle_DISABLED
15. volume with our home dirs has to keep at least 10% free for performance reasons

--> so we did everything above but no. 1 did it --> wan_quantum and wan_threshold

We’ve deployed the modified plist this morning to all of our clients, the server (AFP service) is running normal at 10% - 40%. i think that this was part of the problem. but we have to keep an eye on it, i don’t think this is the final solution.

funny idea:
-> HowTo: Make Ubuntu A Perfect Mac File Server And Time Machine Volume [Update6]
http://www.kremalicious.com/2008/06/ubuntu-as-mac-file-server-and-time-machine-volume/
"1. Modify and install Netatalk (Open Source AFP implementation)" -< last updated version in 2005

my company is a very complex setup with some very different compartements.
we want to get it worked out and don´t want to play around with some 10 or 20 clients...

 

[Les Kern]

yeah but i´m not old and slow, it´s jut a mac! it´s bsd, we can do it.
we are in a very close connection with an applecare technician but he is helpless...
even the "new" appleshare version he gave us didn´t make it.

There are a number of possible workarounds for this, whose effectiveness range, until now we have no solution.
The top 10 are:
1. Setting AFP wan threshold
2. Turning off spotlight on server by marking AFP volume as private
3. Turned off Time Machine even though no backup volumes where defined
4. Set kern.maxfiles=200000 and kern.maxfilesperproc=5000 sysctl’s
5. Turned DS_Store off on both clients and servers
6. Turned off smb (windows samba) file server
7. Disabled auto-disconnect in AFP after idle time.
8. Removed spotlight indexing on all afp volumes and deleted .SpotLight-V100 directories on AFP volumes.
9. Verified that Host Cache Flushing is disabled on external RAID array.
10. Set the following default: defaults write com.apple.desktopservices DSDontWriteNetworkStores true. Set as preference for all groups.
11. Disabled kerberos for AFP authentication.
12. Changed the fibre topology to Point to Point for all 4 fiber connections to the Promise VTrak array.
13. Stop spotlight indexing by using the command: touch /Volumes/Sharename/.metadata_never_index
14. Renamed odpac.bundle in /System/Library/KerberosPlugins/KerberosAuthDataPlugins/ to odpac.bundle_DISABLED
15. volume with our home dirs has to keep at least 10% free for performance reasons

--> so we did everything above but no. 1 did it --> wan_quantum and wan_threshold

We’ve deployed the modified plist this morning to all of our clients, the server (AFP service) is running normal at 10% - 40%. i think that this was part of the problem. but we have to keep an eye on it, i don’t think this is the final solution.

funny idea:
-> HowTo: Make Ubuntu A Perfect Mac File Server And Time Machine Volume [Update6]
http://www.kremalicious.com/2008/06/ubuntu-as-mac-file-server-and-time-machine-volume/
"1. Modify and install Netatalk (Open Source AFP implementation)" -< last updated version in 2005

my company is a very complex setup with some very different compartements.
we want to get it worked out and don´t want to play around with some 10 or 20 clients...

Talk about thorough... copy/paste. I'll be looking more at this....

And I saw this article at 548.
http://www.afp548.com/article.php?story=20060329213629494

Thanks.

 

[fragra]

And I saw this article at 548.
http://www.afp548.com/article.php?story=20060329213629494

some more useful links:
http://www.nabble.com/AFP-eating-up-CPU-td19976358.html
http://discussions.apple.com/thread.jspa?messageID=8798468
http://go2xserve.com/
http://www.afp548.com/article.php?story=MCXRedirector
http://jochsner.dyndns.org/scripts/NHR.html