Very Odd Server Admin / Netboot Problem

[AmestrisXServe]

Some basics:

I'm running a 2008 XServe with 10.5.8 ULC.
It has an SAS controller, driving many devices, plus a Fibre card, driving an XRAID (that is also giving me headaches).

Via a Cisco switch, it is networked to an XServe G5, also running 10.5, and a Windows Enterprise system, with a Linux system planned.

For a good while, I had zero netboot problems. After accidentally damaging the filesystem on my primary boot drive, which is a paid of 1TB SATA drives configured as RAID1, when i load the Netboot prefs pane in Server Admin, the display shows no Ethernet (or other) ports, and no drives.

That is, no checkboxes at all. Just empty, grey fields where they belong. The checkboxes aren't greyed out: They aren't there!

I checked for any similar problem on a variety of websites, including ADE, to no avail.

At first,I thought it may be a permissions problem, so I repaired disc permissions. No success.

Then, i thought it might be a corrupted config file, as I ran into issues with FTPd doing odd things when it became corrupted. No help there either.

next, I tried rolling back one version from a DMG (I do very frequent backups). That didn't help, so I moved back another version, where I know it was working. Nothing changed.

Next, i went back to the original boot drive, which has issues, as I tried to merge some stuff and screwed it up. I never use it, but it still boots and runs services. Even so, the Netboot pane shows no interfaces!

I said, 'Alright, maybe it's a PRAM problem', so i did a triple PRAM flash, then tried the drives, oldest to newest. No change...

I enabled Netboot on the XServeG5, and *it* does show the Ethernet (and FW) interfaces, and the only drive in the system,a lame 80GB paid of SATA drives, again as RAID1. (The G5 Netboot works.)

Nothing that I do seems to restore Netboot on the Intel XServe. When I load the Netboot pane, again, the normal checkboxes for enabling interfaces do not exist, and the drive selection and disc image selections are not there. it's just blank.

After this, I connected the second Ethernet port (en1) to the switch, and still nothing. At that, I stopped and had some whiskey and Xanax...

My questions are as follows:
Has anyone else ever encountered anything like this?
What could be causing it?
Can I manually configure the Netboot service, and then run it somehow (in terminal, or elsewhere)?
Can anyone provide a solution/fix?
Do I somehow need to reset the eth0 interface, if there is even a way to do this?
Is there a reset beyond a PRAM zap to try?

I can't imagine at this point why it is happening...

Another Netboot question: How can I fool Netboot into serving earlier OSX versions,and can I make it serve Linux LiveCDs? Both are part of my long-term plan, and if the included Netboot client won't suffice, if anyone can suggest a third-party method to offer Netboot OSX and Linux images, that would be fantastic.

Finally, although off-topic, the XRAID shows under RAID Admin, and I set up a RAID5 config there, however, despite numerous reboots, I can't view that in Disk Utility to format it and attach it to the system. Am i supposed to use Disk utility to do this, as there seems to be no service in RAID Admin to format the RAID5 and mount it.

Again, this is on 10.5.8, which I need for PPC compatibility, and Classic compatibility on the XServe G5. I have another XServe G5 to add to the clustre, and a Linux system to add when I finish going insane.

Somewhat unimportant specs on the machine with the problem:
System: XServe Intel (2008)
Model Ident: Xserve 2,1
CPU: Quad-Core 2.8GHz
RAM: 10GB
OS: OSX Server 10.5.8 Unlimited Client (Genuine License, not pirated)
Running Services: VNC (Enterprise,fully licensed), LittleSnitch (fully-licensed), MacPorts (not working, need to build via a make file?)
PCIe: RocketRaid 2722+ EJ240
PCIX: Apple FC Card
Expansion:12-Bay enclosure;1x RAID-10(24GB=12GB);XRAID,7x50GB drives, ttotal 3.5TB,set as RAID5 at around 2.5TB.
FW/SATA Bridgeboard from CalOptic; not fully configured.
5:1 SATA-RAID Bridgeboard from CalOptic; not fully configured.
HP LTO4d rive; working, but not attached at present.
FW800 Hotbay w/1TBdrive
No special services yet.
DHCP handled by router/NAT (DD-WRT)
OSX DHCP Service Disabled
AFP, SMB and NFS Enabled
OpenDirectory (Master)
/Users still on boot drive; migration to follow.
eth0 verified working
FW Verified working
eth1 enabled, but not verified; I just connected it.

Remote (WAN/Internet) VNC access possible if needed.

If you need any other info to help, please ask.

Thank you for your time.

P.S. I did Netboot testing w/o the G5 system on the network, with no change. I can use the G5 as a secondary NB server while this is down,,but I want to fix this problem.

Thank you all in advance.

P.S. I will give SSH and VNC access to trusted members on the forums if that would help to fix this.

 

[chrfr]

LittleSnitch (fully-licensed)

First, immediately uninstall this. You do not want Little Snitch on a server, ever. Use the built in firewall if you need one, however, test your systems before enabling the Apple one to see if your problem is solved.

If that doesn't solve it, reinstall the 10.5.8 Admin Tools from Apple and test again.
It's been a long time since I've looked at an xServe RAID but do you have the right Fibre Channel card and cables? Did it ever work in the Intel server? http://support.apple.com/kb/HT1769

 

[AmestrisXServe]

First, immediately uninstall this. You do not want Little Snitch on a server, ever. Use the built in firewall if you need one, however, test your systems before enabling the Apple one to see if your problem is solved.

If that doesn't solve it, reinstall the 10.5.8 Admin Tools from Apple and test again.
It's been a long time since I've looked at an xServe RAID but do you have the right Fibre Channel card and cables? Did it ever work in the Intel server? http://support.apple.com/kb/HT1769

I use LS to monitor what programmes are accessing specific ports, particularly when other users run hosted applications and services, but LS was not installed on two of the boot drives that I tried, and was disabled on a third. I did consider that it may be a problem, but as it wasn't even installed on the original drive, I don't see how it could be connected.

It's easy enough to disable in any event.

What is the best method to reinstall only Admin tools?

With regard to the XRAID, I can see and address it in RAID Admin, but not in Disk Utility. That is, I created the RAID5 in RAID Admin, and it shows all status as green there. When I go to Disk Utility, the RAID5 set is not present to be formatted.

As I am only using seven drives, I only connected the upper controller by ethernet, but there should be no need to have both controllers connected to use only one set, from what I recall.

Is there something else I need to do to add the RAID5 set as a mounted volume (e.g. manually add it in fstab)?

EDIT: I'm not sure what reinstalling Admin Tools would be either, as the original boot drive is doing the same thing. I am going to do a clean install on another volume to see if it is the same too.

What could could possibly be blocking only NetBoot from seeing the two Ethernet ports and the FW port? Everything else on the system sees and uses them...

 

[chrfr]

I don't see how (Little Snitch) could be connected.

You're trying to troubleshoot network connectivity issues. You don't know that the problem you have now is caused by exactly the same problem you had before, and you're running a 3rd party application that's designed to block network connections. Don't just disable it, uninstall it, at least for testing.

What is the best method to reinstall only Admin tools?

Google says http://support.apple.com/kb/DL831
If you are running 10.5.8 server and haven't ever updated the Admin Tools, that may be part of your trouble. I don't remember if they came through Software Update.

When I go to Disk Utility, the RAID5 set is not present to be formatted.

Again, do you have the correct Fibre Channel card and cables? You need to connect the FC cables (2 of them) from the RAID to the Mac. If the card you have came from a PowerPC Mac, it probably will not work in the Intel machine. Here is the link again to check. http://support.apple.com/kb/HT1769
The xServe RAID is not network attached storage and has to be connected to the host Mac via Fibre Channel; the ethernet connections are purely for management.
http://manuals.info.apple.com/MANUALS/0/MA337/en_US/Xserve_RAID_Users_Guide.pdf

 

[AmestrisXServe]

I will try the Admin Tools update, but as I said, Little Snitch is not installed on at least one of the boot devices that I tested. The configuration also previously worked with LS running.

that's why I don't see how it would be related. If it was on all of the boot volumes that I tested, I would agree, but as I used cloned drives from before I added it, I can't understand how a piece of software that isn't on the drive, with no daemons to run, can affect Netboot's ability to detect eth0 as a valid interface.

I would configure Netboot from the shell if I knew all of the required config files, but there is little documentation on what the config files are for Netboot,and where they are located.

If I could configure it from the shell, and launch the daemon, I could live without the GUI. To be honest, the GUI for Netboot has always felt clumsy to me, and likes to nag. I would prefer to be able to force images to work if possible.

It literally stopped working when I had to do a restore, and after that point, anything from which I boot exhibits the same symptoms. That's why I did a PRAM zap, as it seemed to be that kind of problem.

Updating Admin Tools seems wise in any event, if they were not part of the 10.5.8 updater bundle. Is there no 10.5.8 version of Admin tools? (That link is for 10.5.7.)

FC Cables... Yes, I am relatively sure that they are correct.The card is a PCI-X FC card, and is recognised by the system. The system indicators are all green on both ends for the FC connection.

Now... The question I have to ask is this: When using RAID Admin, and building the RAID5 on the XRAID, does that software use the FC bus *at all* for communication, or are all the operations handled by the NIC and the internal drive controllers?

The Intel compatibility may be the problem. I can drop the system onto one of my G5 XServes to check that, as they all have FC cards. I have a pile of FC PCI-X cards, so I out one in every system... I'm still looking into alternative ways to use the interface, other than for drive arrays.

When I do a clean install, I will check the Netboot options, and if the server still shows no ports or drives, then I am at a loss as to how to proceed. If that fixes it, I will try some other things to fix it.

Really, I could use any of the other systems for Netboot, but this main system has the greatest disc capacity, and RAID arrays. if I drop the XRAID onto a G5, I can use that for Netboot if all else fails.

Do you know of any other Netboot daemon that I could use instead of Apple's version? I would like to be able to serve Linux LiveCDs (Intel and PPC), and PXE stuff, and am curious if there is an alternative package that can handle Apple NWR Netbooting as well as PXE.

I would also like to be able to make a 10.4 or 10.3 image on 10.5, or at least host them on 10.5. Why does Netboot Imager not support making older versions? Does it only read kernel and user information for the current, running OS?

Additional Thought: It occurred to me, from your mentioning that the FC card from a G5 may be problematic, and incompatible with an Intel XServe... I think that this problem only started after I added the PCI-X FC card to the system. Could that be the source of this problem, given that OSX is trying to ident connections for Netboot and is panicking on the FC card in the process?

I am somewhat sure that I added the FC card from my stash after setting up Netboot,and that I never checked after installing it, to see if Netboot was still working. I only noticed the problem after the rebuild because I had to reassign the Netboot info after rebuilding the RAID 10 array that I have attached via SAS. It may have been down the entire time. I need to check this, as if that is the problem, then I can actually document a source and a remedy.

I would have thought that an Apple FC card from a G5 (PCI-X) would work in the Intel model on the PCI-X bus, but that may merely be my misconception. Now, I don't know why the G5 PCI-X card wouldn't work in the Intel machine--and I could use and would appreciate the technical details--but as the XRAID isn't visible on the system, and Netboot is refusing to see the ports, this very well may be the root cause.

The funny part is that, while I set up the XRAID using the Intel system, I mean to use it on the G5. I only used the Intel system to initialise it because it was easier at the time.

I still want to do a clean install test, as this will tell me if it is a HW problem. Then, if the problem persists on a clean install, I will pull the FC card, and if that makes the problem go away,I will have a flowchart for diagnostics on this problem.

I checked the guide, and for the PCI-X card (M9274G/C), Apple simply states (for Intel machines) 'Do not use'. No explanation of why, or what problems may arise from trying to use it. I do know that the cables should be correct, at least, for the G5 systems: The PCI-X card types are cable type agnostic, as they operate only at 2GB/s DTR.

I felt rather down yesterday, so I didn't get around to doing the clean install, and that may be a good thing, as it gave me time to think about what kind of HW problems could be the underlying cause. Fingers crossed, I will work on it today, and if it turns out to be the problem, I will need to write up a quick flowchart page for this problem, in the event that anyone else ever runs into it.

I have a spare PCIe bridge, so I can swap out the PCI-X interface, as I have no other use for it, and put another SAS card, or an eSATA card in the other slot. (Extra SAS interfaces are always good to have.)