Recovering A Corrupt OpenLDAP Database On OSX Server

Recovering A Corrupt OpenLDAP Database On OSX Server
Last night we noticed some services provided by an OSX Leopard Server instance were not working correctly. The iChat, AFP and Web services were not authenticating. In Server Admin.app, the “Overview” tab of the Open Directory service reported…
LDAP Server is: Not Running
Password Server is: Running
Kerberos is: Not Running
Looking at the server error logs through Console.app, the following was occuring every 10 seconds..
com.apple.launchd[1] (org.openldap.slapd[27382]) Exited with exit code: 1
com.apple.launchd[1] (org.openldap.slapd) Throttling respawn: Will start in 10 seconds
The slapd daemon appeared not to be starting. Jumping to the command line, I tested the configuration using the `slapd -Tt` command.
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
bdb(dc=openrain,dc=com): PANIC: fatal region error detected; run recovery
bdb_db_open: Database cannot be opened, err -30978. Restore from backup!
bdb(dc=openrain,dc=com): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
backend_startup_one: bi_db_open failed! (-30978)
slap_startup failed (test would succeed using the -u switch)
http://discussions.apple.com/message.jspa?messageID=9548971
With a little research, I concluded that..
The OpenLDAP database had been corrupted, and..
The `slapd_db_recover` tool (as present on some Linux installations) is instead named `db_recover`. Ah!
After carefully backing up the /var/db/openldap folder, I ran the recovery tool and re-tested the configuration..
core:openldap admin$ sudo db_recover -h /var/db/openldap/openldap-data/
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
config file testing succeeded
The errors in Console.app stopped, and the Server Admin.app panel started reporting..
LDAP Server is: Not Running
Password Server is: Running
Kerberos is: Not Running
I had to restart the AFP, iChat and Web services on the machine to get everything working again, but all seems well now.
Last night we noticed some services provided by an OSX Leopard Server instance were not working correctly. The iChat, AFP and Web services were not authenticating. In Server Admin.app, the “Overview” tab of the Open Directory service reported…
LDAP Server is: Not Running
Password Server is: Running
Kerberos is: Not Running
Looking at the server error logs through Console.app, the following was occuring every 10 seconds..
com.apple.launchd[1] (org.openldap.slapd[27382]) Exited with exit code: 1
com.apple.launchd[1] (org.openldap.slapd) Throttling respawn: Will start in 10 seconds
The slapd daemon appeared not to be starting. Jumping to the command line, I tested the configuration using the `slapd -Tt` command.
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
bdb(dc=openrain,dc=com): PANIC: fatal region error detected; run recovery
bdb_db_open: Database cannot be opened, err -30978. Restore from backup!
bdb(dc=openrain,dc=com): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
backend_startup_one: bi_db_open failed! (-30978)
slap_startup failed (test would succeed using the -u switch)
With a little research, I concluded that..
  1. The OpenLDAP database had been corrupted, and..
  2. The `slapd_db_recover` tool (as present on some Linux installations) is instead named `db_recover`. Ah!
After carefully backing up the /var/db/openldap folder, I ran the recovery tool and re-tested the configuration..
core:openldap admin$ sudo db_recover -h /var/db/openldap/openldap-data/
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
config file testing succeeded
The errors in Console.app stopped, and the Server Admin.app panel started reporting..
LDAP Server is: Running
Password Server is: Running
Kerberos is: Running
I had to restart the AFP, iChat and Web services on the machine to get everything working again, but all seems well now.

Redmine w/OS X OpenLDAP, Parallels Server and JumpBox

OpenRain used a slew of crappy Trac sites for issue tracking until we switched to Redmine several days ago. The decision came because..

  • Redmine can authenticate off LDAP with trivial configuration.
  • Redmine has multi-project support out-of-the-box.
  • Redmine has some nifty Gantt chart and calendaring schwag and is generally better.
  • Parallels Server (for OS X) is finally available.
  • JumpBox has a beta Redmine VM image available.

If you’ve got an existing LDAP infrastructure, the whole shebang shouldn’t take more than an hour or two to set up.

  1. Install Parallels Server on your OS X Leopard server.
  2. Download the Redmine JumpBox. Generate a new MAC address and boot it. Do the one-page configuration thingy in your browser.
  3. Log into Redmine and create a new “Authentication Mode” set to LDAP. If you’re using the default OpenLDAP schema that ships with Leopard server, enter the attributes like so..redmine.png
  4. All your users should now be able to log into your Redmine JumpBox using their LDAP credentials! You’ll have to set up your projects, ACLs etc. within Redmine, but that’s some pretty hot shizzle to get running in such a small timeframe.

Mad props to Redmine, Parallels, JumpBox and Apple for further simplifying my business.

Xserve w/Leopard Server (Mac OS X 10.5), First Impressions

picture-4.pngWe just picked up a refurbished 2.66GHz quad-core Xeon from Apple, which we’ll be using for internal infrastructure. (We’re in the process of migrating from a mix of Solaris and Linux). After about 8 hours of learning the ins and outs of Leopard Server over the weekend, we had the box running Open Directory (Kerberos and OpenLDAP), DNS, AFP, SMB, FTP, domain account and machine management, mobile home directories, MySQL, Software Update, Xgrid controller, Wikis, Blogs, iCal and VPN services, all tightly integrated with single sign-on (via Kerberos) into a sexy 1U package.

  • Xserve (refurbished discount, direct from Apple): ~$3K
  • 3 x 750GB Disks (Newegg): ~$450
  • 2 x Apple Drive Module (direct from Apple): ~$380
  • 2 x 2GB FB-DIMM RAM (Crucial): ~$300
  • Infrastructural sanity: priceless. (…or ~$4.5K after tax and random small stuff)

That’s some serious value considering how much of a PITA setting all this up can be in Linux (or whatever) without vendor support, and far cheaper than paying a Systems Administrator in the long run. The Server Admin and Workgroup Manager tools are pretty freakin’ usable, too, relative to the internal complexity of the system. I’m a happy camper for now… let’s see if it lasts.