Don’t Upgrade vCenter Server Appliance From 5.0 to 5.1

Thinking of upgrading your vCenter Server Appliance from a 5.0 image to the 5.1 image using the official VMware upgrade process covered in the documentationDon’t do it. It fails somehow. Every time.

Screen Shot 2013-06-27 at 3.08.32 PM

 

Unexpected error during the upgrade process.

To attempt to upgrade again, restore the old vCenter Server Appliance and external database from backup/snapshot, deploy the new vCenter Server Appliance again and start the upgrade process from the beginning.

Yeah.. you read the right. Now both your old and new appliance images are b0rked, and you’ll need to restore from the vCenter 5.0 backup that you definitely made before attempting the upgrade. In private emails, VMware has acknowledged the issues and has advised to stick with 5.0 and hold off on upgrading the vCenter Server appliance until the next major release since the upgrade/migration tools are such horrible crap. Time to sit back and wait it out!

 

Update: The built-in update tool in 5.0 that allows you to apply point release upgrades is also fairly broken. (See the red “Failed to install..” message in the background?)

Screen Shot 2013-06-27 at 3.26.14 PM

Ubuntu 12.10 to 13.04 Server Upgrade Error

Are you Googl’ing around trying to figure out why you’re getting this error when trying to `do-release-upgrade’ your Ubuntu 12.10 system, even though you’re pretty much up to date?

root@mia:~# do-release-upgrade
Checking for a new Ubuntu release
Traceback (most recent call last):
File “/usr/bin/do-release-upgrade”, line 145, in <module>
fetcher.run_options += [“–mode=%s” % options.mode,
AttributeError: type object ‘DistUpgradeFetcherCore’ has no attribute ‘run_options’
Error in sys.excepthook:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/apport_python_hook.py”, line 137, in apport_excepthook
os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), ‘wb’) as f:
OSError: [Errno 2] No such file or directory: ‘/var/crash/_usr_bin_do-release-upgrade.0.crash’

Original exception was:
Traceback (most recent call last):
File “/usr/bin/do-release-upgrade”, line 145, in <module>
fetcher.run_options += [“–mode=%s” % options.mode,
AttributeError: type object ‘DistUpgradeFetcherCore’ has no attribute ‘run_options’
root@mia:~#
deb http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted

I just burned a couple hours trying to figure out why I only got this error on one specific server. It turns out that it’s a bug in the version of  the ubuntu-release-upgrader-core package, and you must be pulling updates from a “quantal-updates” repository to get the fixed package when you `apt-get upgrade’. To fix this, edit your /etc/apt/sources.list and make sure you have the following two lines:

deb http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted

After making sure you have these lines, get yourself updated again and re-try the release upgrade script:

apt-get update
apt-get upgrade
do-release-upgrade

…and you should be rollin’ towards Raring!

OpenStack Diablo on Ubuntu 11.10 w/MySQL

If you’re installing the OpenStack cloud computing platform “Diablo” release on Ubuntu 11.10 server using the official guide and are using MySQL, don’t forget to:

sudo apt-get install python-mysqldb

The “glance” package (and probably others) are configured to use SQLite by default and do not automatically install this dependency by default. Unfortunately, the November 2011 version of the manual also fails to mention this and the log file (/var/log/glance/registry.log) prints a variety of “INFO [sqlalchemy.engine.base.Engine…” messages but no warnings or errors. I assume a corresponding comment applies to PostgreSQL as well but have not confirmed.

MacPort MySQL “Can’t find file: ‘./mysql/host.frm'” Error

I recently set up a new Mac OS X Snow Leopard laptop for software development purposes. After going through my usual MacPorts installation and installed MySQL using the following steps

  1. sudo install mysql5-server
  2. sudo cp /opt/local/var/macports/software/mysql5/5.1.44_0/opt/local/share/mysql5/mysql/my-medium.cnf /opt/local/etc/mysql5/my.cnf
  3. sudo -u mysql mysql_install_db5
  4. sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist

A few days past, and one day I notice that launchctl was loading the script, but `nmap localhost` did not show the the process listening on port 3306 as expected. I looked at the log file:

sudo tail -f /opt/local/var/db/mysql5/preston.local.err

..and noticed it was terminating with the following errors:

100331 12:05:30 [ERROR] /opt/local/libexec/mysqld: Can’t find file: ‘./mysql/host.frm’ (errno: 13)
100331 12:05:30 [ERROR] Fatal error: Can’t open and lock privilege tables: Can’t find file: ‘./mysql/host.frm’ (errno: 13)

100331 12:05:30 [ERROR] /opt/local/libexec/mysqld: Can’t find file: ‘./mysql/host.frm’ (errno: 13)100331 12:05:30 [ERROR] Fatal error: Can’t open and lock privilege tables: Can’t find file: ‘./mysql/host.frm’ (errno: 13)

After a considerable amount of grief I evertually discovered that the contents of /opt/local/var/db/mysql5/ need to be owned by the appropriate mysql user (in my case “_mysql”), but some files were being owned by the “root” user. After correcting all file ownership and restarting the service…

  1. sudo chown -R _mysql /opt/local/var/db/mysql5/
  2. sudo launchctl unload -w /Library/LaunchDaemons/org.macports.mysql5.plist
  3. sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist

Everything now seems to work fine again. Hope this helps!

Recovering A Corrupt OpenLDAP Database On OSX Server

Recovering A Corrupt OpenLDAP Database On OSX Server
Last night we noticed some services provided by an OSX Leopard Server instance were not working correctly. The iChat, AFP and Web services were not authenticating. In Server Admin.app, the “Overview” tab of the Open Directory service reported…
LDAP Server is: Not Running
Password Server is: Running
Kerberos is: Not Running
Looking at the server error logs through Console.app, the following was occuring every 10 seconds..
com.apple.launchd[1] (org.openldap.slapd[27382]) Exited with exit code: 1
com.apple.launchd[1] (org.openldap.slapd) Throttling respawn: Will start in 10 seconds
The slapd daemon appeared not to be starting. Jumping to the command line, I tested the configuration using the `slapd -Tt` command.
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
bdb(dc=openrain,dc=com): PANIC: fatal region error detected; run recovery
bdb_db_open: Database cannot be opened, err -30978. Restore from backup!
bdb(dc=openrain,dc=com): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
backend_startup_one: bi_db_open failed! (-30978)
slap_startup failed (test would succeed using the -u switch)
http://discussions.apple.com/message.jspa?messageID=9548971
With a little research, I concluded that..
The OpenLDAP database had been corrupted, and..
The `slapd_db_recover` tool (as present on some Linux installations) is instead named `db_recover`. Ah!
After carefully backing up the /var/db/openldap folder, I ran the recovery tool and re-tested the configuration..
core:openldap admin$ sudo db_recover -h /var/db/openldap/openldap-data/
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
config file testing succeeded
The errors in Console.app stopped, and the Server Admin.app panel started reporting..
LDAP Server is: Not Running
Password Server is: Running
Kerberos is: Not Running
I had to restart the AFP, iChat and Web services on the machine to get everything working again, but all seems well now.
Last night we noticed some services provided by an OSX Leopard Server instance were not working correctly. The iChat, AFP and Web services were not authenticating. In Server Admin.app, the “Overview” tab of the Open Directory service reported…
LDAP Server is: Not Running
Password Server is: Running
Kerberos is: Not Running
Looking at the server error logs through Console.app, the following was occuring every 10 seconds..
com.apple.launchd[1] (org.openldap.slapd[27382]) Exited with exit code: 1
com.apple.launchd[1] (org.openldap.slapd) Throttling respawn: Will start in 10 seconds
The slapd daemon appeared not to be starting. Jumping to the command line, I tested the configuration using the `slapd -Tt` command.
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
bdb(dc=openrain,dc=com): PANIC: fatal region error detected; run recovery
bdb_db_open: Database cannot be opened, err -30978. Restore from backup!
bdb(dc=openrain,dc=com): DB_ENV->lock_id_free interface requires an environment configured for the locking subsystem
backend_startup_one: bi_db_open failed! (-30978)
slap_startup failed (test would succeed using the -u switch)
With a little research, I concluded that..
  1. The OpenLDAP database had been corrupted, and..
  2. The `slapd_db_recover` tool (as present on some Linux installations) is instead named `db_recover`. Ah!
After carefully backing up the /var/db/openldap folder, I ran the recovery tool and re-tested the configuration..
core:openldap admin$ sudo db_recover -h /var/db/openldap/openldap-data/
core:openldap admin$ sudo /usr/libexec/slapd -Tt
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
overlay_config(): warning, overlay “dynid” already in list
config file testing succeeded
The errors in Console.app stopped, and the Server Admin.app panel started reporting..
LDAP Server is: Running
Password Server is: Running
Kerberos is: Running
I had to restart the AFP, iChat and Web services on the machine to get everything working again, but all seems well now.

Parallels Server for Mac Pricing

Fresh from my inbox..

Parallels Server for Mac will be available soon. As a thank you to all participating Parallels Server for Mac Beta users, Parallels is offering an exclusive discount on a single Parallels Server for Mac license. Purchase this new software for only $700* – a 30% savings.

Hmm… well, the product has been working fairly well for us at OpenRain, but I’m not sure $700 per major version is going to be worth it as opposed to buying another cheap Dell machine and running VMWare Server on Linux for free, which we already do in some of our hosted environments. Here’s the kicker in tiny font at the bottom of the email explaining the asterisk after “$700”..

The limited-time discount offer is limited to a single server from May 30 – June 30, 2008. Annual maintenance is required at the time of purchase, starting at an additional $249.75 per year. For academic pricing and volume licensing, register now or contact Parallels Sales at +1 (425) 282-6400.

So that’s $950 for the first year of our first system alone. Hmmmm… 

Xserve w/Leopard Server (Mac OS X 10.5), First Impressions

picture-4.pngWe just picked up a refurbished 2.66GHz quad-core Xeon from Apple, which we’ll be using for internal infrastructure. (We’re in the process of migrating from a mix of Solaris and Linux). After about 8 hours of learning the ins and outs of Leopard Server over the weekend, we had the box running Open Directory (Kerberos and OpenLDAP), DNS, AFP, SMB, FTP, domain account and machine management, mobile home directories, MySQL, Software Update, Xgrid controller, Wikis, Blogs, iCal and VPN services, all tightly integrated with single sign-on (via Kerberos) into a sexy 1U package.

  • Xserve (refurbished discount, direct from Apple): ~$3K
  • 3 x 750GB Disks (Newegg): ~$450
  • 2 x Apple Drive Module (direct from Apple): ~$380
  • 2 x 2GB FB-DIMM RAM (Crucial): ~$300
  • Infrastructural sanity: priceless. (…or ~$4.5K after tax and random small stuff)

That’s some serious value considering how much of a PITA setting all this up can be in Linux (or whatever) without vendor support, and far cheaper than paying a Systems Administrator in the long run. The Server Admin and Workgroup Manager tools are pretty freakin’ usable, too, relative to the internal complexity of the system. I’m a happy camper for now… let’s see if it lasts.

The $1,000 (USD), 2TB OpenSolaris File Server

images.jpeg

Here’s how to score a sweet OpenSolaris-compatible 2TB file server for $1000 (USD). I’m running Solaris Express Developer Edition on it with a ZFS RAIDZ1 file system.

  • Motherboard: Asus M2NPV-VM. A great, inexpensive mATX board w/4 SATA ports, 2 IDE, PCI-X, GigE, built-in nVidia graphics w/DVI and VGA outs, FireWire, and tons of USB love. Oh, and the official nVidia driver works like a charm. Sweet! (The only thing that hasn’t run out-of-the-box is the built-in audio controller. Oh well.) $100.
  • CPU: AMD Athlon 64 X2 4800+ Dual Core Processor. 2.5GHz, 2x512KB, 1GHz Bus. $130.
  • Memory: 4x1GB via OCZ OCZ2G8002GK DDR2-800 PC2-6400 kits. $175 total (after MIR).
  • Drives: 4x500GB Western Digital Caviar SE 16 WD5000AAKS 7200RPM 16MB Cache SATA 3.0Gb/s for the ZFS data volume. $450. (Old, small, slow and cheap PATA disks reused for the system volume. FMV ~$25.)
  • Optical Drive: Reused old typical IDE DVD-ROM. FMV ~$20.
  • Case: Reused typical ATX tower w/450 watt PSU. FMV ~$80.
  • Peripherals: Reused optical mouse and keyboard. FMV ~$20.

Total: $1,000. Nice!