Preston Lee's Blog

Category: computer

Keeping Multiple Development Machines Synchronized with Homeboy
Most of us developer types have at least two machines we use routinely, and managing that can be a chore. Specifically, I usually want to do the following every time I sit down at a machine to hack on something:
- Keep config files like .bash_profile and .gitconfig synchronized. (This requires scripting since Dropbox ignores hidden files.)
- Patch all OS-level libraries using the native package management system on OSX, Ubuntu and CentOS.
- Update my database daemons and other system services.
- Upgrade development libraries.
- Merge in ‘git pull origin master’ project source code for all my clones.
Doing all this every time I hop to a different machines was chore, so I wrote a few BASH scripts to help. More importantly, I recently packaged them into a public GitHub and released it! Homeboy is a set of small, plain BASH scripts. After following the simply installation instructions, you just run homeboy every morning:

$ homeboy

This will run the updates you specify, though I’ve only included the stuff I use regularly: namely brew (OSX), rvm (OSX and Linux), apt (Ubuntu), yum (Red Hat/CentOS) etc. I ask that you submit pull requests to add support for updating Perl, Python, MacPorts etc. The synchronization mechanism works by zipping the specified list of files into a .zip in a synchronized directory managed by Dropbox, SugarSync etc. “Pushing” your current set of files to Dropbox is done via:

$ homeboy-push

After pushing, the next time `homeboy’ is run on any configured machine, the .zip file will be unzipped into your home directory. It’s really not complicated, but saves time by having to make the same change a bunch of times across different machines and platforms, and having subtle differences.

When using the git options, homeboy assumes you have a single directory where all your clones are kept, such as ~/Developer/git. Every subdirectory that looks like a git clone will have ‘git pull origin master’ run inside it.

Pretty silly stuff, right? But hey, all I do is run `homeboy’ every morning I plan on doing development work on a machine, and sip on a cup coffee while everything is brought up to date. 🙂

Please help test and submit pull requests!
2013.05.14
Ubuntu 12.10 to 13.04 Server Upgrade Error

Are you Googl’ing around trying to figure out why you’re getting this error when trying to `do-release-upgrade’ your Ubuntu 12.10 system, even though you’re pretty much up to date?

root@mia:~# do-release-upgrade
Checking for a new Ubuntu release
Traceback (most recent call last):
File “/usr/bin/do-release-upgrade”, line 145, in <module>
fetcher.run_options += [“–mode=%s” % options.mode,
AttributeError: type object ‘DistUpgradeFetcherCore’ has no attribute ‘run_options’
Error in sys.excepthook:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/apport_python_hook.py”, line 137, in apport_excepthook
os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), ‘wb’) as f:
OSError: [Errno 2] No such file or directory: ‘/var/crash/_usr_bin_do-release-upgrade.0.crash’

Original exception was:
Traceback (most recent call last):
File “/usr/bin/do-release-upgrade”, line 145, in <module>
fetcher.run_options += [“–mode=%s” % options.mode,
AttributeError: type object ‘DistUpgradeFetcherCore’ has no attribute ‘run_options’
root@mia:~#
deb http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted

I just burned a couple hours trying to figure out why I only got this error on one specific server. It turns out that it’s a bug in the version of the ubuntu-release-upgrader-core package, and you must be pulling updates from a “quantal-updates” repository to get the fixed package when you `apt-get upgrade’. To fix this, edit your /etc/apt/sources.list and make sure you have the following two lines:

deb http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted
deb-src http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted

After making sure you have these lines, get yourself updated again and re-try the release upgrade script:

apt-get update
apt-get upgrade
do-release-upgrade

…and you should be rollin’ towards Raring!

2013.04.26
Proxy Authentication Against Devise-Based Ruby Applications

(Quick GitHub link!)

My team at TGen has a Rails application using a typical devise + cancan installation for authentication and authorization, respectively, and a related Apache HTTPD proxy server we needed to authenticate against active accounts in the webapp before passing traffic to their destination, which is a single host. We tried mod_authn_dbd for a while, but the combination of Devise’s default adapive bcrypt password encryption that is not supported by mod_authn_dbd by default, a complex set of SQL statements to process the authorization directly, and that none of the existing modules seem to be able to do what we want, made it a messy ordeal.

Instead, we created devise-proxy: a simple proxy server written as Rack Middleware that you can deploy using Passenger within Apache or nginx. You simply provide devise-proxy with a config.yml file defining the hostname and port of the devise webapp to use as an authentication web service, and hostname and port of a single server to use as a forwarding target.

Clients connecting to the proxy use the email/password for their proxy username/password credentials. All authenticated clients will be forwarded to your provided host/port, but using the path in the client’s original HTTP request. Clients failing to authenticate will receive a 403 and the forwarding host will never be hit.

gem install devise-proxy # to install

git clone git://github.com/preston/devise-proxy.git # to clone!

devise-proxy is released under a BSD license via the GitHub project, here. Good luck!

2012.10.30
How To Custom Brand The OpenStack “Horizon” Dashboard
I’m deploying OpenStack “Essex” on Ubuntu Server 12.04, and have the openstack-dashboard package installed to provide the web-based “Horizon” GUI component newly added for the Essex release. Canonical also provides an openstack-dashboard-ubuntu-theme package that brands the Python-based Django GUI. Despite that the last major Canonical-maintained packages based on the OpenStack “Diablo” release in Ubuntu 11.10 did not include an administrative GUI, Horizon — as a standalone component — has been very stable for a mainstream debut. In the future, though, I’d like to see a quick and easy way to change the default branding to use your own logo, colors, and titles using only the GUI’s administrative screens.

The horizon documents briefly mention branding customization to give you a head start, but you probably want more specific steps. Here’s my custom-branded Horizon dashboard with custom colors, logo, and site title:

Once you know where to make the appropriate changes, it’s super simple. Step-by-step:
1. Create a graphical logo with a transparent background. The text “TGen Cloud” in this example is actually rendered via .png files of multiple sizes I created with a graphics program. I used a 200×27 for the logged-in banner graphic, and 365×50 for the login screen graphic.
2. Set the HTML title (shown at the top of the browser window) by adding the following line to /etc/openstack-dashboard/local_settings.py
```
SITE_BRANDING = "Example, Inc. Cloud"
```
3. Upload your new graphic files to:
```
/usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/img/
```
4. Create a new CSS stylesheet — we’ll call ours custom.css — in the directory:
```
/usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/css/
```
5. Edit your CSS file using the following as a starting point for customization, which simply overrides the Ubuntu customizations made in the ubuntu.css file. Change the colors and image file names as appropriate, though the relative directory paths should be the same.
```
/*
* New theme colors for dashboard that override the defaults:
*  dark blue: #355796 / rgb(53, 87, 150)
*  light blue: #BAD3E1 / rgb(186, 211, 225)
*
* By Preston Lee <plee@tgen.org>
*/
h1.brand {
background: #355796 repeat-x top left;
border-bottom: 2px solid #BAD3E1;
}

h1.brand a {
background: url(../img/my_cloud_logo_small.png) top left no-repeat;
}

#splash .login {
background: #355796 url(../img/my_cloud_logo_medium.png) no-repeat center 35px;
}

#splash .login .modal-header {
border-top: 1px solid #BAD3E1;
}

.btn-primary {
background-image: none !important;
background-color: #355796 !important;
border: none !important;
box-shadow: none;
}

.btn-primary:hover,
.btn-primary:active {
border: none;
box-shadow: none;
background-color: #BAD3E1 !important;
text-decoration: none;
}
```
6. Open the following HTML template in an editor:
```
/usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html
```
7. Add a line to include your new stylesheet: (I’ve highlighted the new line in bold.)
```
...
 <link href='{{ STATIC_URL }}bootstrap/css/bootstrap.min.css' media='screen' rel='stylesheet' />
 <link href='{{ STATIC_URL }}dashboard/css/{% choose_css %}' media='screen' rel='stylesheet' />
 <link href='{{ STATIC_URL }}dashboard/css/custom.css' media='screen' rel='stylesheet' />
```
```
...
```
8. Restart apache just for good measure:
```
 sudo service apache2 restart
```
9. Reload the dashboard in your browser and fine tune your CSS appropriate.
You’re done!

[ezcc]
2012.05.09
OpenStack Nova Essex MySQL Database Schema Diagram and SQL

I’m in the process of setting up an OpenStack Essex installation on a small cluster of Ubuntu 12.04 servers using the Canonical-supplied and supported packages. I innocently and unfortunately ended up with a state corruption issue causing nova-compute to crash on startup, and found it necessary to dive into the schema as a troubleshooting aid. I used MySQL workbench to reverse-engineer the schema into an EER diagram as well as raw SQL. I’m sure others will find this useful as the OpenStack community slowly upgrades to Essex. Hope it helps!

[Download diagram, SQL, and MySQL workbench file]

2012.05.03
OpenStack Diablo on Ubuntu 11.10 w/MySQL

If you’re installing the OpenStack cloud computing platform “Diablo” release on Ubuntu 11.10 server using the official guide and are using MySQL, don’t forget to:

sudo apt-get install python-mysqldb

The “glance” package (and probably others) are configured to use SQLite by default and do not automatically install this dependency by default. Unfortunately, the November 2011 version of the manual also fails to mention this and the log file (/var/log/glance/registry.log) prints a variety of “INFO [sqlalchemy.engine.base.Engine…” messages but no warnings or errors. I assume a corresponding comment applies to PostgreSQL as well but have not confirmed.

2012.03.05
Quick FASTQ File Parsing Via Memory Mapping In C/C++

I recently had a need to speedily parse through 8GiB+ .fastq text files to calculate a simple statistic of genomic data. My initial “pfastqcount” implementation in Ruby worked fine, but with many files to process took longer than I had hoped in addition to consuming an alarming amount of CPU. I ended up reimplementing the pfastqcount command-line program in C, which takes one or more .fastq files, memory maps them, and creates the statistic. Simply dropping my algorithm down to raw C significantly sped up the process and reduced CPU usage, especially coming from an interpreted language. If any of you bioinformaticians find the need to implement a FASTQ data processing algorithm in C, I encourage you to fork the project and use it as a template. The project is Apache 2.0 licensed for your convenience and publicly available on GitHub.

2011.11.17
Preston And AT&T Give Each Other The Finger

I made the jump the last day before Verizon stopped offering unlimited data plans. The delay for switching to Verizon was not lack of motivation–AT&T service has always paled to Verizon in Arizona and is nearly nonexistent at my summer home–but in desperate procrastination of dealing with the migration process. My longest conversation (highly abbreviated) with AT&T on the matter took about an hour and was so traumatizing that I can’t see myself ever returning. As far as I’m concerned AT&T is dead and buried:

Coverage map showing AT&T's miraculous ability to provide "Best" service without operational towers.

Me: I’m not happy with my AT&T service and would like to cancel my service plan.

Customer Service Representative: I’m sorry to hear that, sir. May I ask why?

Me: I’m in an area with about 1-bar service about half the time, no 3G data (EDGE only), and constant dropped calls. I’m not really getting “service” per se.

Rep: I’m very sorry to hear that. We can cancel your plan for $<huge fee>.

Me: Well… I really don’t think that’s entirely fair. The issue isn’t really that I don’t WANT service, but AT&T isn’t providing what I’m already paying for. I’m paying about $100/month for unlimited 3G data, <list of other features>, and I only get a few of them some of the time. Check the coverage map.

Rep: Yes, sir! I can see you live in a “Best Coverage” area. That is very good!

Me: 1-bar signal 50% of the time, no 3G and dropped calls the other 50% is “Best Coverage”?

Rep: The map shows we have multiple towers in the area! You should be getting great service according to the map.

Me: I understand what the map says; I’ve seen it many times, trust me. The issue is not just me, though. No one else with AT&T seems to get usable service here, either.

Rep: I’m very sorry to hear that, sir. One of the towers is not operational. That may have something to do with it. Would you like us to send out an engineer to test your hardware?

Me: Wait… what? First, my hardware is fine. It works fine in <other cities with service>. No one else’s phone works well here on AT&T’s network, either. Second, if you’re dispatching an engineer wouldn’t it make sense to fix the tower instead? …You know, the NOT OPERATIONAL one that is currently providing “Best Coverage”?

Rep: Unfortunately we cannot do that, sir.

Me: It doesn’t make sense to charge me for a service you just admitted you can’t provide. I understand I’m under contract and don’t dispute that, but AT&T has obligations, too, and if AT&T can’t meet them it isn’t right to punish the customer.

Rep: Unfortunately, sir, it is your fault for choosing to live in an area without good service coverage.

Me: ARE YOU FUCKING SERIOUS??? I checked your goddamn map before, during and after moving here, and the fucking thing says “BEST COVERAGE” despite having a non-operational tower. I’ve been here for some time now and it’s never been any better.

Rep: Yes, sir! Coverage in that area is strong. Would you like us to send out an engineering to test your equipment?

Me: YOU JUST SAID A TOWER IS NOT OPERATIONAL. SOME GUY WAIVING MY PHONE TO THE SKY IS NOT GOING TO MAKE IT CONNECT TO A TOWER THAT DOESN’T WORK. DO YOU UNDERSTAND WHAT THE FUCK A CELL TOWER DOES?

Rep: Like I said, sir, it is AT&T’s policy to charge cancellation fees according to your contract. We cannot even consider overriding them in such a strong service area.

Me: <Infuriated abrupt disconnect.>

So, I’m now staring at a ridiculous cancelation bill. On the bright side, though, I sold my old AT&T iPhone the next week via eBay for over $200, which not only covered the new Verizon hardware cost but activation fees as well. I’m not getting great 4G on my Mifi (which was disclosed though), but at least I’m getting ok 3G and voice service on my iPhone for about the same price. AT&T? Please.

2011.07.27
Apple Wireless Bluetooth Keyboard Dvorak Hacking

It’s very easy to switch around the keys on Apple’s current generation of wireless bluetooth keyboard. The first time doing this hack I only had one “close call”, but know that I know how to get the keys off safely it’s simple, fast and easy. I used a couple razor blades to pop off the keys, but a very small screwdriver would work just as well and be much safer.

The keys are best removed by lifting the cap key from the top left or top right corners. The plastic mechanics beneath the key move analogously to a cherry picker, and you interfere with them less by lifting the top corners of the key.

Once you’re done, the only drawback to your new sleek Dvorak keyboard is the lack of nubs on the U and H keys. Very carefully dab a small drop of superglue on them to address the issue, and enjoy!

(Sorry for the lack of pictures … when I figure out where I put them I’ll update this post. Andy Skelton has some pictures in a similar post.)

2011.07.27
TGen News Dashboard Widget For Mac OS X

The Translational Genomics Research Institute (TGen) is a non-profit 501(c)(3) organization focused on developing clinically relevant medical diagnostics and smarter treatments related to genomic profiling. I’ve had the pleasure of working with the institute for over a year on a project related to breast cancer, and thought I’d share a widget I wrote keep you up to date on TGen’s latest news items.

The TGen News: Dashboard Widget for Mac OS X v1.1 gives you an always up-to-date set of headlines, pulled straight from the TGen news feed.

If you enjoy this widget, please donate a few dollars to the TGen Foundation and do your part to support the many causes that TGen pursues. Enjoy!

From the TGen website:

At TGen, investigators are pushing the limits of cutting-edge research and technology to discover the genetic cause of disease. Experiments that were impossible and impractical only a few years ago are now conducted every day.

Discovery fuels TGen’s translational research and lies at the heart of our scientific investigations. TGen’s research divisions are designed to foster a wide range of genetic discoveries. These divisions draw heavily upon TGen’s scientific platforms to expedite findings. TGen’s labs are staffed by teams of researchers focused on making genomic discoveries in common diseases and disorders in the areas of oncology, neurogenomics and metabolic disease.

Note: This free product is provided by Preston Lee, and is neither officially endorsed nor supported by TGen.

2011.03.08