Ruby Supercomputing: Using The GPU For Massive Performance Speedup

Confreaks just released a Ruby/JRuby GPU presentation of mine from Mountain West Ruby Conference 2011 in Salt Lake City, Utah. Slightly stale, but all still very relevant to HPC today!

http://confreaks.tv/videos/mwrc2011-ruby-supercomputing-using-the-gpu-for-massive-performance-speedup

Prolog for Ruby (ruby-prolog) Updated, Hits v1.0.1

After a long period of inactivity I’ve updated the F/OSS ruby-prolog gem! It’s been updated for ruby 2.0.0, bundler, and minitest, and released as v1.0.1!

ruby-prolog allows you to solve complex logic problems on the fly using a dynamic, Prolog-like DSL. Basic use is encompassed by stating basic facts using your data, defining rules, and then asking questions. Why is this cool? Because ruby-prolog allows you to leave your normal object-oriented vortex on demand and step into the alternate reality of declarative languages.

With ruby-prolog:

  • There are no classes.
  • There are no functions.
  • There are no variables.
  • There are no control flow statements.

You can use all these wonder things — it’s still Ruby after all — but they’re not needed, and mainly useful for getting data and results into/out of the interpreter. Prolog still tends to be favored heavily in artificial intelligence and theorem proving applications and is still relevant to computer science curricula as well, so I hope this updated release proves useful to the Ruby community.

Check out a simple ACL enforcement example or, for the daring, the ruby-prolog solution to the Tower’s of Hanoi problem!

 

Capistrano Hanging During .tgz Upload

I just spent an annoying amount of time troubleshooting a new cap-based deploy.rb for a Rails app to a new CentOS 6.4 server. The deploy.rb worked perfectly on other systems, but something about the configuration was causing `cap deploy’ to hang during the following:

servers: [“myapp.example.com”]
** sftp upload /var/folders/0j/vmvt2by53m901wtmfv8xfjw40000gn/T/20130530225953.tar.gz -> /tmp/20130530225953.tar.gz

Long story short, when manually sftp’ing to the server, I’d get:

Received message too long 458961005

Ah ha! After taking a cue from this dude, I edited the .bashrc to redirect any output to /dev/null. I could then successufully sftp to the server manually, and `cap deploy’ magically worked as expected. The offending line was actually rvm, which of course insists on print a color-coded message when you `rvm use 2.0.0′ (or whatever). Changing the offending line to:

rvm use 2.0.0 >> /dev/null

..did the trick.

Proxy Authentication Against Devise-Based Ruby Applications

(Quick GitHub link!)

My team at TGen has a Rails application using a typical devise + cancan installation for authentication and authorization, respectively, and a related Apache HTTPD proxy server we needed to authenticate against active accounts in the webapp before passing traffic to their destination, which is a single host. We tried mod_authn_dbd for a while, but the combination of Devise’s default adapive bcrypt password encryption that is not supported by mod_authn_dbd by default, a complex set of SQL statements to process the authorization directly, and that none of the existing modules seem to be able to do what we want, made it a messy ordeal.

Instead, we created devise-proxy: a simple proxy server written as Rack Middleware that you can deploy using Passenger within Apache or nginx. You simply provide devise-proxy with a config.yml file defining the hostname and port of the devise webapp to use as an authentication web service, and hostname and port of a single server to use as a forwarding target.

Clients connecting to the proxy use the email/password for their proxy username/password credentials. All authenticated clients will be forwarded to your provided host/port, but using the path in the client’s original HTTP request. Clients failing to authenticate will receive a 403 and the forwarding host will never be hit.

gem install devise-proxy # to install

git clone git://github.com/preston/devise-proxy.git # to clone!

devise-proxy is released under a BSD license via the GitHub project, here. Good luck!

Quick FASTQ File Parsing Via Memory Mapping In C/C++

I recently had a need to speedily parse through 8GiB+ .fastq text files to calculate a simple statistic of genomic data. My initial “pfastqcount” implementation in Ruby worked fine, but with many files to process took longer than I had hoped in addition to consuming an alarming amount of CPU. I ended up reimplementing the pfastqcount command-line program in C, which takes one or more .fastq files, memory maps them, and creates the statistic. Simply dropping my algorithm down to raw C significantly sped up the process and reduced CPU usage, especially coming from an interpreted language. If any of you bioinformaticians find the need to implement a FASTQ data processing algorithm in C, I encourage you to fork the project and use it as a template. The project is Apache 2.0 licensed for your convenience and publicly available on GitHub.

Spring 2011 ASU Webapp Course

In Spring 2011, the College of Technology and Innovation at the ASU Polytechnic campus will again be offering CST 533/494: Database-Centric Enterprise Application Development. The calendar is not yet out, but should soon be listed as a Thursday evening class: one night per week for the Spring semester. The course will be available at both undergraduate and graduate levels for 3 credit hours.
The course essentially teaches dynamic web application architecture concepts and development processes using Ruby (w/Rails), Java (w/Spring, Eclipse, and Tomcat), SQLite, MySQL, and other industry favorite technologies as a means to the learning process.
  • New technologists: If you’re looking for training and experience in this area, definitely check out the ASU Polytechnic campus.
  • Old hats: If you are an “old hat” software engineer just looking for the credit hours, this may provide a fun, project-oriented opportunity towards working on your degree.
  • Companies: I am always on the lookout for opportunities to expose students to industry perspectives. If you’re interested in any sort of corporate sponsorship or collaboration, please contact me directly.

Upgrading From Ruby 1.9.1 to Ruby 1.9.2

I’ve spent half the day so far inbeded in the furious stressful upgrade process of a handful of Ubuntu Linux 10.04 and Mac OS X Snow Leopard systems from Ruby 1.9.1 to Ruby 1.9.2. I haven’t even gotten to the Rails 3.0.0 stuff yet: just the baseline Ruby installation. I’ve gone through the upgrade process on both types of systems so far and the base issues have been the same. Here’s a common issue that many people are running into:

preston$ gem1.9

/opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems/source_index.rb:68:in `installed_spec_directories’: undefined method `path’ for Gem:Module (NoMethodError)
from /opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems/source_index.rb:58:in `from_installed_gems’
from /opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems.rb:883:in `source_index’
from /opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems/gem_path_searcher.rb:81:in `init_gemspecs’
from /opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems/gem_path_searcher.rb:13:in `initialize’
from /opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems.rb:841:in `new’
from /opt/local/lib/ruby1.9/site_ruby/1.9.1/rubygems.rb:841:in `block in searcher’
from <internal:prelude>:10:in `synchronize’
<…and so on…>

Assuming you’re upgrading from a previous Ruby installation, note that the “site_ruby” directories are no longer used, and will eff up your 1.9.2 installation if you fail to delete them after the install. On OS X, run:

sudo rm -rf /opt/local/lib/ruby1.9/site_ruby/

On Ubuntu Linux 10.04, run:

rm -rf /usr/local/lib/ruby/site_ruby/

…to correct this issue. All note that you may see errors such as this:

root@li92-132:~# rake –version

/usr/local/lib/ruby/1.9.1/rubygems.rb:340:in `bin_path’: can’t find executable rake for rake-0.8.7 (Gem::Exception)

from /usr/local/bin/rake:19:in `<main>’

…despite have a rake gem installed. Apparently 1.9.2 comes with a version of rake internally, but is unable to find it for some reason relating to the rake.gemspec file. Remove the file to fix this issue. On Ubuntu Linux 10.04, run:
rm /usr/local/lib/ruby/gems/1.9.1/specifications/rake.gemspec
Notice the “1.9.1” path of the PATH. Yeah.. it’s weird. But for compatibility reasons your 1.9.2 installation will continue to use a path with 1.9.1. To quote the Ruby 1.9.2 FAQ page:
The standard library is installed in /usr/local/lib/ruby/1.9.1
This version number is “library compatibility version”. Ruby 1.9.2 is mostly compatible with the 1.9.1, so its library is installed in the directory.
I’m sure there’s a wonderful technical reason for this, but it’s still misleading and confusing as hell. I ended up manually deleting a bunch of stuff I shouldn’t have because I thought I was innocently “cleaning up” after the old version. Whatever. Additional suggestions:
  • Just to keep things clean, you may also want to remove your old Ruby 1.8.x builds. (I recommend doing so unless you have older apps that haven’t moved to 1.9.x yet.)
  • Phusion Passenger seems to work fine on Ubuntu 10.04 with the latest version of Apache 2 as of this writing, though don’t forget to recompile, reinstall, reconfigure and restart apache2 when you do so.
  • Check if you still need rack v1.0.1 installed (for older Rails app) before nuking everything. 🙁
I need a beer!

3D Desktop Ruby Applications On Linux

Over the past year I’ve put out a few working demos of how to develop full 3D, OpenGL-based OSX applications using Ruby. Most of the comments I’ve received have been positive, but I think the high learning overhead has been the prime limiting factor in addoption. I also decided to focus exclusively on Mac OS X, further limiting the potential audience.

I’m pleasing to learn that Martin “monkstone” Prout has successfully run the code contained within my Starfield.app–basically a folder of code on a OS X system that looks and behaves like an .exe does Windows–on a Kubuntu Linux system. I haven’t personally tried to replicate this myself due to a lack of time, but you can read how monkstone did it on his blog.

Ruby: On The Perl Origins Of “&&” And “||” Versus “and” And “or”.

Overview

Avdi Grimm has a recent post noting the use of “and” and “or” Ruby keywords as essentially control flow operators, and hinting on their Perl origins. This instantly recalled several mental notes of old Perl programs, so I though I’d put out a few quick notes on the Perl equivalents for Ruby programmers not versed in Perl 5.

Perl Semantics

Perl actually borrows the precedence rules of “&&” and “||” from C, though I’m not entirely convinced the Perl and Ruby semantics of “and” and “or” are identical. We have to remember that Larry Wall is a linguist, and that some of Perl’s idiosyncrasies are due more to human considerations than machine. Programming Perl has several pages of great content on “and” and “or”. For example, here’s an excerpt from the circa 2000 version of Programming Perl:

But you can’t just up and replace instances of || with or. Suppose you change this:

$xyz = $x || $y || $z;

to this:

$xyz = $x or $y or $z; # WRONG

That wouldn’t do the same thing at all! The precedence of the assignment is higher than “or” but lower than ||, so it would always assign $x to $xyz, and then do the ors. To get the same effect as ||, you’d have to write:

$xyz = ($x or $y or $z);

The moral of the story is that you must still learn precedence (or use parentheses) no matter which variety of logical operator you use.

What Avdi says about precedence and control flow seems correct, though the reasons for it–in Perl at least–is to differentiate between two subtly distinct human linguistic semantics. For example, consider the following two English statements.

  1. I’m either traveling by car or just staying home.
  2. I’m either traveling by car or by train.

At first glance the semantics seem identical, but on closer inspection they are completely different. Let’s look at each.

I’m either traveling by car or just staying home.

In Perl, this is the equivalent of:

$traveling_by_car = can_use_car($me) or stay_home($me);

The semantics of or is to describe consequence (aka control flow) in a specific order. Either I’m traveling_by_car(), or else screw the whole thing and I’ll just stay_home() instead, in which case it makes little difference what the value of $traveling_by_car is. In other words, the semantics of the assignment trump that of the consequence, and should the consequence occur (which in this case probably has side effects), the assignment probably doesn’t matter. This is why Perl users often use or in the following context instead of ||:

open FOO, $file or die “a horrible death: $!”;
@lines = <FOO> or die “$file is empty?”;

I’m either traveling by car or by train.

In English, this is a misleading statement of truth for several reasons.

  1. What we mean is that one of the two options is true, but one and only one. As programmers we know this better as an xor statement–A is true if and only if B is false, and vice versa–but since there is no colloquial English equivalent of xor, we instinctively infer the meaning from the speakers misstatement.
  2. As programmers we tend to look at || as a short-circuit operator evaluated left to right. But this English statement defines no explicit evaluation order. This could have been written “I’m either traveling by train or by car” (notice “car” and “train” are reversed), but it means the same thing. In other words, we are evaluating the arguments for truth, not consequence. Thus, order should not affect truth.

Closing Thoughts

In most real-world Ruby contexts these operators will work interchangeably so long as precedence order is considered. But even so, keep in mind that mixing the two styles in different contexts is not necessarily a sign of inconsistency. Using the appropriate operator may actually be a sign of maturity: a way of communicating slightly different semantics in an otherwise logically equivalent context.