Amazon AWS EC2 and Cluster SSH

A devops engineer at work

Cluster SSH, or its OSX incarnation CsshX are great tool to quickly connect to a collection of machines and issue interactive commands in parallel. You just configure them with a collection of machines called 'clusters', set your .ssh/config correctly for all those boxes and voila, you are now in the matrix.

The problem arise when you run your stuff on Amazon's AWS on EC2 dynamic instances. It's impossible to know at a given time which instances are effectively running, and it makes your cssh (or CsshX) configuration useless.

The good news is, Amazon provides an API to access and manage EC2 instances. The even better new is there's a Perl package to access that programmatically. So..

Lets put EC2 API Access and Cluster SSH together: ec2-cssh

ec2-cssh is a small command line utility that queries the EC2 API for instances using configured sets of criteria (by tag, by whatever AWS accepts for filtering instances) and generates a cssh (Cluster SSH) or a csshX (CsshX) shell command that connects you to all those maching boxes.

What's even better is that ec2-cssh builds its configuration from several config file, at system level, user level, and current directory level.

I'm not going to enter into details, but the idea is that you can do this:

- Define the sets of instances (by tag, etc...) at project level, shared with all the devs in the project checkout in a file .ec2cssh.conf

- Define your AWS credentials, your favourite cluster ssh command or your own preferred instances sets in the safety of your $HOME/.ec2cssh.conf.

- Stuff can also be configured system wide in /etc/ec2cssh.conf

Of course all of this is optional, and you can perfectly have one config file with everything in it.

O, and by the way, ec2-cssh is written in Perl.

Hope you'll enjoy it!

Jerome.

Image::Magick with Perlbrew.

Image::Magick is great, but it's a pain to install if you are using perlbrew, or if your system's Image Magick library is out of date. After some googling I found that this was the best technique, but cpanm support is missing, making using Image::Magick very difficult to include in your application's dependencies.

So I wrote a perlbrew + cpanm compatible Image::Magick perl package.

Note that it still depends on your perl to be build to generate the shared library libperl.so. So give it a go and if your perl is not compatible, it should tell you what to do:

cpanm -v Alien::ImageMagick

Then use Image::Magick as usual. Your application will have to depend on Alien::ImageMagick, not Image::Magick.

Happy coding!

Blobs in Postgresql with Perl

Are you using, or do you want to use Postgresql Blobs with Perl?

Providing you are in the Moose ecosystem, here is Pg::Blobs. Pg::Blobs is a role that adds blobs handling methods to any Moose based package.

Here is how to use it:

package My::App;
use Moose;
with qw/Pg::Blobs/;
# Just provide:
sub pgblobs_dbh{ .. return the DBH .. }
1;

Now your package My::App is capable of managing Postgresql blobs:

my $o = .. and instance of My::App ..
my $blob_id = $o->pgblobs_store_blob($binary_content);
print $o->pgblobs_fetch_blob($blob_id);

A couple of guidelines:

Blobs in Postgresql are just numeric Object IDs. You WILL have to store them in a OID column for later retrieval. If you don't, there is no guarantee they will persist, as any unreferenced blob can get vacuumed away.

You MUST wrap any blob operation in a transaction.

Happy blobing!

Email::Postman - Yet another email sending package

I know I know, there is Email::Sender, and Mail::Sendmail, the 'send' method of MIME::Lite, Mail::Mailer  and probably other ones I don't know about on the CPAN and also the good old pipe to /bin/sendmail trick.

Each of them have their advantages, but none of them actually does what I wanted.

Email::Postman does all the following:

- It can send anything that's compatible with Email::Abstract (MIME::Entity, Email::Simple,  rfc822 String, etc..)

- It will manage multiple To, multiple Cc and multiple Bcc nicely.

- It will report on each of the recipients individually.

- It will speak to MX servers directly, according to the recipient's domain, so you don't need to manage a separate MTA to send emails.

- It has got a straight forward interface:



my $postman = Email::Postman->new({ hello => 'my-domain.com',
                                   from => 'postmaster@domain.com' });

my $email = ## any Email::Abstract compatible email.
            Email::Simple->create(
              header => [
                From => 'casey@geeknest.com',
                To => 'drain@example.com',
                Subject => 'Message in a bottle',
               ],
              body => '...',
            );

my @reports = $postman->deliver($email);

The only thing you need to be careful about is that depending on the responsiveness of the distant MX servers, calling deliver can be slow. So ideally you want to use it through some asynchronous mechanism. Gearman is a very popular one.

Give it a go!

Jerome.