Blobs in Postgresql with Perl

Are you using, or do you want to use Postgresql Blobs with Perl?

Providing you are in the Moose ecosystem, here is Pg::Blobs. Pg::Blobs is a role that adds blobs handling methods to any Moose based package.

Here is how to use it:

package My::App;
use Moose;
with qw/Pg::Blobs/;
# Just provide:
sub pgblobs_dbh{ .. return the DBH .. }
1;

Now your package My::App is capable of managing Postgresql blobs:

my $o = .. and instance of My::App ..
my $blob_id = $o->pgblobs_store_blob($binary_content);
print $o->pgblobs_fetch_blob($blob_id);

A couple of guidelines:

Blobs in Postgresql are just numeric Object IDs. You WILL have to store them in a OID column for later retrieval. If you don't, there is no guarantee they will persist, as any unreferenced blob can get vacuumed away.

You MUST wrap any blob operation in a transaction.

Happy blobing!

Get a B::Deparse piggy back through Gearmany

Gearman is a great tool to run asynchronous and/or distributed jobs over a cluster of machines. It's not a full general message queue ala RabbitMQ, but a rather minimalist piece of software that is very simple to use, does only one thing and does it very well.
To use gearman from your favourite language, the cpan provides two modules. The original Gearman and the more recent Gearman::XS. For this post we're going to use the original Gearman, but you should be able to implement the example using Gearman::XS without much trouble.
In a nutshell, writing for gearman is a two sides process, writing some client code and some worker code to actually do the job, with gearman just sitting in the middle and distributing the jobs:

Client(s) <--> Gearman Deamon <--> Worker(s)

Vanilla gearman

Let's have a quick pseudo code look at an asynchronous example inspired by the Perl package Gearman:

Worker:

 my $worker = Gearman::Worker->new(...);
 $worker->register_function('sum' => sub {my $job = shift; calculate sum of decoded($job->arg) and store it somewhere });
 $worker->work();

Client:

 my $client = Gearman::Client->new(...);
 my $handler =  $client->dispatch_background('sum', encoded([1, 2, 3]), { uniq => some string unique enough });
 ## wait for the job to be done
 while( my $status = $client->get_status() && $status->running() ){
   sleep(1);
 }
 .. Retrieve sum from the storage and print it ..

By the way 'encoded' and 'decoded' are entirely up to you, as long as they encode to bytes and decode from bytes. I personally use JSON, but it's a matter of taste (and performance but that's another story).
Also, I deliberately skip the task sets and synchronous mechanisms as this is not supposed to be a gearman tutorial :)
So here we go. You make sure that your gearman daemon is running, you fire up your worker script and while it's running, each time you run your client, it will print 6.
You feel great. You've written your first minimalist gearman application and you're ready to gearmanize the rest of your long running and/or easily distributable code.

Why this approach is a pain

So following the example, the temptation is great to just extend the worker like that:

 $worker->register_function('my_long_specific_thing', \&gm_do_long_specific_thing);

I guess that if you're reading this post, you probably already have something in your model code that is long and already packed into a function:

  package MyApp::Object;

  sub do_long_specific_thing{
    my ($self, $arg1 , ... ) = @_;
    ...
  }


So in reality your gearman specialised 'gm_long_specific_thing' will probably look like that:

sub gm_long_specific_thing{
  my($job) = @_;

  my $args = decoded($job->arg());

  my $application = .. Build or get application ..;

  $application->get_object( build object getting from args )->do_long_specific_thing( build arg1 , arg2 from the args);
}

Then you know the rest of the story. Every time you need something else to be gearmanized or every time you need to make a change to the arguments of one of your gearmanized method, you have to propagate your changes to the specific gearman registered functions. Your code has become a bit less maintainable, just because you want the benefits of gearman.

Fixing it

But what if.. you could write something like that:

$client->application_launch(sub{ my $app = shift;
                                  my ($oid) = @_;
                                  $app->get_object($oid)->do_long_specific_thing(1, 'whatever');
                                },
                              [ $oid ] );


No more worker specific code to write. Everything is done from the application's point of view, leaving the back-end details out of the way.
Want to change the API of do_long_specific_thing? No problem, just apply parameter changes where they appear in the code. No more headaches propagating the API change through the gearman specific methods.
Want to make your gearmanized process longer without changing do_long_specific_thing? No problem:

$client->application_launch(sub{ my $app = shift;
                                  my ($oid) = @_;
                                  $app->get_object($oid)->do_long_specific_thing(1, 'whatever');
                                  .. and something else ..
                                },
                              [ $oid ] );

What if you have another long thing to gearmanize? Well, you get the picture:

$client->application_launch(sub{ my $app = shift;
                                  my ($oid, $arg1) = @_;
                                  $app->get_object($oid)->do_another_thing($arg1);
                                  .. and something else ..
                                },
                              [ $oid, $arg1 ] );

Implementing it

The client 'application_launch' method

Thanks to B::Deparse, we can turn any sub into a plain string. The rest is trivial, so

here we go:

## In some object that wraps the $gearman_client
## you can also inherit if you prefer. But I like wrapping more, cause you can store utilities.
sub application_launch{
   my ($self, $code, $args ) = @_;

   $code //= sub{};
   $args //= [];

   my $gearman_client = $self->gearman_client() OR just $self;
   my $deparser = B::Deparse->new('-sC'); ## C style
   my $json = JSON::XS->new()->ascii()->pretty(); ## I like pretty. I know it's larger but well..

   my $code_string = $deparser->coderef2text($code);

   my $gearman_arg = $json->encode({ code => $code_string,
                                     args => $args });

   my $uniq = Digest::SHA::sha256_hex($gearman_arg);

   my $task = Gearman::Task->new('gm_application_do', \$gearman_arg, { uniq => $uniq });

   my $gearman_handler =  $gearman_client->dispatch_background($task);
   unless($gearman_handler){
    confess("Your task cannot be launched. Is gearman exposing the gm_application_do function?");
   }
   return $gearman_handler;
}

And that's pretty much it.

The worker 'gm_application_do' code

The worker code is very similar.

$worker->register_function('gw_aplication_do', sub{ _gm_application_do($application, shift) });
sub _gm_application_do{
    my ($app , $job) = @_;

    my $gm_args = $json->decode($task->arg()); ## Note you need a $json object.
    my $code_string = $gm_args->{code};
    my $code_args = $gm_args->{args};
   
    my $code = eval 'sub '.$code_string;
    $code || confess("EVAL ERROR for $code_string: ".$@); ## That shouldnt happen but well..

    ## This is not supposed to return anything, as we call that asynchronously.
    &{$code}($app, @$code_args);
    return 1;
}

Adapting to synchronous calls

Adapting that to synchronous calls is quite straight forward.
Remember gearman exposed functions should always return a single scalar.

Conclusion

gotchas:

• This doesn't work with closures, so really your sub's should be pure functions and all parameters should be given as such.
• As far as the magic goes, the parameters can ONLY be pure Perl structures; something that's serializable in vanilla JSON.
• If you try to pass bless objects, bad things will happen.

Disadvantages:

• Insecure. What if anyone injects sub{ destroy_the_world(); } in your gearman server. That's kind of easily fixed. Just implement some secure signing of the code in transit.
• No strict control of what can be done through gearman. Developers enlightenment is the key here.
• No strict control about what 'flavour' of gearman worker is running your code. Some people like to have specialised gearman workers exposing only a subset of functions. This can easily be fixed by adding a 'target' option to the application_launch method and exposing the same general purpose gm_application_do under different names on different machines. But again, choosing the right target falls under the developers responsibility.

To sum up the advantages:

•  Stable gearman worker code that's decoupled from the application code itself.
•  Flexible gearmanization of any application code you like.
•  Clarity of what's going on. No more parameter encoding/decoding to write, and no API change propagation through what should be infrastructure only code.

Perl offers us the flexibility that empowers us to clearly separate code that deals with different concerns.
As developers, we should take advantage of it and build reactive, flexible and generic enough business components.
As an infrastructure developer, I don't really know what people are going to do with gearman, nor should I care too much. This approach let me concentrate on what is important: the stability, scalability and the security of my gearman workers.
As an application developer, I don't care that I have to encode my functions parameters in a certain way. And I don't want to bother changing code in some obscure gearman module when I make changes to my business code. What I want is to use gearman as a facility that helps me design the best possible application without getting on my way too much.

Hope you enjoyed this post.

Until next one, happy coding!

Jerome.

Give your application a shell (and never write any ad-hoc scripts again)

As developers, we sometimes have to help operations going smoothly by fiddling with the data "by hand" because there's no GUI to allow people doing some rare and obscure things. One common way of doing it is by connecting to the DB and writing some SQL. But often, accessing your database is not enough, because you need your application code to be run.

And for that, you need a shell.

I mean a Perl application shell

Perl offers a great module to implement this: Devel::REPL . By the way, if your using a debian based distribution and if you're wondering about the quickest way to install a CPAN module, CPAN ↔ Debian is the site you want !

Let's say your application is called 'My::Bob'. In this example, it's fully implemented in the shell script itself, because well, it doesn't do much.

So here it is, the magic bob.pl shell for your My::Bob application:


use strict;
use warnings;
package My::Bob;
use Moose;
## Application bob just holds a name and a couple of methods.
has 'name' => ( is => 'rw', isa => 'Str' , default => 'Bob' );
sub hello{
 my ($self) = @_;
 return 'Hello, my name is '.$self->name();
}
sub count_to{
 my ($self, $limit) = @_;
 if( ( $limit // 0 ) < 1 ){ confess "limit should be >= 1"; }
 return map{ $_ } 1..$limit;
}

package main;
## The shell code starts here
use Devel::REPL;

## You probably want to skip output buffering
$| = 1;
# Build an instance of your application.
our $BOB = My::Bob->new();

print "Hello, this is bob\n";


my $repl = Devel::REPL->new;

## Inject the instance of bob in the shell lexical environment.
$repl->load_plugin('LexEnv');
$repl->lexical_environment->do(q|my $bob = $main::BOB ;|);

## Tell something useful in the prompt
$repl->load_plugin('FancyPrompt');
$repl->fancy_prompt(sub {
 my $self = shift;
 sprintf ('Bob (%s) > ',
 $BOB->name()
 );
});

## Various autocompletion.
$repl->load_plugin('CompletionDriver::LexEnv');
$repl->load_plugin('CompletionDriver::Methods');
$repl->load_plugin('CompletionDriver::INC');

## Allow multiline statements.
$repl->load_plugin('MultiLine::PPI');

# And run!
$repl->run();

print "\nThanks for using Bob. Bye\n";

Now, let's look at a session:


$ perl bob.pl 
Hello, this is bob
                                                                                                                                                        
Bob (Bob) > $bob->hello(); ## Call any method.
Hello, my name is Bob                                                                                                                                   Bob (Bob) > $bob->name('Bobbage'); ## Change of name is reflected in the prompt
Bobbage                                                                                                                                                 Bob (Bobbage) > $bob->count_to(3) ## Arrays are printed
1 2 3

Bob (Bobbage) > my $count_hello = sub{ my ($n) = @_;  ## Define multiline subs                                                                                                                                                       re.pl(main):005:1* map{ $bob->hello() } $bob->count_to($n);                                                                                                                            re.pl(main):006:1* }

CODE(0x2fcbea0)
Bob (Bobbage) > &$count_hello(3) ## And call them
Hello, my name is Bobbage Hello, my name is Bobbage Hello, my name is Bobbage

Bob (Bobbage) >  ## CTRL-D ends the session
Thanks for using Bob. 

Hope this short example was useful to you. If you can build your application as an object, writing your own shell should be straight forward. Also you might want to add some credential checking before you let anyone playing with the guts of your application.

Until next time, happy coding!

Jerome.

What Javascript programmers should learn from Perl mistakes

Although I spend most time developing stuff in Perl, sometime I have to fix Javascript code that's been written by JavaScript specialists. Sorry, ECMA script as it should be called. And the more I look at 2013 JavaScript programming, the more I can see some similarity with the state of Perl programming 12 years ago.

Remember the Perl obfuscated code contest? Basically it was all the "serious" Perl programmer boasting about being able to write the most f...d up original version of Yet Another Perl Hacker. While this was quite fun and made quite a good showcase of Perl's flexibility and expressiveness, some Perl programmers of the time were not quite serious enough and started to (unwillingly ?) use obfuscation techniques in production code, gaining Perl a reputation of being unreadable and unmaintainable, thus not fit for serious industrial development. You know the rest of the story: PHP, the offspring of 1994 Perl took over.
As programmers, we know that you can write unreadable and unmaintainable code in any language, and you can certainly write elegant things in any language too. Although I'm not sure about brainf.ck. The problem lies in the fact that non-programmers easily confuse the Programming language itself and the nasty habits of programmers. Have enough people write bad *any language* code, even for fun, and *any language* will gain a bad reputation. It comes from the extraordinary appetite our lazy human brains have for generalisation. Show enough retarded programs on TV and very soon people will think that the TV media itself is retarded.
So please, Javascript programmers, stop writing stuff like that:

var layout_options = {}, optgroups = {
                    globals: 'Public',
                    userones: 'Private'
                };
Or like that:
var block = conf[i], options, sel_conf, k, j;
Or like that:
opt.lytId == cfg.actLyt_id && opt.lytType === cfg.actLytType && (option_conf.selected = true);

Maybe some prehistoric browsers will actually run that faster than if you were using the 'var' keyword sensibly or if you were using good old 'if' statements. Otherwise, I think it's safe to assume these are completely useless optimizations, paid at the high price of unreadability. Plus, you know, text editors have auto-completion, so it's useless too to save keystrokes on variable names, specially if your code goes though some optimizer/packager in production.
Maybe I shouldn't rant about Javascript programming, maybe stuff like CoffeeScript or Dart will take over and it JavaScript will only live as an assembly language. Personally I have the feeling it would be a good thing. A higher level - cleaner language compiled to JavaScript taking all browsers specificities into account. Why not? I don't see anything wrong with that.
But JS programmers, if you love JavaScript and want to make sure it's still used in its current form over the next few years, stop writing code that's only barely readable by your fellow gurus.

Edit: Actually, someone has pointed out to me that Perl is still more popular than PHP in terms of Job offers.

Perl, Python, Php, Ruby trends

Perl jobs | Python jobs | Php jobs | Ruby jobs

Although PHP easily wins on Google fight.

Friday Time Waster: Watch the world go by in ASCII with Reuters' API

I've recently released WebService::ReutersConnect. It's a Perl modules that interfaces with the ReutersConnect's API in OO style. To demonstrate it and hopefully entertain you on this Friday, here's how to use it to watch the world go by in glorious ASCII and from the comfort of your command line. To put it shorter: The perfect Friday Time Waster.

The ingredients

To cook this recipe, you will need:

- WebService::ReutersConnect

- Text::AAlib

While you can install all of this with the CPAN command, I suggest you install most of their dependencies through your OS packaging system first.

If you're debian based, here are the debs to install: libmoose-perl libdatetime-perl libdatetime-format-iso8601-perl libtest-fatal-perl libxml-libxml-perl libwww-perl liblog-log4perl-perl liburi-perl liblibaa1-dev

Then just install WebService::ReutersConnect and  Text::AAlib via your favourite channel.

The Recipe

The Recipe is pretty straight forward, it goes like this:

Preamble. We just load the required packages and also make sure that Log4perl is not going to complain about lack of initialisation:

#! /usr/bin/perl -w                                                                                                                                                                             

use strict;

use warnings;

use WebService::ReutersConnect qw/:demo/;

use Log::Log4perl qw/:easy/;

use Text::AAlib;

use Imager;

Log::Log4perl->easy_init($WARN);

Building the $reuters object and querying the freshest image. Here we use the demo credentials and search for the latest picture:

my $reuters = WebService::ReutersConnect

 ->new({ username => REUTERS_DEMOUSER,

         password => REUTERS_DEMOPASSWORD

      });

my ( $item )  = $reuters

            ->fetch_search({ limit => 1,

                             media_types => [ 'P' ],

                             sort => 'date' });

Building the ASCII of the picture. We just download the preview URL and render it in ASCII using Text::AAlib:

## Load preview image                                                                                                                                                                              

my $res = $reuters->user_agent->get($item->preview_url());

unless( $res->is_success() ){

  die $res->status_line();

}

## Build and scale image                                                                                                                                                                           

my $bin_image = $res->content();

my $img = Imager->new( data => $bin_image , type => 'jpeg' ) || die Imager->errstr();

$img = $img->convert( preset => 'grey' );

$img = $img->scaleX(scalefactor => 2); ## Tweak to your taste

$img = $img->scale(scalefactor => 0.5); ## Tweak to your taste

## Build ASCII Version                                                                                                                                                                             

my ($width, $height) = ($img->getwidth, $img->getheight);

my $aa = Text::AAlib->new( width  => $width, height => $height );

Rendering the whole thing on the console. The rendering parameters are OK for me, but you might have to play with them depending on your taste/colour scheme:

## Print ASCII Image                                                                                                                                                                               

print $aa->render( dither => 0 ,

                   gamma => 1.83,

                   bright => 50,

                   contrast => 60,

                   color => 0

                 );

## And some info                                                                                                                                                                                   

print "\n    ".$item->date_created().' : '.$item->headline()." \n";

print "\n    ".$item->preview_url()."\n";

And that's it! Now you can put all of that together, or if you're lazy, you can just copy/paste the whole thing from this pastebin. 

Feel free to experiment with the options and the code and tell us about your improvement in the comments!

Happy coding!

Jerome.

Understanding Unicode and UTF8 in Perl

I know there's a lot of things out there on the subject, but when it comes to Unicode, a quick review of core concepts and bug avoidance guidelines is never a waste of time.


I'm not a Unicode Guru, but working with third parties, I often find that a lot of people consistently fail to get the basics right about Unicode and encoding. There must be something esoteric about it. So here's yet another set of slides about Unicode/UTF8 in Perl.

 It's not meant to be a comprehensive presentation of all Unicode things in Perl. It's meant to insist on a couple of guidelines and give some pointers to get a good start writing a unicode compliant application and avoiding common issues.

Comments are open for questions. Happy coding!

Yes, One, I don't know - three tales about understanding people

As application developers, it's crucial that we understand what other people are saying, and we're often in a position where misunderstanding is an easy trap to fall into, specially when we speak with people with no computer science background. Yes that happens. Scary hu? The key point when this happens is to forget about everything you know about basic understanding, even in your native language. So here are three tales about how even the most simple language elements can lead to misunderstanding, and what to do about it.




Yes

I started working for (very little) money  as a bio-informatics consultant. My main duty was to write and use software to produce - in a few hours of number crunching - enough hypothetical results that would take molecular biologists decades and millions of investors hard earned Euros to validate in the wet lab. At this time, my brain was fresh from university and I was formatted to think about the world in terms of 1's and 0's. And there were only 10 categories under which my mind was ultimately classifying things. True and false. So the word "Yes" for me had only one meaning. It meant something was true in the past, now and in the future, independently of circumstances. So as you can guess, there were not many questions you could answer with a plain "Yes" to me. For instance to the question "Do you like hummus", if you answered me "Yes", then I would have served you hummus for the rest of your life. Well only until you've finally decided by yourself to tell me that "actually yes I like hummus most of the time, but not always". Which is actually closer to the reality. If it just about culinary tastes, it's OK, but when it's about work then the misunderstanding is annoying at best. And I've made this mistake quite a few times: Taking a biologist "yes" for a computer science "yes" and mis-design something as a consequence. Now I try to remember, I don't take "yes" as an answer. When I hear "yes", I try to question it further, think twice and make sure I deal with the case where this "yes" is not true. Because in the real world, it will happen.

One

I bet that like me you've noticed the following fact about software development:

"Given enough time and business analysis, any unique relationship will eventually have to turn multiple."

It works for almost everything. Your users can set their preferences? One day you'll want to manage multiple set of preferences. You display the hottest deal of the day on your homepage? Quite soon you'll want multiple. A document has got a language property? What about mixed language documents? multiple. A user has got a postal address? No, they've got multiple addresses - work/home/mum/dad. OK so now they have multiple addresses, but they choose a favourite one. Humm, what if they want a different favourite depending on the context. Think "Deliver to work, but send the bill to home". Multiple again. Programming language has got single inheritance? The next one will have multiple. And you can go on and on about it. There's no end to multiplying. I wonder sometimes why unique relationships are included in UML. I think it's to trick developers into believing people saying "One". But don't be fooled. There's no such thing as "One". If you don't believe me, read this multiple times.

I don't know

This one is not strictly speaking about computer science, but about science in general. I think that people with a scientific background of any kind are used to deal instantly with equivalence and equations. For instance, given the speed and the mass of a moving object, we instantly know its kinetic energy because we know that E = 1/2mv^2. It also works transitively. I don't want to go into too complex examples, but let's say you have a,b,c, d=f(b,c) , e=f(d,a), and let's say I give you a,b,c and ask you if you know 'e'. Someone with science background will tell you "yep, because you know a and d, and you know d because you know b and c".  I did the test with my wife - who's got a biology background - yesterday, so my statistics are pretty solid. To me this is a perfectly natural way of thinking. Until I took a class in finance.

In an attempt to avoid spending my elderly years begging in the streets of London, I took this class in finance on coursera to understand how the wheels turn. And the lecturer looks to me like a brilliant man, full of common sense with very good pedagogy. I highly recommend this class. Anyway. One thing that stroke me is that sometimes I was kind of lost into the teacher's thinking. He was presenting a example, where a,b and c were known and was asking the question: "Do you know e". And the right answer was not "Yes of course, because e=f(f(b,c),a)". The correct answer to him was "I don't know". And I understood why I sometimes didn't understand him. Because for him , "I don't know" doesn't mean the same as my "I don't know". To me "I don't know" means "It cannot be reached within a reasonable computing time". To a mathematician or a physicist, or a biologist,  it will probably mean more or less the same. But to him it means something different. It means "I don't know it yet because I haven't calculated it yet". Once I understood that, the rest of the class was suddenly crystal clear, because I had understood that the meaning of this simple sentence "I don't know" meant something different between him and me.


The fact that we cannot assume that everybody shares the meaning of such simple concepts like "Yes", "One" , "I don't know" and probably lot of other ones is quite fascinating and tells us that effective communication between human beings of different backgrounds is probably one of the most difficult things to achieve at work and in life in general.

I don't know if there's a solution to this, but until people more clever than me find one, I'll try to stick to this rule: Don't speak to non IT people Try to understand what meaning others attach to basic phrases.

JS/HTML/CSS: A braindump from a backend guy

It's a wonderful sunny Saturday outside yet I'm here bloging about web development. I should see a doctor, but in the meantime please suffer my brain dump about things browser related.

I'm not really a front end developer, in the sense that I have very poor graphic design skills, that my knowledge of Javascript and CSS is limited. However over the time I had my fair bit of front end development, more recently here, not counting the things I do for fun, so yeah, I don't have expert level knowledge about all of this, but I feel I have enough experience to broadly think about some general guidelines about what should be fed to the browsers. So the goal of this post is not to dictate what I think is right, but rather to put some ideas on the table and calling for critics and alternative and/or improvement suggestion. So if you're a JS Ninja, or a CSS old-master, please be indulgent, and share your art in the comment.

Ok let's get started with the general page structure. I like a page source to be clean and organised (not like my bedroom), but also the goal here is to allow the browser to render the page as soon as possible. In other words, to reduce the time to first pixel:

<html>

  <head>

      <title>My Wonderful Page</title>

      <meta name="description" content="Some meaningful and reasonably long description">

      <script type="text/javascript" src="/js/head.js"></script>

      <link type="text/css" rel="stylesheet" href="/one/css/file.css" />

      

  </head>

  <body>

     <div>

         All sorts of HTML stuff.

     </div>

     <script type="text/javascript">

        

     head.js('https://ajax.googleapis.com/.../jquery.min.js',

                 'https://ajax.googleapis.com/../jquery-ui.min.js',

                 'http://mysite.com/js/site.js'

                );

    head.ready(function(){

      installFancyJS($(document));

    });

     </script>

   </body>

</html>

Let's go though this step by step.

Use a JS Loader. So the idea here is to do as little as possible in the header, and deport the heavy lifting to the end of the body. In your header, you should have only one and only one minimal script, and this script should allow you to load things later at your page bottom. Here I use head.js. It's got other fancy base features but load.js is also a good alternative if you like to make it even smaller. Let your page be pear shaped.

One CSS File. Not two, not three. One, and preferably on the same domain as your main site (so relative URLs are welcome here). Why is that? Here we reduce the number of extra requests the browser has to do to a minimum of erm ... one, and by having the CSS hosted on the same hostname, you avoid doing extra DNS queries, saving you precious milliseconds. Some people even argue that you should have your CSS in the page itself. I'm not sure about this one. It's true that it saves you the extra query entirely, but at the same time, your page is now heavier and therefore longer to download. There's probably a reasonable limit under which this actually saves time, but I'm not sure about its value. Of course in the development environment, you probably want to have your css neatly organized, so you don't have a giant ball of spaghetti monster to deal with. I've tried different techniques to achieve that, but didn't come with something I really like yet. Suggestions?

All sorts of HTML Stuff. Warning. Controversial bit. What? HTML stuff, controversial? Well yes, as it seems to me that with the advent of the uber-ubiquitous Ajax coolness, a large amount of people chose to ignore a fundamental fact of life (at least of life as a developer): Browsers were born to render HTML and dynamic web servers can generate HTML. For instance, I know a site where every time they need to display a table of something (let's say historical stock prices), they first display an empty table, and then, when all the page JavaScript is loaded, they make a second query to get the data to populate the table. It drives me mad. Not only I don't see the point* , but more importantly, you can imagine how it kills the 'time to useful pixel' of their pages. It's not that I'm against Ajax. not at all. I love it, I was doing it wayyyy before jQuery became popular. In fact you'll find here some posts I wrote about how to easily 'ajax' anything. But I kind of believe that if you want to do JS object models and get JSON from the server instead of HTML, you'll be much better off developing your backend as an API and generate your GUI using something like flex, or GWTK (please add yours to the list :)). I certainly reckon the incredible value and power of OO JS/Json for complex and specific information manipulation and display (down with flash and applets!!), but for general pages development, I prefer to stick with good old HTML, specially if the application is 'browsing' oriented.

One JS Function that does it all. I call it (observe my creative naming skills) 'installFancyJS', and it takes one argument: the jQuery** context it's going to used in. So what does this function do? Well, it installs callbacks and all sort of responsive code in the page just after it's loaded. More precisely in the given context. To give you a very minimal example:

function installFancyJS(context){

    installTopClick(context);

    ....

}

function installTopClick(context){

    $('div#topbar' , context).on('click', function(ev){

        $('html,body').animate({scrollTop: 0 },

                               { duration: 'slow', easing: 'swing'}

                              );

    });

}

The reason why I propagate the context everywhere is that when I get a new bit of HTML using Ajax, I want to be able to install the full set of Javascript features on the newly created slice of document. In here's an minimal example of Ajax callback that uses the function:

function(data){

                  var newbit = $(data);

                  container.replaceWith(newbit);

                  // install fancy js on the container's new content.                                                        

                  installFancyJS(container);

              }

That's is for now. I hope you enjoyed reading this and found some interesting/ridiculous/horrible things to talk about.

Jerome.

* But I see the reason: they got tied to the horrible datatables.net.

**yes I use jQuery, please rant and suggest in the comments if you know a better one.

The goodness of testing

Wall e-coyote testing a new rocket design

There’s a common view among programmers that testing (that is writing tests) is a painful thing you have to do to please your manager, or something you can avoid doing if your manager is not bothered. Let’s try to debunk that and show that testing is actually fun.

It guarantees code correctness. That’s obvious. Less obviously it forces you to think about what your code correctness actually is. By writing test, you will more easily think about those pesky edge cases that other users of your code will inevitably notice. Further techniques like property testing or mutation testing can also help automating thorough correctness.

It gives you a safety net when you refactor. Imagine you’ve finally decided to repay some technical debt and re-write your poorly written methods (don’t tell me you haven’t got any). If you’ve got good test coverage, no worries. You can refactor and rewrite your module, increase performance, cleanup poorly written code without worrying about breaking anything. At least anything that's tested.

It forces you to write better APIs. By putting you in the role of a user of your code/module/API, it forces you to think about the nicest way to use your module. This is creative, fun and interesting: How, as a client of your library, would you like to be able to use it. How would you like to interact with your piece of software? What do you expect from it? What should it expects from you? You decide, and because you're lazy and imaginative, you'll come to think about something neat. If anybody else uses your code in the future, you will attract a lot of developer love. Because people will go: "hum, that’s neat, and it works first time." That's lot of doughnuts for you.

You can see your test as a kind of contract about what is supposed to work now and forever more (biblical voice needed). It’s a contract between You, your future You, anyone who touches your code in the future, and also with the outside world. Why the outside world? I sometime think that everything that’s not tested doesn’t exists. It’s probably a philosophical view, but as a programmer, I like to think in binary terms. Something is there or something is not there. At least until quantum computers are on our desktops. So what about a feature that is not tested? It can break without anyone noticing, especially if it's a minor thing, or it can break and only the users notice (very bad). So is it there or not?

Well, if it can go away silently, I think it’s safer to assume it’s not there. Ask your sales team if they feel comfortable about selling stuff that can vanish at any time. Anything outside testing shouldn’t be assumed to be existing. That leads to what I deeply think about testing.

Testing does not help to make your product more reliable. Testing is your product.

Comments welcome!

A few difficult things in application development

Here are some thoughts I had about developing applications over the last months.

Maintaining documentation

Yeah I know what you think: nobody reads it, waste of time. People just click everywhere and see what it does. This is probably true for most of the geeks, but observing my parents using a computer taught me  you need to document your application features. At least the less obvious ones.

Error reporting

The same way you can judge a restaurant by the look of its toilets, you can judge an application by the way it deals with errors. It's often a taboo subject, but every single customer will use it sooner or later (sooner in general). Give your application a first class feel: Implement error catching/reporting sensibly.

Making stuff contextual

This is an empirical law of application development: Whatever the complexity of the system is, there will always be a client saying 'Nice but if <a potentially always false large predicate)>, then the behaviour should be <something crazy no one will understand in one month time>'. At this point, this is an occasion to:
- Add nasty 'if' statements to the appropriate palces and slowly but surely f***-up your beautiful codebase. The client will be happy.
- Or refactor your application to implement this new stuff in an elegant way. This will take longer. The client will be happy - eventually.

In short: You're doomed.

Moving from single to multiple

Another empirical law of application development: Whatever you think should be unique will eventually need to be multiple. If you think about it, in nature there's no absolute 1-1 relationships between entities but only probabilities differences. For instance the likelihood of you having one nose is higher than the likelihood of you having one arm. Design for multiple stuff from the beginning to save you some pain in the future.

Complying with unicode.

Complying with unicode is at heart a simple matter of understanding that 'an encoding of the data', 'the data itself' and the 'glyphs to represent it on a piece of screen|paper' are different things. Unfortunately this is still challenging to get for a lot of P* language programmers. If you're used to "print whatever" assuming it "just works" in your english language environment, it's very likely you're wrong.

More thoughts to come in the future!

Perl: Avoiding the HTML::Mason/TemplateToolkit schism

As a Perl web developer, I've been using HTML::Mason intensely for many projects. Over the past year, I've also been working with Template Toolkit as a rendering layer for a web application.

As always, choosing between two similar technologies seems to be mainly a matter of taste and yet another occasion for an holy war. To pacify the situation, let's try to figure out what are these beasts good at.

TT feels more simple.

Consider the following code:

[% IF customer.is_in('GB') %]
We'll ship your order by Royal Mail
[% ELSE %]
Sorry [% customer.firstname %], delivery will take a while.
[% END %]

It's much easier to understand for a non-programmer than the Mason equivalent:

% if ( $customer->is_in('GB') ) {
We'll ship your order by Royal Mail
% } else {
Sorry <% $customer->firstname() %> ...
% }

So here clearly, TT wins, although as a programmer, and specially as a Perl programmer, Mason also feels just fine. However, a designer or a configuration person will feel more at home with TT, as it resembles infamous visual basic excel macros.

Mason simplifies page composition.

Let's implement a set of pages, each with a specific title and content, but with the same structure (here just a navigation box). In fact the common bread of web development

Let's do it in HTML::Mason:

autohandler:
<html>
 <head>
  <title><& SELF:title &></title>
 </head>
 <body>
  <div id="nav_box">...</div>
  <div id="payload">
% $m->call_next();
  </div>
 </body>
</html>
<%method title>A default title</%method>

content1.html:
  <p>content 1</p>
  <%method title>Page Content 1</%method>

content2.html
  <p>content 2</p>
  <%method title>Page Content 2</%method>

etc..

Now let's do it TT:

page:
<html>
 <head>
   <title>[% title || 'A default title' %]</title>
 </head>
 <body>
 <div id="nav_box">...</div>
 <div id="payload">
  [% content %]
 </div>
 </body>
</html>

content1.html:
[%WRAPPER page
          title = 'Page content 1'
%]
 <p>content 1</p>
[%END%]

content2.html:

[%WRAPPER page

          title = 'Page content 2'

%]

 <p>content 2</p>

[%END%]

From a programmer's point of view, HTML::Mason wins, as its inheritance mechanism allows much better simplicity, elegance and flexibility in page composition. In this example, I only considered a simple example that's possible to implement both in Mason and in TT, but for more advance use, the inheritance mechanisms (with call to PARENT (super) methods), and other Mason-isms largely beat the TT WRAPPER directive.

❧

From these two simple examples, it appears that HTML::Mason has got much more to offer to programmers, as it allows them to use Perl natively, use a fairly complete inheritance mechanism, and much more. On the other hand, what TT has to offer is definitely an easier and simpler solution for simple templating jobs.

To sum up:

- Use Mason where you need elegance and simplicity to build complex things.

- Use TT where the 'easier than Perl' feel is important, with a restricted set of directive to build simple things.

Perl: Separating the persistence and the business model layers - Part 1

Today, thanks to modern Perl technologies a growing number of Perl project use DBIx::Class to access the database, and a MVC engine, such as Catalyst or other frameworks and a nice template engine to expose the application on the web. Usually, the framework tutorial assumes that you will use DBIx::Class to access and store your objects from your MVC controller. I think this approach is wrong for the following reasons:

- DBIx::Class is a perfectly fine ORM mapper but it's not a business model framework. You can add business code to the result Classes but what happens if you want to migrate some of your stuff to a NoSQL storage? Or change the way some stuff is stored in the DB. Things can and will get ugly.

- Dbic objects do not provide the flexibility you would need to implement common object oriented patterns. What if you want to use Roles, enforce interfaces, implement your own search facility? You'll eventually get so much application business code in your Dbic object that they will no longer look like dbic objects.

- Your web framework is directly fed with the ORM model, and because you want to code things quickly, you start putting business code in your controller, and this business code assumes you're using this or that ORM. And the rest will come. Web applications are difficult to test for regressions, especially if they use Ajax. Things will break without being detected before release, you will fix them quickly in your controller, and as time goes on, your shiny new application using modern Perl technologies will end up being as difficult to maintain as a good old CGI mud-ball.

- What if you want to turn your application into an API? Or if you want to make command line tools for quick administration, or for asynchronous runs? Well you can't. Because the your business logic is scattered everywhere, in all the layers of the application.

What we end up with this approach is our persistence layer and our MVC layer (and in the worst case our template layer) become polluted with a blob of sloppy hard to maintain business model code. Also the sad thing is that your business code IS your business' added value, so it shouldn't be considered polluting something else. I don't think MVC's and ORM's are designed to be the natural container of your business code. I think they should be kept, and loved, and used for what they are: an easy and fast way of making non-business related code thin and minimalistic.

Blob of business code longing for a home on its own

For the development of libsquare.net, I've designed a reasonably fast and easy solution to this problem. I'll be talking about it in the next part of this sequence.