Product Development, Prototyping and the Adventure on Monkey Island

Title

Product Development, Prototyping and the Adventure on  Monkey Island

Summary

On the value of coding against a use case, testing, and how to move things along when you are stuck.

Content

Pretend for a moment you are one of several script writers working on a new action heroine movie. You've just completed a hair raising scene where your actor battles her arch nemisis aboard a flaming airplane high above the South Pacific Ocean. Forced to parachute to safty, she lands on what is to all appearances a deserted island.

Excellent, but now what? Where does the plot go? How does she get off the island and move on to the next big adventure?

Sometimes when working on a project you get stuck. If you are writing a screen play, or a book, you might get writer's block. If you are an athlete or a sports team, you get a run of subpar performance that seems to just go on and on. And as a programmer who is trying to improve the system he or she is working on, we can legimately get stymied in our progress with no obvious way forward.

Screenwriters have a trick they sometimes use which I have personally found equally applicable to both programming and product development. It goes a little like this. No matter what the problem you have, there's 1000 monkeys that can solve it in some way! So for example you could say that a tribe of monkeys jump out of the trees and assembles a boat on which your protagonist is able to leave the island in safety.

It doesn't need to be monkeys (although monkeys are fun). It can be any sort of dirty hack that gets you past the point where you are currently stuck. This often frees you to actually continue the story in a meaningful way. That way, later on, you or one of your fellow script writers can come along and iterate on your 'monkey' solution. That's why the solution needs to be at least remotely plausible, since it needs to be something that a followup can iterate on and improve. For example, you might later on come back and say that the boat idea isn't bad at all, but instead of monkeys jumping down from the trees and building a boat, we can say the heroine finds the remains of boat which has been beached in a hidden cove and although very old appears sound.

The key thing here is that we have a hack that advances the plot and is good enough that you can iterate on it and improve it and that it is interesting enough that it can provoke discussion amongst the team in a way that might lead you to conclusions and discoveries you might otherwise have missed. Like in the hidden boat example, we could say that this is a pirate boat that is haunted by evil spirits and the protagonist needs to overcome them in order to get access to the boat. Or perhaps as it turns out she is the great, great granddaughter of the pirate captain. In any case, you have all sorts of opportunities for plot twists and moments of self discovery.

In application development I find the working prototype neatly fits the same sort of solution that the monkey horde does for script writers. For example, as someone who hacks on Catalyst core, there's a number of challenges that I find blocking. Like what is the best way to incorporate RESTful content negotiation and more flexible parsing of POST/PUT content bodies? Or how can we better leverage the Plackness we've baked into Catalyst? Or what can we do to take better advantage of Catalyst's built into service discover and extend some meaningful dependency injection concepts into mainline development?

All these problems have various possible solutions but there is no clear method to decide which one is best using pure reason or theory alone. This is why an empirical investigation, the non trivial prototype, can assist you in evaluating an approach and in provoking discussion about your discovery. For this to be valuable this prototype should meet the following constraints:

• It needs be a non trivial use case. Personally I'd choose something that is a well known and possible modeled use case already, so that you minimize 'bike shedding'.

• It needs to provoke meaningful discussion by indicating what is missing so that you can do further experiments and investigations.

• It needs test cases so that you can iterate on the code solution and improve it without breaking your use case.

• Mad hacks are allowed if it allows you to demonstrate something useful.

For Catalyst and in prototyping some of the stuff I worked on for the zombie- boomstick release, I chose to work on a simple Todo style appliction and to base my models on http://addyosmani.github.io/todomvc/ which is a javascript gallery of this application done in various frameworks. I also decided that in order for the prototyping to be sufficiently complex, that I would provide both a 'classic' server side appliction as well as an more modern Ajaxified version. This way I would be sure to flog out the code in various ways and hopefully reveal any missing pieces and also to better provoke discussion.

The application lives over at github:

https://github.com/jjn1056/CatalystX-Example-Todo

and it is not complete both in its use case and in its testing. It does have basic functionality (its a todo list and you can add and remove elements, but it currenly does not yet track status or allow any useful sorting of the elements. However I thought to present a taste of it as it does show off some of the ideas I've been speak of and perhaps a few of you will get excited enough to step in and start sending some code patches!

So here's a taste and a bit of a walkthrough. Next time I will put down in more detail some of the underlying reasons I choose this approach, but for now the code is all there on Github for you to check out, or pester me on IRC!

This controller provides json API services to the ajax version of the Todo application. Basically lets you get a list of your current tasks via json and add new ones and delete old ones. It probably represents a not altogether unfamilar job we as modern web developers working on the server side face.

Here's the full commit link if you'd prefer to see that:

https://github.com/jjn1056/CatalystX-Example-Todo/blob/e54f8ef54506cc6c258b8aad69d76ffa8e2ff7be/lib/CatalystX/Example/Todo/Web/Controller/API.pm

And the (minimal) walkthrough:

    package CatalystX::Example::Todo::Web::Controller::API;
     
    use Moose;
    use MooseX::MethodAttributes;
    use Catalyst::ResponseHelpers;
    no warnings::illegalproto;

    extends 'CatalystX::Example::Todo::Web::Controller';

So far this is basically just the normal controller preamble. The only bits that might confuse you off is that we are using some non standard modules (both the custom Catalyst::ResponseHelpers which is part of this repository and supports the prototype for the response domain specific language, and you might not be familiar with warnings::illegalproto which is what is allowing use abuse subroutine prototyes further along in the code.) Ultimately all that would likely be incorporated into a new version of the Catalyst controller such that you'd avoid the need for this boilerplate, but for the purposes of the prototype its probably better to leave this magic in plain view.

    sub start : ChainedParent Provides(JSON)
       PathPrefix CaptureArgs(0) { }

A standard chained action. The only thing new here is the Provides method attribute. This is telling the dispacher that endpoints under this action will return JSON content (application/json) and that this should match what the requesting client is asking for (via HTTP Header Accepts).

      sub todolist(Model::TodoListViewBuilder<Model::Schema::TodoList>)
       : GET Chained('start') Args(0)
      {
        my ($self, $todolist) = @_;
        Ok json +{todolist => $todolist};
      }

Ok, so now there's wonky stuff here that you've not seen. First of all, what is going on with that method prototype? And where's $c? Well, here's the idea. Wouldn't it be better if an action could declare all its dependencies and let the framework just provide them, rather than having to reach into the context over and over just to get access to things like your models, body parameters, and all that stuff? You'd replace code with declaration, reduce tight binding between your action and the application all while encouraging programmers to think in terms of reusable models instead of thick and code heavy action bodies.

So instead of an action always getting the controller instance (nearly always useless given how anemic Catalyst controllers are) the context (which leads to bloated boilerplate and encourages tight application bindings in your actions) and the list of path arguments (can be useful but tends to lead to a lot of repeated code that marshalls database objects based on those argument), we say that you can declare what that action wants to get instead. In this case I am saying provide to me a TodoListView model that is suitable for sending to a JSON renderer. That model is also parameterized and we can fulfill those parameters as well, which leads us to have more reusable models.

Ok, fair enough, but what's with that last line:

    Ok json +{todolist => $todolist};

Basically what we have here is a domain specific langauge for constructing the HTTP response. Read this like "HTTP 200 OK, content type JSON encoded data".

So, if you know me, you know that I find domain specific languages a bit suspicious. Often they tend to be more limiting then the lines of saved code is worth. In this case I am wondering if the use case is sufficiently small enough that a good DSL is meaningful. Also, I am wondering if we could not use the return value of an Action to good use (classic Catalyst has generally left the return value of an action as meaningless).

Part of my thinking here is to let the action author have more clear control over the http status and content body render then we easily have with traditional Catalyst and how we just stuff the stash and allow a end action in Root deal with it using something like Catalyst::Action::RenderView (which I have alway personally found a little too little control between the Controller/Action and the view).

BTW, here's a 'classic Catalyst' version of this action for the purposes of comparison.

      sub todolist : GET Chained('start') Args(0)
      {
        my ($self, $c) = @_;
        my $todolist = $c->model('Schema::TodoList');
        $c->stash(todolist => $todolist);

        $c->detach
      }

So you can see we've saved some bloated boilerplate code, reduced tight structural binding between our action and the overall application (making things like unit testing easier, for example) and moved repeated logic to a more central location, such as to promote DRY. BTW, this example is not really exactly the same since we are passing the raw DBIC resultset to the view, a practice that is common but causes me to cringe!

Next action!

      sub add(bparams, Model::FormNewEntry<Model::Schema::TodoList>)
       : Chained('start') POST PathPart('todolist') Args(0)
         Consumes(JSON) BodyParser('Any')
      {
        my ($self, $params, $form) = @_;
        my $result = $form->run($params);

        if($result->validated) {
          Ok json +{entry => {$result->form->item->get_columns}};
        } else {
          NotAcceptable json {errors => $result->errors}
        }
      }

So this action handles HTTP POST of JSON data to add new Todo elements. In the method attributes you will see two proposed method attributes Consumes and BodyParser. Consumes just says this action only matches if the POSTed content type matches (in this case we only match on JSON). So this is the inversion of the Provides attribute we spoke of earlier.

BodyParser allows you to control at the action level how that POSTed body is converted into something that Perl can understand. Right now Catalyst can only parse classic HTML posted form fields, which really seems quaint for 2013. Also, this parsing is handled by the global request object and doesn't not encourage one to create content specific parsers. BodyParser prototypes a way to allow one to both handle new and emerging content types as well as let you control how that type is created and stored for your actions.

We've already discussed using the method prototype signature as a way introduce dependency injection into the action. In this case we are using an HTML::Formhandler form to validate our incoming JSON and to create if needed a new entry into the list of Todo items. We also show off a bit of what using the DSL for creating responses can do for action authors. In this case I think it lets the author exercise a bit more control over exactly how the response is generated.

Another thing you might notice is that I am requesting the parsed body parameters as the first argument to be send to the action. Compare this to:

  my $bparams = $c->request->body_parameters;

One thing that I'm not happy with here is how I need to reach deep into the HTML::Formhandler object to get the created database row.

    $result->form->item->get_columns

Which feels to me like evil structural binding. I've proposed to the Formhandler group an abstraction on their model to better encapsulate this, so lets see if that works out.

and in brief the last action for deleting todos:

      sub delete(Model::Todo<Model::Schema::TodoList,Arg0>)
       : Chained('start') DELETE PathPart('todolist') Args(1)
      {
        my ($self, $todo) = @_;
        if($todo) {
          $todo->delete;
          return SeeOther UriOf 'tasks/list';
        } else {
          return NotFound;
        }
      }

    __PACKAGE__->meta->make_immutable;

The highlight of this action for me is how we construct the redirect. In classic Catalyst code I'd probably have:

    my $url = $c->uri_for_action('tasks/list');
    $c->response->redirect($url, 303);
    $c->detach;

Again, I don't love DSLs but this seems to be shaping up into something that I hate a lot less than the full code alternative!

So that's the 'taste' I'll be writing more and hopefully speaking on in the hallway track about it, particularly as I write more and come closer to completion of the prototype. There's a lot going on here, and I doubt I well explained all my thinking that went into the design choices. And of course as we finishe the prototype things will change as we meet limitation in the design.

Got thoughts, critics, constructive abuse? Comment away!