Thoughts On Catalyst, SOA and Web Services

SUMMARY

Reflecting on recent Perl Blog's article covering Catalyst and Service Bus Architecture in light of recent Catalyst changes and current / future Catalyst development goals.

CONTENT

A big shout out to Kahill Hodgson, who recently published:

http://blogs.perl.org/users/kahlil_kal_hodgson/2013/08/a-catalyst-service-bus-from-scratch.html

which is strong both on service bus theory and actual code technology (well worth a read if you've not seen it yet, in fact, go read it right now before even continuing if you haven't).

Those of you that know me, know that I favor Catalyst for this use case, even though Catalyst has a reputation for being a heavyweight choice (and often people writing these kinds of services have high performance in mind). Although I do see a long term evolution of Catalyst moving toward a more modular, possible middlware oriented core, given how great Catalyst is for managing large applications and providing long term code maintainability I do believe it should be a contender for your services related projects.

Here's some new features added to Catalyst this year that I think help strengthen it for the web services use case:

HTTP Method Matching

One key aspect of good services design is to build them in line with how HTTP works, following the key constraints and principles outlined in Roy Fielding's dissertation (http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm). Doing so allows you to leverage everything that made the web great in the first place. One constraint of REST is to have a uniform interface for interacting between the layers of the network stack. This general interface defines how components in your stack interact, greatly enhancing visibility and simplicity in your design, while promoting strong decoupling between elements of your architecture.

In HTTP, one aspect of the uniform interface constraint is that all access to URI addressable resources is mediated via HTTP Methods such as GET, POST, PUT, etc. Each HTTP Method has clearly defined semantics and the total number of available methods is constrained enough such as to promote the goals of simplicity and network visibility (although it should be noted that REST does not specify any particular set of HTTP Methods, only that the allowed set must be uniform and semantically understood across the full network stack, which generally constrains us to the common ones when building service architectures that are intended to work widely and over a public network).

Earlier this year HTTP Method matching landed in core permitting you to write Controllers like this:

    sub base :Chained('/') PathPrefix CaptureArgs(0) { }

        sub chained_get :Chained('base') Args(0) GET {
            pop->res->body('chained_get');
        }

        sub chained_post :Chained('base') Args(0) POST {
            pop->res->body('chained_post');
        }

        sub chained_put :Chained('base') Args(0) PUT {
            pop->res->body('chained_put');
        }

        sub chained_delete :Chained('base') Args(0) DELETE {
            pop->res->body('chained_delete');
        }

This feature is handy not only for web services, but is also really useful for classic form processing, (making it so you can avoid url structures like "/login/show_login_form", "/login/process_login_form").

Should you need (or find it wise) to use Uncommon HTTP Methods, you are allowed to use the paramterized subroutine attribute Method like so:

    sub myspecial_action :Local Method("SomethingSpecial") { ... }

Although as previously noted, you might prefer not to do this.

A few limitations of the existing approach include the fact that the Catalyst dispatcher will return an Action based on a 'first to look correct' decision tree, which means that you need to place your catchall actions lasts (least they match first):

    sub check_default :Chained('base') CaptureArgs(0) { }

      sub default_get :Chained('check_default') PathPart('') Args(0) GET {
          pop->res->body('Found GET');
      }

      sub default_post :Chained('check_default') PathPart('') Args(0) POST {
          pop->res->body('Found POST');
      }

      ## Note that we place the 'match any other HTTP Method' as the last
      sub chain_default :Chained('check_default') PathPart('') Args(0) {
          pop->res->body('chain_default');
      }

Another limitation of the Catalyst Dispatcher makes it difficult for us to automatically populate the Response HTTP Header 'Allow' based on your declared allowed HTTP Methods. For now if you need to set this manually in your controller actions.

Support for nonblocking I/O

Not specifically a REST constraint, but often when building these types of service architectures we care a lot about performance and the ability to handle as many concurrent connections as possible. In support of this goal earlier this year we build into Catalyst basic support for running your application inside common event loop. We currently support AnyEvent, and expect to support IO::Async in the near future.

This support in Catalyst is considered early access, and its feature set is minimal (no suger, no helpers). None the less I think it is ready for people to start beating on it and letting the core team know where improvements need to be made. There's some examples on Github, including basic streaming and even a few websockets examples:

https://github.com/jjn1056/Perl-Catalyst-AsyncExample

Parsing Common incoming Request Body Content

This one is currently baking in the latest Catalyst development release 'Hamburg' which you can grab from CPAN:

    cpanm --dev Catalyst

For a long time Catalyst would parse and provide helper methods on the Catalyst Request object for standard HTML form 'application/form-data' fields, via the body_parameters method. However nowadays it is common to use JSON when building advanced AJAX enabled websites and for services oriented architectures. Starting in the next release of Catalyst, we will build in JSON parsing, and make it easy for you to declare application wide parsing of alternative request body data. Here's a taste of things to come:

    sub test_json :Local {
        my ($self, $c) = @_;
        my $message = $c->req->body_data->{message};
    }

The new method body_data is attached to the Request object, and will contain a Perl ready version of incoming body content. There's a default body data handler for JSON, which does what people mostly expect (it takes a JSON string and converts it to a Perl data structure). But you can define your own Handler's and easily override existing ones via the new global configuration key 'data_handlers':

    package MyApp::Web;

    use Catalyst;
    use XML::Simple;

    __PACKAGE__->config(
      data_handlers => {
          'application/xml' => sub {
            local $/; return XMLin $_->getline;
          },
      });

Parsing is done globally at the Request level, which I know might not alway be ideal, but it is the way Catalyst has traditionally worked, so lets start with this and perhaps down the road we'll work toward letting Controllers inform the parsing process (and for people that REALLY need this, you can continue to use the stuff that comes with Catalyst::Action::REST, which does give you that).

The Future: Catalyst and negotiated Request / Response

That concludes it for Catalyst features in stable or in development that you can use right now for your services oriented code. However there's some ideas around improving Catalyst's ability to perform content negotiation (another key but often overlooked constraint of REST) that have been spoken about and talked about on IRC and on the mailing list. Some of the ideas are proposals with the intention of building a version suitable for sending to the next development release; others are still in question and various possibilities are still under discussion. But here's some of those ideas for you to ponder. Hopefully it will inspire your comments and desire to help the Catalyst development team share these features in the best way possible.

Request content type matching

One thing we'd like to have is the ability to dispatch actions based on the incoming Request content type. This is another key aspect of REST and is what allows you to build flexibly code that can handle various types of request in a straightfoward way. For example, you might have a URL endpoint that supports both HTML and JSON requests. In the case of HTML, you want to return a full HTML webpage, but the JSON request only returns some basic meta data. We could provide this via subroutine attributes in the same way we do HTTP Method matching and Argument matching:

    sub base :Chained('/') PathPrefix CaptureArgs(0) { }

      sub as_html :Chained('base') Args(0) Consumes('HTML') { ... }
      sub as_html :Chained('base')  Args(0) Consumes('JSON') { ... }

This would make it easier to make sure you constrain your actions that expect to deal with JSON to the correct incoming request body content type.

Negotiated Response

This one is a bit trickier. Generally when I see people providing JSON view endpoints, for AJAX enabled webpages or services, I see code like this:

    use JSON::MaybeXS;

    sub as_json :Local {
      my( $self, $c) = @_;
      $c->res->body(
        encode_json(
          $c->model('DB::Entity')->get_row
        ));

Personally I don't like the "View Controller" approach, although I understand the seductiveness of the simple approach. Its great for demos and for when the requirement is very simple (or the endpoint is totally under the programmer's control). However I find that it doesn't age well over time since it overly couples your domain data objects with your view requirements. Additionally once you need to introduce a bit of display logic into the JSON rendering, you end up with ugly code (or with view logic in unexpected places.) So kudos to:

http://blogs.perl.org/users/kahlil_kal_hodgson/2013/08/a-catalyst-service-bus-from-scratch.html

for suggesting using a real view for returning JSON. Personally I'd take it a step further and use a real templating system such as Template Toolkit or another to actually build the JSON response, but I guess I'll talk more about the whys of that another time (yes, I make TT views like this one):

    {[% FOREACH names %]
      "[% key %]": "[% value %]"[% loop.last ? '':','%]
    [% END%]}

However the real issue that we face in developing Catalyst for services is not if you use Catalyst::View::JSON or something else. The issue is how do we best work with the RESTful constraint regarding negotiating the response content type. Just to quickly recap this, a properly designed REST system would examine the request desired and understood content types and try to return what is wanted, to the best of the servers ability to do so. Unlike the request Body, this is something that clearly is owned by the controller. Two (possibly complimentary) approaches suggest themselves. The first is to let the action define the possible responses (this follows the approach similar to Ruby on Rails and other popular frameworks).

    sub myaction :Local {
      my ($self, $c) = @_;
      $c->stash( ... ); # populate the stash with data
      $c->response->body_format(
          'text/html' => 'View::HTML',
          'application/json' => sub { encode_json +{ %data } },
      );
    }

Basically the idea here is that the action says, "I know how to return these content types, please review the request and pick the best one for me." The approach is nice and simple, is would be familiar to developers who have used some other popular frameworks. It has the downside that it might lead to some repeated code, and it isn't so declarative. One could say it is overly binding the action to the application at large. Additionally, since the application can't introspect the action's defined responses, there is no way to have the dispatcher know which actions handle which incoming content type requests.

We could introduce a some new action subroutine attributes, similar to what we did with the proposal to dispatch based on incoming request content types:

    sub html_json :Path('mydata') Provides('HTML','JSON') {
      my ($self, $c) = @_;
      $c->stash( ... ); # populate the stash with data
      $c->response->body_format(
          'text/html' => 'View::HTML',
          'application/json' => sub { encode_json +{ %data } },
      );
    }

    sub as_xml :Path('mydata') Provides('XML') {
      my ($self, $c) = @_;
      $c->stash( ... ); # populate the stash with data
      $c->response->body_format(
          'application/xml' => 'View::XML',
      );
    }

Now the Catalyst application would know that the 'html_json' matches requests for the HTML and JSON and that 'as_xml' matches XML for the same URL endpoint. However this feels a bit verbose (even for Catalyst :) ) and the current dispatcher design does not make it easy to properly negotiate the response, it would follow the 'first match wins' approach. And of course you still have to setup the corret HTTP response headers for allowed content, just like you did with HTTP Methods, and that feels even worse to me. So at this point these features are still proposal status and there is not clear excitement about the direction. It is not clear if this approach is really in line with the Catalyst way, and it doesn't make it easy to do the right thing for the rules of content negotiation. Perhaps what we are learning here is that we need to double down on refactoring the Catalyst dispatcher system before we can really touch this stuff?

What else?

Well, I spoke a lot about content negotiation and content types, but actually content negotiation covers much more, including language and encoding, but I will leave that for another time (least I get to many people telling me my blogs are 'tl;dr'..

I want to give another thanks and ++ to Kahill Hodgson. Thanks for the great post, thanks for sharing it with the Catalyst and Perl community at large and thanks for reminding me I need to write all this stuff down.