Modern Perl

Help Me Write the next Perl Catalyst Community Poll!

Hello Catalyst Developers!

Its getting to be that time of year again.  Here in the Northern Hemisphere we are trending toward the Fall and with the waning of 2013 I find myself reflecting on what has been done, not done and if its all ok really.  In light of that I'd like to sent around a new Poll, similar to the one we did a little more than a year ago.

To help remind you, here's a link to that poll from 2012.

For 2013 I want to add a few question, and possible drop some.  So, here's your chance to help me shape the poll!  What questions would you like to be asked?  What questions would you like answers from your peers?  Which questions should definately be asked again and what should be dropped or otherwise altered?

Somethings I am thinking about...

I'd like to drill down into the whole 'barriers to contribution' thing.  And I'm trying to find some creative ways to engage our diverse community.  For examples, all my blogs are in English, and I wonder if that is hurting...   Or perhaps we should get together leaders from companies that are using Catalyst and see if we can convince them to give their programmers some time to kick in?

Ultimately I am trying to figure out if there's tasks out there people are into seeing done enough, or is the roadmap just not interesting and compellng.

Some stuff I'd like to ask about...

As we consider what changes the next major release of Catalyst will bring, (call it Cat6 or Cat7, whatever), there's a growing thought that we should take the opportunity to do some cleanup and removal of long deprecated or disused stuff.  This of course would represent the first version of Catalyst to proposal possible significant backward compatibility problems.  One thought I have is to have a 'Catalyast5.9 "virtualmachine"' or similar embedded into Catalyst6 architecture so that you could transistion in parts, similar to how Apple embedded support for pre OSX.  I think this could be done since Catalyst is now through and through PSGI (I would envision the pre Catalyst6 layer as middleware).  However  I am not sure the best way to present the idea and would love to get early 'pre poll' feedback on the thought.

Thanks everyone.  Participation is mandatory ;)

Thoughts On Catalyst, SOA and Web Services

SUMMARY

Reflecting on recent Perl Blog's article covering Catalyst and Service Bus Architecture in light of recent Catalyst changes and current / future Catalyst development goals.

CONTENT

A big shout out to Kahill Hodgson, who recently published:

http://blogs.perl.org/users/kahlil_kal_hodgson/2013/08/a-catalyst-service-bus-from-scratch.html

which is strong both on service bus theory and actual code technology (well worth a read if you've not seen it yet, in fact, go read it right now before even continuing if you haven't).

Those of you that know me, know that I favor Catalyst for this use case, even though Catalyst has a reputation for being a heavyweight choice (and often people writing these kinds of services have high performance in mind). Although I do see a long term evolution of Catalyst moving toward a more modular, possible middlware oriented core, given how great Catalyst is for managing large applications and providing long term code maintainability I do believe it should be a contender for your services related projects.

Here's some new features added to Catalyst this year that I think help strengthen it for the web services use case:

HTTP Method Matching

One key aspect of good services design is to build them in line with how HTTP works, following the key constraints and principles outlined in Roy Fielding's dissertation (http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm). Doing so allows you to leverage everything that made the web great in the first place. One constraint of REST is to have a uniform interface for interacting between the layers of the network stack. This general interface defines how components in your stack interact, greatly enhancing visibility and simplicity in your design, while promoting strong decoupling between elements of your architecture.

In HTTP, one aspect of the uniform interface constraint is that all access to URI addressable resources is mediated via HTTP Methods such as GET, POST, PUT, etc. Each HTTP Method has clearly defined semantics and the total number of available methods is constrained enough such as to promote the goals of simplicity and network visibility (although it should be noted that REST does not specify any particular set of HTTP Methods, only that the allowed set must be uniform and semantically understood across the full network stack, which generally constrains us to the common ones when building service architectures that are intended to work widely and over a public network).

Earlier this year HTTP Method matching landed in core permitting you to write Controllers like this:

    sub base :Chained('/') PathPrefix CaptureArgs(0) { }

        sub chained_get :Chained('base') Args(0) GET {
            pop->res->body('chained_get');
        }

        sub chained_post :Chained('base') Args(0) POST {
            pop->res->body('chained_post');
        }

        sub chained_put :Chained('base') Args(0) PUT {
            pop->res->body('chained_put');
        }

        sub chained_delete :Chained('base') Args(0) DELETE {
            pop->res->body('chained_delete');
        }

This feature is handy not only for web services, but is also really useful for classic form processing, (making it so you can avoid url structures like "/login/show_login_form", "/login/process_login_form").

Should you need (or find it wise) to use Uncommon HTTP Methods, you are allowed to use the paramterized subroutine attribute Method like so:

    sub myspecial_action :Local Method("SomethingSpecial") { ... }

Although as previously noted, you might prefer not to do this.

A few limitations of the existing approach include the fact that the Catalyst dispatcher will return an Action based on a 'first to look correct' decision tree, which means that you need to place your catchall actions lasts (least they match first):

    sub check_default :Chained('base') CaptureArgs(0) { }

      sub default_get :Chained('check_default') PathPart('') Args(0) GET {
          pop->res->body('Found GET');
      }

      sub default_post :Chained('check_default') PathPart('') Args(0) POST {
          pop->res->body('Found POST');
      }

      ## Note that we place the 'match any other HTTP Method' as the last
      sub chain_default :Chained('check_default') PathPart('') Args(0) {
          pop->res->body('chain_default');
      }

Another limitation of the Catalyst Dispatcher makes it difficult for us to automatically populate the Response HTTP Header 'Allow' based on your declared allowed HTTP Methods. For now if you need to set this manually in your controller actions.

Support for nonblocking I/O

Not specifically a REST constraint, but often when building these types of service architectures we care a lot about performance and the ability to handle as many concurrent connections as possible. In support of this goal earlier this year we build into Catalyst basic support for running your application inside common event loop. We currently support AnyEvent, and expect to support IO::Async in the near future.

This support in Catalyst is considered early access, and its feature set is minimal (no suger, no helpers). None the less I think it is ready for people to start beating on it and letting the core team know where improvements need to be made. There's some examples on Github, including basic streaming and even a few websockets examples:

https://github.com/jjn1056/Perl-Catalyst-AsyncExample

Parsing Common incoming Request Body Content

This one is currently baking in the latest Catalyst development release 'Hamburg' which you can grab from CPAN:

    cpanm --dev Catalyst

For a long time Catalyst would parse and provide helper methods on the Catalyst Request object for standard HTML form 'application/form-data' fields, via the body_parameters method. However nowadays it is common to use JSON when building advanced AJAX enabled websites and for services oriented architectures. Starting in the next release of Catalyst, we will build in JSON parsing, and make it easy for you to declare application wide parsing of alternative request body data. Here's a taste of things to come:

    sub test_json :Local {
        my ($self, $c) = @_;
        my $message = $c->req->body_data->{message};
    }

The new method body_data is attached to the Request object, and will contain a Perl ready version of incoming body content. There's a default body data handler for JSON, which does what people mostly expect (it takes a JSON string and converts it to a Perl data structure). But you can define your own Handler's and easily override existing ones via the new global configuration key 'data_handlers':

    package MyApp::Web;

    use Catalyst;
    use XML::Simple;

    __PACKAGE__->config(
      data_handlers => {
          'application/xml' => sub {
            local $/; return XMLin $_->getline;
          },
      });

Parsing is done globally at the Request level, which I know might not alway be ideal, but it is the way Catalyst has traditionally worked, so lets start with this and perhaps down the road we'll work toward letting Controllers inform the parsing process (and for people that REALLY need this, you can continue to use the stuff that comes with Catalyst::Action::REST, which does give you that).

The Future: Catalyst and negotiated Request / Response

That concludes it for Catalyst features in stable or in development that you can use right now for your services oriented code. However there's some ideas around improving Catalyst's ability to perform content negotiation (another key but often overlooked constraint of REST) that have been spoken about and talked about on IRC and on the mailing list. Some of the ideas are proposals with the intention of building a version suitable for sending to the next development release; others are still in question and various possibilities are still under discussion. But here's some of those ideas for you to ponder. Hopefully it will inspire your comments and desire to help the Catalyst development team share these features in the best way possible.

Request content type matching

One thing we'd like to have is the ability to dispatch actions based on the incoming Request content type. This is another key aspect of REST and is what allows you to build flexibly code that can handle various types of request in a straightfoward way. For example, you might have a URL endpoint that supports both HTML and JSON requests. In the case of HTML, you want to return a full HTML webpage, but the JSON request only returns some basic meta data. We could provide this via subroutine attributes in the same way we do HTTP Method matching and Argument matching:

    sub base :Chained('/') PathPrefix CaptureArgs(0) { }

      sub as_html :Chained('base') Args(0) Consumes('HTML') { ... }
      sub as_html :Chained('base')  Args(0) Consumes('JSON') { ... }

This would make it easier to make sure you constrain your actions that expect to deal with JSON to the correct incoming request body content type.

Negotiated Response

This one is a bit trickier. Generally when I see people providing JSON view endpoints, for AJAX enabled webpages or services, I see code like this:

    use JSON::MaybeXS;

    sub as_json :Local {
      my( $self, $c) = @_;
      $c->res->body(
        encode_json(
          $c->model('DB::Entity')->get_row
        ));

Personally I don't like the "View Controller" approach, although I understand the seductiveness of the simple approach. Its great for demos and for when the requirement is very simple (or the endpoint is totally under the programmer's control). However I find that it doesn't age well over time since it overly couples your domain data objects with your view requirements. Additionally once you need to introduce a bit of display logic into the JSON rendering, you end up with ugly code (or with view logic in unexpected places.) So kudos to:

http://blogs.perl.org/users/kahlil_kal_hodgson/2013/08/a-catalyst-service-bus-from-scratch.html

for suggesting using a real view for returning JSON. Personally I'd take it a step further and use a real templating system such as Template Toolkit or another to actually build the JSON response, but I guess I'll talk more about the whys of that another time (yes, I make TT views like this one):

    {[% FOREACH names %]
      "[% key %]": "[% value %]"[% loop.last ? '':','%]
    [% END%]}

However the real issue that we face in developing Catalyst for services is not if you use Catalyst::View::JSON or something else. The issue is how do we best work with the RESTful constraint regarding negotiating the response content type. Just to quickly recap this, a properly designed REST system would examine the request desired and understood content types and try to return what is wanted, to the best of the servers ability to do so. Unlike the request Body, this is something that clearly is owned by the controller. Two (possibly complimentary) approaches suggest themselves. The first is to let the action define the possible responses (this follows the approach similar to Ruby on Rails and other popular frameworks).

    sub myaction :Local {
      my ($self, $c) = @_;
      $c->stash( ... ); # populate the stash with data
      $c->response->body_format(
          'text/html' => 'View::HTML',
          'application/json' => sub { encode_json +{ %data } },
      );
    }

Basically the idea here is that the action says, "I know how to return these content types, please review the request and pick the best one for me." The approach is nice and simple, is would be familiar to developers who have used some other popular frameworks. It has the downside that it might lead to some repeated code, and it isn't so declarative. One could say it is overly binding the action to the application at large. Additionally, since the application can't introspect the action's defined responses, there is no way to have the dispatcher know which actions handle which incoming content type requests.

We could introduce a some new action subroutine attributes, similar to what we did with the proposal to dispatch based on incoming request content types:

    sub html_json :Path('mydata') Provides('HTML','JSON') {
      my ($self, $c) = @_;
      $c->stash( ... ); # populate the stash with data
      $c->response->body_format(
          'text/html' => 'View::HTML',
          'application/json' => sub { encode_json +{ %data } },
      );
    }

    sub as_xml :Path('mydata') Provides('XML') {
      my ($self, $c) = @_;
      $c->stash( ... ); # populate the stash with data
      $c->response->body_format(
          'application/xml' => 'View::XML',
      );
    }

Now the Catalyst application would know that the 'html_json' matches requests for the HTML and JSON and that 'as_xml' matches XML for the same URL endpoint. However this feels a bit verbose (even for Catalyst :) ) and the current dispatcher design does not make it easy to properly negotiate the response, it would follow the 'first match wins' approach. And of course you still have to setup the corret HTTP response headers for allowed content, just like you did with HTTP Methods, and that feels even worse to me. So at this point these features are still proposal status and there is not clear excitement about the direction. It is not clear if this approach is really in line with the Catalyst way, and it doesn't make it easy to do the right thing for the rules of content negotiation. Perhaps what we are learning here is that we need to double down on refactoring the Catalyst dispatcher system before we can really touch this stuff?

What else?

Well, I spoke a lot about content negotiation and content types, but actually content negotiation covers much more, including language and encoding, but I will leave that for another time (least I get to many people telling me my blogs are 'tl;dr'..

I want to give another thanks and ++ to Kahill Hodgson. Thanks for the great post, thanks for sharing it with the Catalyst and Perl community at large and thanks for reminding me I need to write all this stuff down.

Perl Catalyst "Hamburg" Development Release 002 on CPAN

Hi there!

Last night we released the second development version of Perl Catalyst "Hamburg", which is the current development release cycle. Highlights include fixes to the PSGI middleware integration that was release in 001 and a new ability to define custom request body data handlers so that you can graft into Catalyst the ability to parse PUT / POST request bodies other than the classic form-data and multipart. We also included one default body data parser to parse application/json, meaning that you no longer need additional modules to do basic JSON ajax with Catalyst. For example:

    package TestDataHandlers::Controller::Root;
    use base 'Catalyst::Controller';
    sub test_json :Local {
        my ($self, $c) = @_;
        $c->res->body($c->req->body_data->{message});
    }
    1;

Would define the controller and here's a test case:

    use JSON::MaybeXS;
    ok my $message = 'helloworld';
    ok my $post = encode_json +{message=>$message};
    ok my $req = POST $c->uri_for_action('/test_json'),
       Content_Type => 'application/json',
       Content => $post;
    ok my $response = request $req, 'got a response from a catalyst controller';
    is $response->content, $message, 'expected content body';

The new request method 'body_data' is lazy, so it won't attempt to parse anything unless you ask for it. Adding custom parsers is easy. You just add content type => parser subref to the Catalyst application, for example:

    package MyApp::Web;
    use Catalyst;
    use JSON::MaybeXS;
    __PACKAGE__->config(
      data_handlers => {
        'application/vnd.user+json' => sub {
            ## $_ is localized as a filehandle object of the PUT/POST request body
            local $/; decode_json $_->getline;
        },
      },
    );
    __PACKAGE__->setup;

Body data parsers are global for now since this feature is not intended to compete with a full on approach like Catalyst::Action::REST. If the feature is useful, perhaps in the future we'll add the ability to let controllers define local body data parsers.

Unlike some frameworks, we purposefully distinguish between classic form parameters, which is certain to be key values, and body data, which is not (although often is, in the case of JSON AJAX use cases).

I want to thank everyone that participated in the email thread regarding how this might work.

Ok, so that's the new development release. Go on, download it, trying, complain now before it goes gold!

    cpanm --dev Catalyst::Runtime

If you want to get involved with Catalyst Development, be sure to hangout on the Perl Catalyst IRC channels:

    irc.perl.org#catalyst
    irc.perl.org#catalyst-dev

And check out the open quests over on Questhub:

    http://questhub.io/realm/perl/explore/latest/tag/hamburg

I know contributing to Catalyst can feel daunting because the code is not the most accessible right now, but there's definitely a handful of small tasks that I'm certain a moderately aware programmer could handle with a bit of mentoring, as well as plenty of opportunity to test, blog and do general advocacy. So what's holding you back? Scared!? :) Go ahead, I dare you!

More to come...
Jnap

Just noticed the Perl Catalyst repository on Github hit 100 stars recently! Thanks everyone for helping raise awareness about how awesome Catalyst is, and for the vote of confidence this gives all the core team developers and contributors!

https://github.com/perl-catalyst/catalyst-runtime/stargazers

Ok, now onwards to 200!

Perl Catalyst - better built in Content Negotiation and support for more Content Types

The questhub quest related to improving Content Negotation in Catalyst continues to be popular, but I've been hesitant to put this in a development cycle quest since I know its a pretty big subject and I'd hate to get it really wrong in core.

I had a bit of time on a few plane rides recent to get a chance to write down all the various thoughts and ideas I've heard around what could be a minimally useful expansion of how Catalyst handles content types beyond classic form data, etc as well as some features to make it easier to use Catalyst for building web services.

These are new core features not intended to compete so much with existing tools like Catalyst::Action::REST but instead to provide some minimal and hopefully correct support out of the box, to help beginners find enough in Catalyst to stick with it. And hopefully better core support will make it easier to improve specific tools (like CAR).

I wrote it up in pod format and put it out on the github organization for any and all to comment, add tickets to, fork and send me corrections, etc.

I don't specifically see this as targeted for the Hamburg release, but most of it I think is straightforward, so I'd be happy to help mentor anyone that wanted to run with it.

https://github.com/perl-catalyst/CatalystX-Proposals-REStandContentNegotiation

John