Catamoose

Recently I hijacked some time from two leading Perl developers who are currently working on the Moose port of Catalyst.  Although this is a project that greatly excites me, I realize such a deep change can lead some current developers feeling fear, uncertainty and doubt.  So this is the first of an open ended series of interviews and articles about this new direction for Catalyst.  It is my hope to:

1. Cure any fears users of the current version of Catalyst may have about this upgrade.
2. Try to let developers of existing plugins, controllers, etc., know what they might need to change in order to achieve full compatibility (surprisingly little).
3. Serve as an additional communication forum between the core developers and everyone else regarding this project.  So this will supplement the mailing list (catalyst@lists.scsys.co.uk and catalyst-dev@lists.scsys.co.uk, see links before for information about signing up) and IRC (irc://irc.perl.org#catalyst and (irc://irc.perl.org#catalyst-dev).

What this series is not going to be is a general introduction to Catalyst, Moose and Perl (see http://catalystframework.org/, http://moose.perl.org/ and http://perl.org/ for help getting started.)

What is Catamoose?

<groditi> Catamoose, Cat-Moose is the informal codename given to the branch of Catalyst-Runtime 5.80 using the Moose framework. In due time we should be able to just refer to it as Catalyst::Runtime.

When the first version of Catalyst was released (CPAN tells me this was back in January of 2005), there was no strong consensus as to the best Object Oriented framework for large scale applications.  Perl's default support for OO development is minimalistic and although many packages for smoothing out some of it's rougher edges exist, most oriented around solving very basic problems, such as simplifying instance accessors creation or clarifying ambiguity with inheritance.  Even worse, with so many different solutions available, our community was fragmented, making it hard to share knowledge and the best practices that come with experience.  

Moose, a postmodern object oriented development platform (see: http://search.cpan.org/dist/Moose and http://moose.perl.org/ ) offers an elegant solution to all these problems and a bunch you probably haven't even thought of.  It has a very active support community and a growing ecosystem of best practices.    

Let's hear a little more about Moose.

What’s the value of using Moose as the underlying framework for Catalyst development?  

<konobi>  Moose is awesome... It really does most of the work for me and lets me get on with doing the interesting parts, plus it allows me to do much more.  It's a little slower, but we've been working on that and it's pretty good these days.

Just a quick note about Moose and speed.  Since this interview took place, there have been two releases of Moose, each offering speed improvements of approximately 20%-25%.

<groditi> The inspiration has been there for over a year.  Various parties have voiced their interests in using Moose alongside other frameworks because of the syntactical niceties Moose provides, the rich meta introspection API and a feature set that allows us to offload certain operations to Moose, reducing code base size, complexity, and features we need to test. Over the last year Moose has matured significantly and with the arrival of new features such as Traits, an enhanced Roles API as well as important fixes and improvements  with regards to immutable classes, which are essential for runtime performance. While Matt Trout and I had spoken extensively about the port, when I found out Scott McWhirter (konobi) was taking the initiative to start the port I decided to join the project.

If you are interested in learning more about the features of Moose you should really check out the website (http://moose.perl.org) and follow some of the excellent presentations.  Then read through the documentation and tutorials at http://search.cpan.org/dist/Moose/ to  learn the details.  You can also review the Moose source code from that link or view the excellent test suite for more examples.  

What major gains would completing this project offer to the community?

<konobi> I found that catalyst in terms of its guts were hard to learn and get a full grasp of how it all worked, I think Moose will help make the process a bit more transparent and hopefully easier to understand. It should also allow component/plugin/extensions to be a lot simpler.

<groditi> a leaner and more maintainable code base with access to the powerful introspection features provided by Moose and Class::MOP.

I think Konobi really hit the key problem here.  With the lack of depth in existing OO frameworks, most developers take a 'roll it yourself' approach that often made it difficult for new members to join an existing project.  Additionally, the shallowness of the tools did not easily enable a programmer to always choose the best practice.

Class::MOP (http://search.cpan.org/dist/Class-MOP) is the core Meta Object Protocol that Moose is built on top of.  Class::MOP was originally inspired by Perl6 and CLOS (http://dev.perl.org/perl6 and http://en.wikipedia.org/wiki/Common_Lisp_Object_System).

Can you talk to how you’ve gone about developing this port?  For example, are we going to see all new code in Catalyst, generally compatible with the existing API, or are you mostly updating the most promising/useful bits to use Moose?

<groditi> We have decided that in order to make the release successful we would do it across several stages. We are now approaching the end of the first stage where we are using Moose to replace Class::Accessor::Fast. We have also simplified ISA hierarchies through the use of roles and are performing some lower level operations (such as changing @ISA) through Moose / Class::MOP. In the future we will more fully utilize the features of Moose, like Roles, but at the moment we are mostly committed to maintaining 100% back-compatibility.  Almost every class in the Catalyst-Runtime distribution has now been updated to use Moose (Catalyst::Test and Catalyst::Stats being the exceptions) and we are in the process of using Class::MOP instead of manually performing some lower level tasks.  Walking through the code base line-by-line has forced us to look at a lot of seldom-seen code and we have found areas where improvements may be made. No radical changes have been proposed and the changes are mostly limited to the internals of Catalyst.

The mentioned Perl module Class::Accessor::Fast (http://search.cpan.org/dist/Class-Accessor) is used extensively in the existing release version of Catalyst.  It is a tool to make it easier to create simple classes and to define instance accessors; however those accessors cannot be type constrained nor easily introspected.  Moose based accessors have a lot more features.  Since so much of Catalyst uses Class::Accessor::Fast creating a compatibility layer is a must in order to make existing applications work.

Groditi also mentioned Roles (Moose::Role) and Class::MOP in his response.  Roles can be used when you want to abstract some class behavior, but that behavior cuts across more than one Class hierarchy.  They are a clean way to abstract your functionality.  Class::MOP  is the underlying Meta Object Protocol that Moose is built upon.  See  http://search.cpan.org/dist/Moose/lib/Moose/Role.pm for the basic documentation on Roles and the previously listed links for more detailed explanation and tutorials.  See http://search.cpan.org/dist/Class-MOP for documentation for Class::MOP.

So your goals with this first stage would be to preserve as much compatibility as possible?

<groditi> We are aiming for 100%. At the moment the branch passes 100% of the original test suite, but we are looking forward to developers helping us find any bugs and helping us extend the coverage.

If you want to help out, join the discussion on IRC at “irc://irc.perl.org#catalyst-dev”.

One of the worries about using Moose for the underlying framework for Catalyst is that it has the perception of being slow.  Care to speak to that?

<groditi> There is some associated cost with the building of the meta objects at startup time, as well as the cost of loading Moose, Class::MOP and their dependencies. As with any module, there is going to be a performance penalty with loading and executing more code, but Moose's immutability features allow us to minimize this cost. Moose accessors are as fast as Class::Accessor::Fast and we are moving from NEXT to Class::C3, which has an XS component. We are confident that compared to the application's code the runtime overhead from using Moose will be negligible. We have as of yet been unable to lower the initial startup cost associated with Moose, but MooseX::Compile is promising and could deliver very welcome speedups in the future.  I often find the additional startup time to be unnoticeable compared to the rest of catalyst  startup, particularly if DBIC is involved, which also has a startup cost for large databases.

Please see my above notes regarding speed.

What should plugin authors and developers for projects built on top of Catalyst know in order to make their code compatible?

<groditi> Module authors should be ready phase out NEXT and use Class::C3 instead. We have provided a Class::Accessor::Fast compatibility layer which is available on CPAN, but moving to the Moose attribute syntax would likely benefit them.

 <konobi>  Currently I'm not 100% sure, we've mainly been concentrating on doing things _right_ rather than compatibly.  The current 5.7x test suite is what we have been testing against and so far that's all good.  Unfortunately, the test suite doesn't capture the full API or usage that plugins, etc use, so we can't really tell very well.  Brownie points for extension authors who fill out the 5.7x test suite with the features they need.

The modules NEXT and Class::C3 are used to improve the way Perl does Method Resolution, which determines how method calls are delegated in a class with inheritance.  Catalyst used NEXT which was a good choice for its time, but the newer methods are better.  Please see the documentation for Class::C3 at http://search.cpan.org/dist/Class-C3 for more information.

Matt Trout added: “DBIx::Class went NEXT -> C3 for 04 and got not only performance boosts but much cleaner code, so that's why we knew it would be good for Catalyst”.

Where is Catalyst going in the future, with Moose and more?

<groditi> As far as a longer term roadmap, I originally envisioned using a role-based plugin system, but we are exploring using Stevan Little's Bread::Board as a way to provide extensibility to Catalyst apps, once again, the core developer team is better prepared to answer questions about future architecture decisions. [Additionally] We are working on what is informally known as the app/context split. This is a significant change in how the internals work, but should be a generally seamless change for most users. This change, which is essentially composed of splitting the application and context classes into two distinct classes will allow us to eliminate more code, increase flexibility, and probably performance as well as providing cleaner namespace boundaries. The core developers can once again comment more on his architectural decision.

Moose Roles are a good fit for plugins, and will hopefully clarify some of the problems we used to have with people creating Catalyst plugins when that functionality was better off created as a base controller or model.  Bread::Board (http://search.cpan.org/dist/Bread-Board/) is a Moose based Inversion of Control system, which can assist with
developing large applications by reduce the complexity of object creation.  Bread::Board is actually Steven Little's second IOC framework for Perl. For more information about IOC and how it can benefit application development, please see ”http://en.wikipedia.org/wiki/Inversion_of_Control”.  Or you can read the documentation for Bread::Board and review the test cases.

Matt Trout clarified the possible use of Bread::Board in a future version of Catalyst: “Bread::Board is going to replace the custom code in setup_components, it's -not- a replacement for the plugin system. Although we may extend Bread::Board to compose class+N roles classes for extensions, sort of like a more large-scale MooseX::Object::Pluggable.”

The mentioned module, MooseX::Object::Pluggable (http://search.cpan.org/dist/MooseX-Object-Pluggable) is a system to make it easy to extend your core object with 'plugins' that can be added at runtime and are configurable by the developer.

Konobi had more to say on the project to clarify application classes from context classes.

<konobi> Currently a lot of information is stored in catalyst classes rather than objects... while it made sense at the time, we've learnt and played and discovered that it restricts us in a lot of ways. A lot of the restrictions are not knowing if we are dealing with an "object" or a "class" as the context and the application are the same thing at the moment.  It means that things that are supposed to be context specific are being played with in the application and vice-versa.  If we move to having "application" and "context" objects, it allows us to interact, extend and modify everything from the bottom up and even give us the ability to run more that on catalyst app in a single interpreter.

For Catalyst, the context refers to a particular request being served, while the application refers to the overall container for your Controllers, Models, Views, etc.  Since in the current release version of Catalyst your context object (that's the second argument you see in your actions, typically we call it '$c') refers to both the application and your current dispatched request context, there can be confusion about what methods are available to $c at what time.  Splitting this should help people understand the code and usage better.

Why personally are you involved in this project?

<groditi> I have been a big fan of the Moose framework for a long time and we are heavily using it's features in Reaction, a set of extensions for Catalyst spearheaded by myself and Matt Trout. We found ourselves really wishing we had access to MOP, which is what led to me finally diving in and joining Scott (Konobi).

<konobi>  I have some spare tuits and I thought doing moosification of catalyst would be a great way to spend some time refactoring for fun (possibly profit) and learn the guts of catalyst.

Reaction is an advanced framework for representing user interfaces and domain models which uses Catalyst for rapidly creating web applications.  As of this moment, it has not been released to CPAN, however you can track the repository at http://code2.0beta.co.uk/reaction/svnweb/index.cgi/reaction or join in the discussion at "irc://irc.perl.org#reaction".

What can the catalyst community do to help?

<konobi>  More tests (of the unit testing variety) and general testing of the branch is needed... failing unit tests would be awesome. Send stuff to the catalyst-dev list.

To join in I recommend joining the Catalyst developers mailing list and our IRC forums. There are two IRC channels, one for developers (irc://irc.perl.org#catalyst-dev) and another one for general Catalyst questions (irc://irc.perl.org#catalyst).  There  are also two mailing lists, broken along the same lines.  My experience is that IRC is much more active and is where all the hot discussions occur, but here's the list information:

• http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
• http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst-dev

Who are Groditi and Konobi?

<groditi> groditi is a Boston, MA Perl hacker currently working with Moose Catalyst and DBIx::Class.

<konobi> I'm Scott, I'm an Irish perl hacker living in Vancouver, BC.   I enjoy hacking on apps and the like, I really enjoy refactoring, doing performance testing and improvements and general useful stuff.

Thanks to the both of you and for all the anonymous help on IRC.  Discussion?  What do you want to know more about?  Should I talk about Moose?  Or would you rather hear details about new technologies like Bread::Board?  Need a demonstration of how to convert a pre-Moose Catalyst plugin to take advantage of cool Moose features?  Speak up!

Latebreaking News!: Looks like the Catamoose branch was merged back to the 5.80 trunk repository.  We are one step closer to shipping this, so I recommend downloading it and seeing how it works with your application.  You can see this at: http://dev.catalyst.perl.org/repos/Catalyst/Catalyst-Runtime/5.80/trunk/

EDIT: Turning off comments to reduce SPAM