Yup. Moose.

So, a couple of things. The die-handler thing is all in and on mainline and working great. There's a branch that has had everything moved to straight callbacks, so no more need to do the (extremely weird) set_callback dance, and no need for postbacks in PoCo::Generic. Those are all extremely positive things!

Now, I'm going to be merging the callback changes into my mainline (they're tested as well as I can test them). Then, it'll be on to the Moose branch. Yup, Moose. It's just going to make the codebase more comfortable to work in. After that, I want to do some general refactoring of the code, and then it's on to what I've said in previous posts.

Moose

I also want to think about moving all the MQ objects to Moose. Just think about it. *think*
*think*

...*think*

The Big Paradigm Shift

The MQ is going to become a clusterable animal. Before this happens, the front and back store dance needs to act more sanely. The front store will hold a fixed size of data, and at the timeout will issue writes to the backstore as it does not. But instead of removing messages from the front store, it will cache them there (issuing writes for things like claims). This means that a relatively inactive MQ will have the same persistence guarantees it does now, but will serve most requests from memory. A very active MQ will look the same, probably.

Also, to keep starvation of messages from happening (which is a problem the current MQ really does have), we're going to refresh a portion of the front-store cache with the oldest backstore messages every so often. Interval and amount will be configurable as usual, but that's in the pipe.

We're going to tweak Storage::Throttled to be a real, sane storage engine (that is, you can claim and remove out of it instead of that just being a proxy to Filesystem). This should fix a HUGE bottleneck. This will be super easy to implement - just give it its own "front store" type argument (probably BigMemory) where it'll keep things until the filesystem is ready.

The major thing that will be built on this, though, is The Clustered MQ. We'll have little MQ nodes sitting around, each with their own front-store service and all talking to a common backstore. There will be a load balancer sitting in front of them. Either they'll talk to each other about how many messages they have and redistribute the load, or we'll just rely on backstore pulls to keep things fresh and useful. Either way, communication about new messages WILL have to be implemented for topics. So, we'll have to come up with a way for MQs in a cluster to discover each other (something like passing an existing MQ to which a discover request is posted on creation). But that'll come when we get to that point. That is Where We're Going, though, so it's on the horizon. The goal is to be able to start up new MQs and shut them down willy nilly and tweak the load balancer and have things sort of Just Work.

*pour brain, mix thoroughly*

Wow, what a morning. Waiting outside in the WAY BELOW freezing cold for 45 minutes and still no bus. *wrap hands around tea mug to restore circulation* Guess I'm working from home today.

Anyways, as I was laying in bed last night thinking "What exactly am I working on tomorrow?", it occured to me that that was the entire reason I started this blog. So, what I've been working on in January:

• Storage API normalization in the "newguts" branch. All storage engines must now (and do!) implement remove, remove_multiple, and remove_all (with an optional callback argument that is called with the messages removed). Also in newguts, the fixed callbacks are made via set_callback(name, cb) and call_back(name, args). This cleans up the code in a couple of places, but eventually I think I'd like to move to passing explicit continuations. It'll make things less tightly coupled. That may or may not happen in this branch.
• Topics. They're in and merged into David's mainline.
• POE::Filter::Stomp is into David's mainline. Both it and topics have been stress tested, and stress tests added for topics and binary images.

I'm sure I'm missing some things, but that's what I get for not updating this blog like I'm supposed to. Anyways, to the more important issue of what I am now working on (note: it's a bad sign when I start making todo textfiles and scattering them around my code directories).

• The __DIE__ handler should fire off an email. This is what I'll be implementing today.
• Callback changes need to be either rewritten as explicitly passed continuations (I'm leaning this way) or documented better. (Note: current call_back implentation relies on a hacked up version of PoCo::Generic that the maintainer hasn't accepted a patch for yet...)
  
• Each store should be able to act as either front-store or back-store. This doesn't blow up, but it currently doesn't work. Should wait for the callback changes before trying this again.
• Once those storage changes are solidified, the Big Paradigm Shift needs to happen. That deserves its own post, so...
• Think about some way to notify people when messages can't be stored for some reason. RECEIPT support is mostly in, so maybe...