So, I was let go at Carspot for budgetary reasons, which means the likelihood of me having occasion to work on PoCo::MQ any time soon is rather low. I had this as a private blog and I don't think there's any reason for it to remain so, so now it's public.

expiring messages

David had at some point wanted an optional "expires-after" header. What it would do is cause a message to be marked as "okay to delete" after a certain period of time. I guess most stores could ignore this header, but Complex in particular has a bugbear with this situation: if a message isn't persistent and someone has specified this header, then we would like to keep it until it expires (which means pushing it to the backstore, then deleting it after a certain time). This can be done by saying "does it have an expires-after thing? If so, persist it (at bump), and remember that you did, so you can clean it up later."

*vomits thoughts into blog*

stomp test

Let's see if POE::Filter::Stomp works with the latest released version of MQ, shall we?

MQ progress

After much blood and sweat, we now have fixed-size stores (with a unit test to prove that they work). All the other unit tests have been fixed, and there's a smoke-testy sort of "chuck a bunch of messages into the stores and try all or most of the storage functions on them" kind of test, so we have Good Testing Stuff (hopefully that will continue in future!). I might think about making a base test class that subclasses MooseX::POE and clean up some of the tests today, because they really all should be doing their testy things as POE events. I wonder how easy it'd be to add an attribute to a sub that says "make calls to this really be posts to this object's session." Hmm.mm.mm. Oh well, that's off in the blue sky. Cleaning up the tests a littl bit would be positive, anywho. And I'm pretty sure I don't have an explicit test in 05_storage_api for disown_all, and the only case where it's useful seems to be in a place where we're already looping destinations anyway...

Well, this got rambly quick! I'm going to need to ask David what we meant by a "cleanup cycle", but the bad-pump-bug seems to be mostly gone. I will fix it further by trying to make the storage-calling-code as callbacky as possible. Real $destinations all over the place is probably next on my list today. That's all!

Places, faces

Place needs to be renamed to Destination to be less confusing, and then we need to hunt for where we're using $destination and either convert them to real destination objects (and use $destination->name) or call them $destination_name.

The Valentine's Day File

The following are the MQ issues discussed on 2008-02-14 (herein after
referred to as the "Valentine's Day File"):

* Complex: granularity and timeout
* Race condition: pumping queues
* Clean up cycle
* Non-persistent messages: expel from front-store per normal, remove
when expired during clean-up cycle

non-persistent messages

So I don't forget, we need to look into what guarantees ActiveMQ makes about non-persistent messages and try to do something sane with them. Currently, we just sort of let them fall out of the front store.

Mooooooo...se.

So, the moose branch has everything but statistics all moosed up. Statistics isn't breaking anything, at least, but I don't really want to spend any time mucking with it because I have painted a giant red target on it. I want it to Go Away, which it probably will when I write real monitoring tools. So. But all the core stuff is Moosed, now, and all the callback stuff is chugging right along being happy, so I think it's time to yank myself out of the land of refactoring and into New Feature Territory. Which is more fun anyways.

I went to open up a text file to talk to myself in about clustering, then I remembered this blog. And I read some of my old posts, and I remembered that the Next Thing is the fixed-size stores. This is going to be some changes to Complex more than anything, and will just be changing the way expire works first of all. Need peek() for all stores. Need to remember some data about whether messages have been persisted in Complex, so I can let things all out of the front store if they have. I think peek() first. Then persistence memory. Then expire change. Then backstore-pulling.

Yup. Moose.

So, a couple of things. The die-handler thing is all in and on mainline and working great. There's a branch that has had everything moved to straight callbacks, so no more need to do the (extremely weird) set_callback dance, and no need for postbacks in PoCo::Generic. Those are all extremely positive things!

Now, I'm going to be merging the callback changes into my mainline (they're tested as well as I can test them). Then, it'll be on to the Moose branch. Yup, Moose. It's just going to make the codebase more comfortable to work in. After that, I want to do some general refactoring of the code, and then it's on to what I've said in previous posts.

Moose

I also want to think about moving all the MQ objects to Moose. Just think about it. *think*
*think*

...*think*

The Big Paradigm Shift

The MQ is going to become a clusterable animal. Before this happens, the front and back store dance needs to act more sanely. The front store will hold a fixed size of data, and at the timeout will issue writes to the backstore as it does not. But instead of removing messages from the front store, it will cache them there (issuing writes for things like claims). This means that a relatively inactive MQ will have the same persistence guarantees it does now, but will serve most requests from memory. A very active MQ will look the same, probably.

Also, to keep starvation of messages from happening (which is a problem the current MQ really does have), we're going to refresh a portion of the front-store cache with the oldest backstore messages every so often. Interval and amount will be configurable as usual, but that's in the pipe.

We're going to tweak Storage::Throttled to be a real, sane storage engine (that is, you can claim and remove out of it instead of that just being a proxy to Filesystem). This should fix a HUGE bottleneck. This will be super easy to implement - just give it its own "front store" type argument (probably BigMemory) where it'll keep things until the filesystem is ready.

The major thing that will be built on this, though, is The Clustered MQ. We'll have little MQ nodes sitting around, each with their own front-store service and all talking to a common backstore. There will be a load balancer sitting in front of them. Either they'll talk to each other about how many messages they have and redistribute the load, or we'll just rely on backstore pulls to keep things fresh and useful. Either way, communication about new messages WILL have to be implemented for topics. So, we'll have to come up with a way for MQs in a cluster to discover each other (something like passing an existing MQ to which a discover request is posted on creation). But that'll come when we get to that point. That is Where We're Going, though, so it's on the horizon. The goal is to be able to start up new MQs and shut them down willy nilly and tweak the load balancer and have things sort of Just Work.

*pour brain, mix thoroughly*

Wow, what a morning. Waiting outside in the WAY BELOW freezing cold for 45 minutes and still no bus. *wrap hands around tea mug to restore circulation* Guess I'm working from home today.

Anyways, as I was laying in bed last night thinking "What exactly am I working on tomorrow?", it occured to me that that was the entire reason I started this blog. So, what I've been working on in January:

• Storage API normalization in the "newguts" branch. All storage engines must now (and do!) implement remove, remove_multiple, and remove_all (with an optional callback argument that is called with the messages removed). Also in newguts, the fixed callbacks are made via set_callback(name, cb) and call_back(name, args). This cleans up the code in a couple of places, but eventually I think I'd like to move to passing explicit continuations. It'll make things less tightly coupled. That may or may not happen in this branch.
• Topics. They're in and merged into David's mainline.
• POE::Filter::Stomp is into David's mainline. Both it and topics have been stress tested, and stress tests added for topics and binary images.

I'm sure I'm missing some things, but that's what I get for not updating this blog like I'm supposed to. Anyways, to the more important issue of what I am now working on (note: it's a bad sign when I start making todo textfiles and scattering them around my code directories).

• The __DIE__ handler should fire off an email. This is what I'll be implementing today.
• Callback changes need to be either rewritten as explicitly passed continuations (I'm leaning this way) or documented better. (Note: current call_back implentation relies on a hacked up version of PoCo::Generic that the maintainer hasn't accepted a patch for yet...)
  
• Each store should be able to act as either front-store or back-store. This doesn't blow up, but it currently doesn't work. Should wait for the callback changes before trying this again.
• Once those storage changes are solidified, the Big Paradigm Shift needs to happen. That deserves its own post, so...
• Think about some way to notify people when messages can't be stored for some reason. RECEIPT support is mostly in, so maybe...