The Big Paradigm Shift

The MQ is going to become a clusterable animal. Before this happens, the front and back store dance needs to act more sanely. The front store will hold a fixed size of data, and at the timeout will issue writes to the backstore as it does not. But instead of removing messages from the front store, it will cache them there (issuing writes for things like claims). This means that a relatively inactive MQ will have the same persistence guarantees it does now, but will serve most requests from memory. A very active MQ will look the same, probably.

Also, to keep starvation of messages from happening (which is a problem the current MQ really does have), we're going to refresh a portion of the front-store cache with the oldest backstore messages every so often. Interval and amount will be configurable as usual, but that's in the pipe.

We're going to tweak Storage::Throttled to be a real, sane storage engine (that is, you can claim and remove out of it instead of that just being a proxy to Filesystem). This should fix a HUGE bottleneck. This will be super easy to implement - just give it its own "front store" type argument (probably BigMemory) where it'll keep things until the filesystem is ready.

The major thing that will be built on this, though, is The Clustered MQ. We'll have little MQ nodes sitting around, each with their own front-store service and all talking to a common backstore. There will be a load balancer sitting in front of them. Either they'll talk to each other about how many messages they have and redistribute the load, or we'll just rely on backstore pulls to keep things fresh and useful. Either way, communication about new messages WILL have to be implemented for topics. So, we'll have to come up with a way for MQs in a cluster to discover each other (something like passing an existing MQ to which a discover request is posted on creation). But that'll come when we get to that point. That is Where We're Going, though, so it's on the horizon. The goal is to be able to start up new MQs and shut them down willy nilly and tweak the load balancer and have things sort of Just Work.