Vacation!

So, I only worked one day last week, and that was on pluggable message ID generation. The framework is there and a couple of simple ID generators are in. I think it's back to a callback framework now, and getting my branches in a more manageable state.

Just doing some brain dump so that I can start getting to work. :)

i did it, on monday

So BigMemory is really in, having been tested and all the code braced and indented and commented and all that stuff. It handles a 100k message load with no problems, which is -very- good, and a big step in the right direction towards making the MQ more scalable.

Time to look at other ways to improve performance: like maybe a persistent database connection to a real DB? Hmmmmmm?

BigMemory

So, BigMemory is in now. I haven't tested it AT ALL, and DLList needs to be reformatted and documented. So do that on monday!

Sprint 6

Hmph! So much for updating this every day. I knew this would happen eventually, but /sigh.

Anyways, signal handling is in! And I'm sick. Suck. Anyways, Sprint 6 is MQ time: my next task is to solve the Stalin bug (clients whose messages are moved Memory, bypass Throttled, and acked before file write get stalled forever, it seems). After that, the misdelivery bug (problems with delivery cause shutdown to fail). Then, barring new bugs, it'll be time to profile and optimize Storage/Memory and Storage/Throttled. Huttah!

POE, MQ, yay!

So I'm working on making the MQ handle signals properly (graceful shutdown branch, yo!). That's what I'm doing today. Yay for focus!

I'm not missing this day!

So, I spent most of today reading, it seems. I'm trying to understand POE, and also exactly what the MQ does and how it's used. Erik is off testing my RSS tools (yay!), so I think tomorrow I'll try to do like... processing tasks... using the MQ. Like... I don't know. I'll think of an interesting problem, or get one off Euler, and then I'll use the MQ to do something with them to get a feel for what's going on there.

hmm, missed a day there

The export I was working on is done, and ride65 integration is going to have to....wait.

The RSS tool and the user preferences page, that whole mess, though. That's pretty much done. It might get tweaked between now and release by what Erik says, but it's done. :)

thoughts, thoughts...

It looks like the more general problem is: how do we map one/more of our values to one/more of their values in a general way?

Dave was talking about options. We may have option "foo" and "bar", and they represent that in their system as "foobar". The one-off way is to say
<xsl:when test="foo and bar">foobar</xsl:when>.
It might go the other way round, too, so that we want to say
<xsl:when test="foobar">foo|bar</xsl:when>.

You can just use a couple of many-to-many tables for that if you really want to keep those in a database instead of in an export. Doesn't seem too awful, but it might be best just to leave that in the xslt, honestly.

And really, I'm not terribly sure that the xslt isn't just the best place for all of this stuff. Hmmmm..

Monday, monday...

So wow, it's a miracle that even got to work this morning. Cassie sort of rolled me out of bed and onto the bus.

Anyways, a few things. One is that ChangeTracker needs to have a slight update: it will allow nodes of the structure it's tracking to be replaced by complex objects (arrays or objects) and then track changes to the whole structure (i.e. changing any child of the node you set as a complex object means the entire child-structure under that node is marked as changed). This is to allow tracking of arrays and lists in the only way that makes sense to me. It's not perfect, but until I think of something better, it will work (and it really should be good enough for the most common cases).

This morning I'm doing export configs - there's one for Commonwealth Audi/VW that I'm doing first, and then a more complicated one with Ride65 that will require either a perl hack or changes to the exporter. Changes to the exporter is more attractive. The problem is that ride65 maps values to ids, gives us a list of ids, and wants us to use those in our export instead of the values. If the mappings were small we'd just hack them right in there, but it's quite the hefty list, and that's not maintainable. So I'm thinking we can put a "mapping" section into input.xml (or somewhere) that's got a lookup table for values (drawn from a special table in cs3 db) and then make xslt queries on the table a-la "give me the id for make Volvo". I'm not sure how possible this will be to do, and I'll have to talk to Dave about it. Anyways, that's for later. Just wanted to get it down on paper. Er, whatever.

Flow

Getting into flow, it seems. I got the ChangeTracker done today, and the user preferences page is using it. It's very nice, actually, and both cuts down on code and ugliness (as opposed to how I was doing it) and makes less work for the server/database, and presents a nicer user experience. It's a win-win! It's always better to do things the Right Way. :)

Universal Tab Saver

Okay, so the RSS tools in their current state work beautifully, but ALS and DRS don't like the way my form does save-confirmation. So, being programmers, the immediate thought is "can we make one thing that does this all over?" That way, there'd be One Way To Do It, and people wouldn't have room to have flame wars about which was more intuitive.

So with input from ALS and DRS, here's my thought:

All tracked data must be input by a big pile of JSON:

{mytabs: {
mytab1: {
},
mytab2: {
}
}};

We can check whether an item has changed by naming the field in the input and giving some data:

hasChanged('mytabs.mytab1.myfield', 'newfieldvalue')

We can say "yup, let's keep that change" by calling...

dirty.setDirty('mytabs.mytab1.myfield', 'newfieldvalue')
(or maybe later)
dirty.setClean('mytabs.mytab1.myfield')

Then later, we can say

dirty.getValues()

to get a JSON structure of all the things that we claim have changed. That will get passed to a save script, and all will be well.
{mytabs: { mytab1: { myfield: newfieldvalue } } }

In the general case, the programmer will want to attach onChanged handlers to input fields and just call if(hasChanged) setDirty else setClean.

In the more complicated case, it's still flexible enough that you can do your own custom logic (translate a bunch of listbox widgets into some json and check things). So it makes easy things easy and hard things possible.

We might be able to specify a default kind of helper, like...

onChanged = simple_dirty_checker(original_data,change_tracker);

for the simple cases.

Anyways, that will take a little while to write. But I think I've documented what I want it to do, and it should be awesomeness.

current status

The caching was re-wired successfully on Friday, and the panes are almost functional again. Working on that this morning, and then moving into new-SCRUM land hopefully today. It might be the land of meetings...

Caching stuff

This morning I'm going to get right cracking on RSS caching scheme. Just put a timestamp on feed objects, figure out how to parse it into a datetime, and see if (interval) has passed. So that should be reasonably quick. The second, slightly more difficult part is develop a json-outputting script that reads a feed based on a url parameter. It might be able to share some code with add_feed, we'll have to see. But the workflow for the control will be:

Get passed in a list of feeds (no items).
Construct the panes, along with a front-displaying pane with some informative text about where to add rss feeds.
The other feeds will be lazy-loading but will NOT use AccordionPane's url feature - on show they will make an XHR request, grab their feed, construct their DOMs (because there's fancy javascript and there's no reason to put more load on the server) from JSON, and be done with it. The cache checks will be done server side, and feeds older than time X (specified in one place, is_feed_expired) will be re-done. A loading page will be shown while the XHR is in progress.

RSS widget done

So, the RSS widget is up and displaying on the front page. I'm rethinking the caching scheme, though, so it works like this:

When we go to display a feed, we call cache_feed on it. This gives us the feed back, in addition to re-fetching it if it's stale (we'll define stale to be say, older than 10 minutes). We'll also rig it up so that there are no feeds showing off the bat, and they only get loaded if someone tries to look at them (and thus only get recached if someone tries to look at them). This should cut down on overall server load.

refresh_cache got renamed to clean_cache, and it deletes feeds that aren't being subscribed to. It also nixes all the items off of feeds that are stale anyway. This should get called maybe once an hour in a cron job?

Working on RSS tools!

I really and truly finished up the rest of the page, so here's the roadmap now.

• Finish implementing rss control tools
• stick an accordion pane on the main page for viewing rss feeds
• set up cs_dojo structure in trunk
• get Aaron's themes etc. into trunk
• merge trunk into .rss
• migrate to dojo 1.0, test for anything that breaks
• **possibly break new_prefs into seperate files - even just includes would be fine
• one last trunk merge
• test, test, test
• submit!

I feel productive this morning

Got "my" export tweaked for Annie Kopish, so yay. Also, figured out that you test perl string equality with eq, not with ==. That's what was making problems in save_general.pcgi. For a DWIM language, that's pretty stupid.

I'm more or less done with the general section, so now I can write the rss tool prefs. :)

So that's done

LazyFilteringSelect has been rewritten and works as described in the last post. Going to have to figure out now why timezone is giving me such evil errors - must be something funny about the data. Anyways, that's probably going to take all freaking day. MEH.

Refactoring, rewriting...

So yesterday, I became very frustrated because my LazyFilteringSelects (some of them) weren't loading right. So I thought about things some more, and decided to do two things today.

The first is rewrite the _list pcgis to use CS3::SimpleDojoList. This is just to make sure they're all outputting their JSON in a uniform way - it helps reduce typos. Also, it makes the code very, very slightly cleaner. So that's done now.

The second is to rethink LazyFilteringSelect. I'm making it less generic (it will only make sense for ItemFileReadStores, and you pass it a URL instead of a function that generates a store) and it does an XhrGet to retreive the data rather that entrusting that to the store, so that it can publish loadStarted and loadFinished events and do something intelligent with errors. The store api is lacking.

half done...

Things that are half done annoy me, because they taint anything that has to touch them. In this particular case, there's some braindead timezone support in Pro, and it's hacked into the general preferences page somehow. The spaghetti in this code is... not tasty.

Anyways, the proper thing to do is to get the list of timezones from the database (which means I'll have to make an Xmldoom adapter, of course, cause there isn't one. Why would there be?) and use that to populate a LazyFilteringSelect. So that's what I'll do, but there goes another few hours.

I hate things that get half-done.

So, moving on...

Now that the cache thing isn't driving me nuts, on to buisness. We should look at our other pcgi scripts to make sure they're not lame in the same way index.pcgi was, and then do the general preferences page, at which point we're finally done with the user preferences rewrite and can add the rss tools. Hooray!

my "caching" problem

Unable to drop it like a sane person when I went home, I was obsessing over my "caching problem." I did find the solution, and it was a caching issue. Sort of.

For those who weren't at the programmer's meeting last night, here's the mysterious behavior. When I change things on the User Preferences page (my version) and save them, the changes happen in the database but don't show up on a refresh of the page! Most mysterious. Immediate thought is "browser cache problem." Clear browser cache, close browser, reload browser, changes still haven't shown up. Restart apache... hey presto, changes showing up! Even curiouser, Aaron goes to the page and gets the info I'm seeing, prior to changes. It looks like one of Apache/mod_perl/HTML::Template/mysql is caching something somewhere.

My initial suspicion was that it was mod_perl doing it somehow. And you know what? I was right.

See this Perlmonks article.

We use Apache::Registry, of course, since it rules. Here's the relevant bits of code as it was when I was having the problem (or a reasonable facsimile):


my $page = CS3::Page->new();
my $user = $page->get_userX();



sub main {
# body of the script happens here, except that now I'm a closure over the
# initial instance of user that was here when this script was first compiled. D'oh!
# We should instantiate $page, $user, etc. here and pass them to our other
# subroutines...
}

main();


The problem with this code is described in the perlmonks article. Lesson learned: do not mix globals and subroutines inside Apache::Registry. Also, Apache::Registry is a bit of a hack, and it's best to be aware of how it does its magic, cause we use it. Everywhere.

Saving!

Woo, almost missed a day there! Was too busy working to write in blog. :)

So, saving is implemented on the user_info tab now, and everything validates as it should. That means (gasp) all the conceptual stuff should be done, and it's just typing to implement the other two tabs now. !!

Why it's taking so long

It's the little snags. This morning, I spent... looks like almost three hours tracking down a bug that was preventing the state dropdown from validating correctly. I found it, it works now. But that was a "wasted" 3 hours not spent writing new code, but finding a finicky little bug.

That's why it takes so long.

I think it's time to save.

Well, well. I think the main part of the user preferences page is all done. I'm pretty much happy with the architecture - all the savable data is passed in JSON, it fills out a form, the user can edit fields, and send it back. I'll then have a very regular format that will require very little munging to get it back into writable form. Should work well.

Then, it's just a matter of doing pretty much exactly the same thing with the general_preferences. And then I can add RSS controls. I should be able to finish the front page today (8 hours), the general preferences page in let's say a day and a half (14 hours), and the RSS page in another 8 hours. So that's 30 hours left, and I should be able to get it done by the end of the week. I think that's what's actually listed on the burndown. I don't know why this is taking me such a long time, it's really not that hard conceptually. It's not even a lot of typing. I keep having to look up dojo things things though, and that's slowing me down. Also, my debugging environment is stupid and slow. Maybe after I get this done I can do something that doesn't require an extremely silly testing process.

Oh well. Time to finish this page up.

Monday

So, I got the _list.pcgi's and the Xmldoom stuff set up. This morning, I'm setting up a new export (weird), and then once that's all hammered out I'll be populating the state and country list boxes from Ajax calls. Won't that be a joy.

I think it makes the most sense to grab all the state information from state_list and just use a query filter based on the state for the state box. We'll have to see how that works. But it doesn't make a lot of sense to call state_list.pcgi every time the country box changes. So we won't do that.

So at the moment, I'm waiting for jkw to add me to the exports mailing list so I can get started conversing with the export contact. Maybe I should work on user_preferences... he seems to be taking a while.

States

Maybe I need to get rid of my obsession with doing things the Right Way. Or maybe I need to never, ever lose it. It's hard to decide.

So, the user_info part of the user preferences page is mostly done. Or I thought it was. Then I looked down at that troublesome "state" box in the address section. And I thought, "I should make another script to return state info in JSON and make this box read it, like security questions." Then I looked at the state table, and found that it has a country foreign key, which isn't specified in database.xml and doesn't have an xmldoom object associated with it. *cry*

So, I've got to make one of those, and then think of a good scheme for getting that data. What I'm thinking is having a parameter to state_list.pcgi of country, that defaults to USA. Because generally speaking, nobody cares about the stuff from Country, but compatibility dictates. I'll need a country_list.pcgi too, because I'll need a dropdown for country that determines what states you can select.

The whole reason we care about validating the state is that they aren't all in the database. Rumble.

Anyways, I can only work on this for so long today (let's say until lunch), and then it's all learning exports.

Success!

Such a dojo widget has now been created, and tested, and it works seamlessly and is beautiful. Now I need to start learning exports...

New dojo widget

So, it occurs to me that the extremely lazily loading FilteringSelect is something I might want to use in the future, so I'm making a new dojo widget. What it will do is accept an (optional) initial label and value, and that will show up. getValue returns the value, isValid returns true, all that goodness. But when you go to change the widget (click on it, etc.) it'll call the function you gave it as its storeGen attribute to create its store. I.E., the store does not get created until you need it. I'll call it (dun dun dun!) cs.dijit.form.LazyFilteringSelect. :)

Edit: A couple more features. setValue will simply set what gets returned by getValue, and setDisplayedValue will simply set the label (no lookups) UNTIL either you try to change the widget or you call widget.loadStore();

Field values!

Major perceptual breakthrough yesterday: I've got the fields populating from the database, and it's just a clean form.setValues() call. Yay infrastructure work. :)

So I'm adding validation right now, then it's a very similar thing on the next page (general preferences). Then, I can finally add the rss tools!

waste of time

I hate it when I spend a lot of time chasing down a bug that I don't end up fixing. Complete waste of time. I spent most of the morning trying to find the apache dbi bug and trying to figure out why IE's caching policies were screwing up my page. Blah.

Server-side pagination

Also, it's worth noting that while ItemFileReadStore implements client-side paging (i.e. it won't create controls for the data until you want to see them, see the pageSize attribute), it has no effect at all on server-side paging. That means that if you have 50,000 records and you only want to load 100 at a time, you have to implement a new store that passes like start and end parameters to the server. This can be done, and might benefit us here if we have really large record sets. We'll have to see.

See here for some ideas.

Got some stuff done!

So, yesterday I managed to get a good bit done. I've got some database-backed dojo widgets, which is something. Aaron's going to show me how to fix my db pooling issues this morning, which should help the reliability of the ajax calls. I'm still not positive if it's doing exactly what I want, that is loading those fields on-demand (I'm especially concerned about the default dealer list - probably no reason to be, but I would like to know how to avoid loading db stuff if possible).

If I have to, I'll hack in an ajax call that sets the store. In fact, that might be the first thing I work on this morning. First, test to make sure it's getting loaded before the click. If it is, register a click handler that does an XHR call that'll set the store on return. That seems like the best approach.

Anywho, after that it's just a matter of plugging in something similar for the dealers - that can probably be a seperate script, cause there's other places that might benefit from being able to call it. But I won't break it out until I actually run across such a case, cause premature generalization is evil.

Then, we figure out how saving works - it should be as simple as pulling form.get_values and sending the data to the server with an XHR. We'll have a generic POST save handler, and what gets saved will depend on parameters passed (general: {json}, user_info: {json}).

Let's get some stuff done today

Over the weekend I got XP working on my virtual machine, so that's all set to go (no more screwing with the desktop!). So I'm good to go to work on my page. I need to duplicate the functionality of the old preferences page, and then add the new RSS stuff. I've got most of the fields specified, so it's just a matter of working out how the AJAX calls are going to happen.

New macbook!

So I spent the end of yesterday and the beginning of today screwing with my new macbook. I've got it mostly set up the way I want now, except that I don't have an XP install working on it yet. (I did make a nice little Mac OS .app that runs a perl script that connects to windows dev via rdesktop, but that machine has too many connections at the moment, so that's not usable). So, when I get home tonight I'll get XP up and running on my Q vm, and that'll make a good platform from which to test Pro. I think I'll still probably use rdesktop most of the time, but it's often not available and I can't be dependant on it. Also, X is running invisibly and giving me the illusion of nativity for things like rdesktop, and that's super duper.

As far as actually getting any programming done today... I'm gonna do lunch with Aaron in about 15 minutes, I'm thinkin', so it won't be until after lunch. I need an IE to test against, so I'll probably have to (ugh) use the desktop. Oh well. Gotta get some work done today.

It'll be further updating the form and adding more tabs, then figuring out how to make the ajax calls that save the various sections.

oh my god.

So, I figured out today you can shove coderefs right into your object's symbol table at compile time. Like a macro, kinda. You can use this to generate accessor methods, etc.

Did I mention I don't know perl? This kind of hackery seems to be the norm for this silly language...

Gung-ho!

Stayed up too late last night, but here I am, bright eyed and bushy-tailed. I think I will specify the list of things to get, and only send the appropriate data (for wire-size as well as security). I can have the list in one spot, and specify either "simple" (undef) or a handler (for complex fields like address). The handler will be a two-tuple, with one method for getting and another for setting the value.

where I got today

Today has been confusing, and large parts of it felt unproductive. I think I understand the code I'm modifying a lot better though. Anyways, I've started the preliminary work on generating a JSON object to pass to the dojo-page. Note to self: When you update xmldoom object definitions, you may have to restart apache for the changes to stick.

Preliminary thoughts: We may want to specify which fields we want to pass instead of passing everything in _get_attributes. The reason for this being: we don't want the page to be able to modify, say, the session_id. So we may need to be explicit.

Nothing is Xmldoom'd

It's really kinda depressing how very little has been Xmldoom'd. I've made some progress converting the user object, but there's so much extra weird stuff going on all over the place that it's hard to know what will work and what will break. I guess you just keep hacking and fixing.

Anyways, I've got rdesktop now, so I can test Pro from my laptop instead of having to boot up the slow retarded XP box. So that's good. Also, I've...thought a lot more about what user_preferences.pcgi is doing. It's all slowly becoming clear, and I think I have a migration plan, but it's too lengthy to put here. It's in a text file on my laptop.

Note to self...

Never do port upgrade all. All is a good way to get MacPorts to install EVERY PORT. Which is not what you want at all. Unless it is.

But probably, you want...

sudo port sync
sudo port selfupdate
sudo port upgrade outdated

New Thoughts

Okay, so after talking to David and Aaron about JSON/Xmldoom, the idea is that we set up AJAX scripts to load and save the various parts of the user preferences, with JSON as a transport. This page is simple enough that they can all be in one script, I think, with like a load/save function for each tab.

Also, I've started using Blogger to keep track of these to-do list things, although I think I'm still going to compose them in WriteRoom, cause of the cool Matrix-feel. I wonder if there's a Blogger plugin?

JSON, hmm?

Well, I thought about it tonight, and I've decided that one intriguing way to solve this problem would be to write a JSON adapter for Xmldoom objects (if there isn't one already). Then I could Xmldoom-ize the data being talked about in user_preferences.pcgi, serialize it as JSON, pass it to the dojo-page to parse, modify that JSON object, and pass it back to be saved. GET on user_preferences.pcgi would render HTML by default, but if you pass it &json, it will just give you the JSON for the objects being talked about. This will not only make for nicer code in this one spot, it's a good model for modifying existing Xmldoom objects.

If we don't want that whole thing, we could do a comprimise - and just do the JSON handling manually. That wouldn't be too bad, and it's certainly a better structure than what we've got. I like this model...

one. giant. blob. of perl.

Well. Here's the thing. If I make a Dojo UI for the RSS preferences, I really need to make it a Dojo UI for the whole thing, and rewrite user_preferences.pcgi. It's like, there's two options. Either

1. leave it ugly and stupid and just hack in my additions or
   
2. clean it up and bring it into the New Way.
   

Now, I'm all for doing number 2 (lol). But it will take longer and it's more or less a waste of time, and it will probably break something. We could instead do some custom AJAX-y stuff (easy enough) and make everyone just as happy.

It might still make some sense to clean up the perl. Maybe that's just the first thing I'll do. Clean up the retarded perl.

This file is getting kinda long...

I should seriously consider making some sort of simple blogging tool for this, otherwise the file may get long and unmanageable. Or, rather than making one, finding one. Making one would take too long. At any rate, today I'm making a dojo UI for the RSS stuff. *hesitant laugh*

arg!

So, David says that Xmldoom is indeed broken in this respect and that he'll work on fixing it Monday. Till then, I'll content myself with the ugly syntax.

That went surprisingly quickly

So, I got everything on this list done except for one minor hitch: the pretty syntax for adding subscriptions isn't working, and you seem to have to manually create and save a Subscription object. That's silly, so I'm going to spend some time trying to track that down and make the sane-looking call work.

let's get busy...

So, big meeting this morning. Most of it is stuff I can't do anything about, other than trying to take some of the work off of Dave and John where that's possible (exports is probably something I can help with, so I'll get going on that as soon as I get done with RSS). So, for RSS, what are we doing?

• Change all the new Objects() to Objects->new() to make David happy
• Find that goddamn debugging statement in Xmldoom that prints out count()
• Find out why recreate_database() isn't working so I don't have to drop cs_replication
• Fix the unit tests and make sure they pass.
• Give users an ID property so I can use that in the tests...
  

That really should take most of the day. I'll need to talk to David about the unit tests, probably, but I'll try to figure them out on my own first.

What's up today?

What's up today?

I've finished writing CSS::RSS.pm for the most part, and the database objects are all implemented. They do need testing, though. The next thing to do, I suppose, is to write comprehensive unit tests for them. Once they're thoroughly tested and I'm sure that they work, I need to get together with Aaron and have him point me in the correct direction about Dojo. Just an idea of what they're using it for, so I can do things the same way they're doing them rather than being inconsistent.

So, plan for today is:

• Write tests for RSS.pm and Xmldoom objects -- CS3/RSS/test.pl
• Figure out what Aaron's been doing with Dojo
• Write a plan for the Pro stuff.
• Two things:
  • Subscription editing
  • Feed Viewing