I'm not missing this day!

So, I spent most of today reading, it seems. I'm trying to understand POE, and also exactly what the MQ does and how it's used. Erik is off testing my RSS tools (yay!), so I think tomorrow I'll try to do like... processing tasks... using the MQ. Like... I don't know. I'll think of an interesting problem, or get one off Euler, and then I'll use the MQ to do something with them to get a feel for what's going on there.

hmm, missed a day there

The export I was working on is done, and ride65 integration is going to have to....wait.

The RSS tool and the user preferences page, that whole mess, though. That's pretty much done. It might get tweaked between now and release by what Erik says, but it's done. :)

thoughts, thoughts...

It looks like the more general problem is: how do we map one/more of our values to one/more of their values in a general way?

Dave was talking about options. We may have option "foo" and "bar", and they represent that in their system as "foobar". The one-off way is to say
<xsl:when test="foo and bar">foobar</xsl:when>.
It might go the other way round, too, so that we want to say
<xsl:when test="foobar">foo|bar</xsl:when>.

You can just use a couple of many-to-many tables for that if you really want to keep those in a database instead of in an export. Doesn't seem too awful, but it might be best just to leave that in the xslt, honestly.

And really, I'm not terribly sure that the xslt isn't just the best place for all of this stuff. Hmmmm..

Monday, monday...

So wow, it's a miracle that even got to work this morning. Cassie sort of rolled me out of bed and onto the bus.

Anyways, a few things. One is that ChangeTracker needs to have a slight update: it will allow nodes of the structure it's tracking to be replaced by complex objects (arrays or objects) and then track changes to the whole structure (i.e. changing any child of the node you set as a complex object means the entire child-structure under that node is marked as changed). This is to allow tracking of arrays and lists in the only way that makes sense to me. It's not perfect, but until I think of something better, it will work (and it really should be good enough for the most common cases).

This morning I'm doing export configs - there's one for Commonwealth Audi/VW that I'm doing first, and then a more complicated one with Ride65 that will require either a perl hack or changes to the exporter. Changes to the exporter is more attractive. The problem is that ride65 maps values to ids, gives us a list of ids, and wants us to use those in our export instead of the values. If the mappings were small we'd just hack them right in there, but it's quite the hefty list, and that's not maintainable. So I'm thinking we can put a "mapping" section into input.xml (or somewhere) that's got a lookup table for values (drawn from a special table in cs3 db) and then make xslt queries on the table a-la "give me the id for make Volvo". I'm not sure how possible this will be to do, and I'll have to talk to Dave about it. Anyways, that's for later. Just wanted to get it down on paper. Er, whatever.

Flow

Getting into flow, it seems. I got the ChangeTracker done today, and the user preferences page is using it. It's very nice, actually, and both cuts down on code and ugliness (as opposed to how I was doing it) and makes less work for the server/database, and presents a nicer user experience. It's a win-win! It's always better to do things the Right Way. :)

Universal Tab Saver

Okay, so the RSS tools in their current state work beautifully, but ALS and DRS don't like the way my form does save-confirmation. So, being programmers, the immediate thought is "can we make one thing that does this all over?" That way, there'd be One Way To Do It, and people wouldn't have room to have flame wars about which was more intuitive.

So with input from ALS and DRS, here's my thought:

All tracked data must be input by a big pile of JSON:

{mytabs: {
mytab1: {
},
mytab2: {
}
}};

We can check whether an item has changed by naming the field in the input and giving some data:

hasChanged('mytabs.mytab1.myfield', 'newfieldvalue')

We can say "yup, let's keep that change" by calling...

dirty.setDirty('mytabs.mytab1.myfield', 'newfieldvalue')
(or maybe later)
dirty.setClean('mytabs.mytab1.myfield')

Then later, we can say

dirty.getValues()

to get a JSON structure of all the things that we claim have changed. That will get passed to a save script, and all will be well.
{mytabs: { mytab1: { myfield: newfieldvalue } } }

In the general case, the programmer will want to attach onChanged handlers to input fields and just call if(hasChanged) setDirty else setClean.

In the more complicated case, it's still flexible enough that you can do your own custom logic (translate a bunch of listbox widgets into some json and check things). So it makes easy things easy and hard things possible.

We might be able to specify a default kind of helper, like...

onChanged = simple_dirty_checker(original_data,change_tracker);

for the simple cases.

Anyways, that will take a little while to write. But I think I've documented what I want it to do, and it should be awesomeness.

current status

The caching was re-wired successfully on Friday, and the panes are almost functional again. Working on that this morning, and then moving into new-SCRUM land hopefully today. It might be the land of meetings...

Caching stuff

This morning I'm going to get right cracking on RSS caching scheme. Just put a timestamp on feed objects, figure out how to parse it into a datetime, and see if (interval) has passed. So that should be reasonably quick. The second, slightly more difficult part is develop a json-outputting script that reads a feed based on a url parameter. It might be able to share some code with add_feed, we'll have to see. But the workflow for the control will be:

Get passed in a list of feeds (no items).
Construct the panes, along with a front-displaying pane with some informative text about where to add rss feeds.
The other feeds will be lazy-loading but will NOT use AccordionPane's url feature - on show they will make an XHR request, grab their feed, construct their DOMs (because there's fancy javascript and there's no reason to put more load on the server) from JSON, and be done with it. The cache checks will be done server side, and feeds older than time X (specified in one place, is_feed_expired) will be re-done. A loading page will be shown while the XHR is in progress.

RSS widget done

So, the RSS widget is up and displaying on the front page. I'm rethinking the caching scheme, though, so it works like this:

When we go to display a feed, we call cache_feed on it. This gives us the feed back, in addition to re-fetching it if it's stale (we'll define stale to be say, older than 10 minutes). We'll also rig it up so that there are no feeds showing off the bat, and they only get loaded if someone tries to look at them (and thus only get recached if someone tries to look at them). This should cut down on overall server load.

refresh_cache got renamed to clean_cache, and it deletes feeds that aren't being subscribed to. It also nixes all the items off of feeds that are stale anyway. This should get called maybe once an hour in a cron job?

Working on RSS tools!

I really and truly finished up the rest of the page, so here's the roadmap now.

• Finish implementing rss control tools
• stick an accordion pane on the main page for viewing rss feeds
• set up cs_dojo structure in trunk
• get Aaron's themes etc. into trunk
• merge trunk into .rss
• migrate to dojo 1.0, test for anything that breaks
• **possibly break new_prefs into seperate files - even just includes would be fine
• one last trunk merge
• test, test, test
• submit!

I feel productive this morning

Got "my" export tweaked for Annie Kopish, so yay. Also, figured out that you test perl string equality with eq, not with ==. That's what was making problems in save_general.pcgi. For a DWIM language, that's pretty stupid.

I'm more or less done with the general section, so now I can write the rss tool prefs. :)

So that's done

LazyFilteringSelect has been rewritten and works as described in the last post. Going to have to figure out now why timezone is giving me such evil errors - must be something funny about the data. Anyways, that's probably going to take all freaking day. MEH.

Refactoring, rewriting...

So yesterday, I became very frustrated because my LazyFilteringSelects (some of them) weren't loading right. So I thought about things some more, and decided to do two things today.

The first is rewrite the _list pcgis to use CS3::SimpleDojoList. This is just to make sure they're all outputting their JSON in a uniform way - it helps reduce typos. Also, it makes the code very, very slightly cleaner. So that's done now.

The second is to rethink LazyFilteringSelect. I'm making it less generic (it will only make sense for ItemFileReadStores, and you pass it a URL instead of a function that generates a store) and it does an XhrGet to retreive the data rather that entrusting that to the store, so that it can publish loadStarted and loadFinished events and do something intelligent with errors. The store api is lacking.

half done...

Things that are half done annoy me, because they taint anything that has to touch them. In this particular case, there's some braindead timezone support in Pro, and it's hacked into the general preferences page somehow. The spaghetti in this code is... not tasty.

Anyways, the proper thing to do is to get the list of timezones from the database (which means I'll have to make an Xmldoom adapter, of course, cause there isn't one. Why would there be?) and use that to populate a LazyFilteringSelect. So that's what I'll do, but there goes another few hours.

I hate things that get half-done.

So, moving on...

Now that the cache thing isn't driving me nuts, on to buisness. We should look at our other pcgi scripts to make sure they're not lame in the same way index.pcgi was, and then do the general preferences page, at which point we're finally done with the user preferences rewrite and can add the rss tools. Hooray!

my "caching" problem

Unable to drop it like a sane person when I went home, I was obsessing over my "caching problem." I did find the solution, and it was a caching issue. Sort of.

For those who weren't at the programmer's meeting last night, here's the mysterious behavior. When I change things on the User Preferences page (my version) and save them, the changes happen in the database but don't show up on a refresh of the page! Most mysterious. Immediate thought is "browser cache problem." Clear browser cache, close browser, reload browser, changes still haven't shown up. Restart apache... hey presto, changes showing up! Even curiouser, Aaron goes to the page and gets the info I'm seeing, prior to changes. It looks like one of Apache/mod_perl/HTML::Template/mysql is caching something somewhere.

My initial suspicion was that it was mod_perl doing it somehow. And you know what? I was right.

See this Perlmonks article.

We use Apache::Registry, of course, since it rules. Here's the relevant bits of code as it was when I was having the problem (or a reasonable facsimile):


my $page = CS3::Page->new();
my $user = $page->get_userX();



sub main {
# body of the script happens here, except that now I'm a closure over the
# initial instance of user that was here when this script was first compiled. D'oh!
# We should instantiate $page, $user, etc. here and pass them to our other
# subroutines...
}

main();


The problem with this code is described in the perlmonks article. Lesson learned: do not mix globals and subroutines inside Apache::Registry. Also, Apache::Registry is a bit of a hack, and it's best to be aware of how it does its magic, cause we use it. Everywhere.

Saving!

Woo, almost missed a day there! Was too busy working to write in blog. :)

So, saving is implemented on the user_info tab now, and everything validates as it should. That means (gasp) all the conceptual stuff should be done, and it's just typing to implement the other two tabs now. !!

Why it's taking so long

It's the little snags. This morning, I spent... looks like almost three hours tracking down a bug that was preventing the state dropdown from validating correctly. I found it, it works now. But that was a "wasted" 3 hours not spent writing new code, but finding a finicky little bug.

That's why it takes so long.

I think it's time to save.

Well, well. I think the main part of the user preferences page is all done. I'm pretty much happy with the architecture - all the savable data is passed in JSON, it fills out a form, the user can edit fields, and send it back. I'll then have a very regular format that will require very little munging to get it back into writable form. Should work well.

Then, it's just a matter of doing pretty much exactly the same thing with the general_preferences. And then I can add RSS controls. I should be able to finish the front page today (8 hours), the general preferences page in let's say a day and a half (14 hours), and the RSS page in another 8 hours. So that's 30 hours left, and I should be able to get it done by the end of the week. I think that's what's actually listed on the burndown. I don't know why this is taking me such a long time, it's really not that hard conceptually. It's not even a lot of typing. I keep having to look up dojo things things though, and that's slowing me down. Also, my debugging environment is stupid and slow. Maybe after I get this done I can do something that doesn't require an extremely silly testing process.

Oh well. Time to finish this page up.

Monday

So, I got the _list.pcgi's and the Xmldoom stuff set up. This morning, I'm setting up a new export (weird), and then once that's all hammered out I'll be populating the state and country list boxes from Ajax calls. Won't that be a joy.

I think it makes the most sense to grab all the state information from state_list and just use a query filter based on the state for the state box. We'll have to see how that works. But it doesn't make a lot of sense to call state_list.pcgi every time the country box changes. So we won't do that.

So at the moment, I'm waiting for jkw to add me to the exports mailing list so I can get started conversing with the export contact. Maybe I should work on user_preferences... he seems to be taking a while.

States

Maybe I need to get rid of my obsession with doing things the Right Way. Or maybe I need to never, ever lose it. It's hard to decide.

So, the user_info part of the user preferences page is mostly done. Or I thought it was. Then I looked down at that troublesome "state" box in the address section. And I thought, "I should make another script to return state info in JSON and make this box read it, like security questions." Then I looked at the state table, and found that it has a country foreign key, which isn't specified in database.xml and doesn't have an xmldoom object associated with it. *cry*

So, I've got to make one of those, and then think of a good scheme for getting that data. What I'm thinking is having a parameter to state_list.pcgi of country, that defaults to USA. Because generally speaking, nobody cares about the stuff from Country, but compatibility dictates. I'll need a country_list.pcgi too, because I'll need a dropdown for country that determines what states you can select.

The whole reason we care about validating the state is that they aren't all in the database. Rumble.

Anyways, I can only work on this for so long today (let's say until lunch), and then it's all learning exports.

Success!

Such a dojo widget has now been created, and tested, and it works seamlessly and is beautiful. Now I need to start learning exports...

New dojo widget

So, it occurs to me that the extremely lazily loading FilteringSelect is something I might want to use in the future, so I'm making a new dojo widget. What it will do is accept an (optional) initial label and value, and that will show up. getValue returns the value, isValid returns true, all that goodness. But when you go to change the widget (click on it, etc.) it'll call the function you gave it as its storeGen attribute to create its store. I.E., the store does not get created until you need it. I'll call it (dun dun dun!) cs.dijit.form.LazyFilteringSelect. :)

Edit: A couple more features. setValue will simply set what gets returned by getValue, and setDisplayedValue will simply set the label (no lookups) UNTIL either you try to change the widget or you call widget.loadStore();