Brian's Waste of Time

Testing Dojo Widgets

I've been mucking with Dojo widgets a lot lately, and in writing a new base widget (for doing inplace widget creation using existing annotated dom nodes rather than templates) I finally ground my teeth against browser-refresh based testing enough to do something. I was stupid, automated testing of dojo widgets is easy!

Take this example:

    dojo.hostenv.setModulePrefix("skife", "/dojo-stuff/skife");
    dojo.require("skife.test");
    dojo.require("skife.widget.InplaceWidget");
    
    var create = function(s) {
        var node = document.createElement("div");
        node.innerHTML = s;
        var parser = new dojo.xml.Parse();
        var frag = parser.parseElement(node, null, true);
        dojo.widget.getParser().createSubComponents(frag);
        return node;
    }
    
    
    skife.test.suite("Testing org.skife.InplaceWidget", function(s) {
        s.test("Widget is instantiated declaratively", function(t) {
            var called = false;

            dojo.widget.defineWidget("skife.test.DummyWidget1", skife.widget.InplaceWidget, {
                postAttachDom: function() {
                    called = true;
                }
            });
                        
            var div = create("<div dojoType='DummyWidget1'></div>");
            
            t.assert("postAttachDom was not called", called)
        });
        
        
        s.test("In place nodes are attached", function(t) {

            dojo.widget.defineWidget("skife.test.DummyWidget2", skife.widget.InplaceWidget, {
                postAttachDom: function() {
                    t.assertNotNull("hi node not attached", this.hi)
                }
            });
                        
            var div = create("<div dojoType='DummyWidget2'>" + 
                             "<span dojoAttachPoint='hi'>Hi</span></div>");             
        });
    })

In this case I need to use declarative widget creation, so the hardest part was figuring out how to get the parser to run over the dom tree I had just made. The Dojo FAQ had two (different) ways of doing it, neither of which work anymore. Luckily the kindly folks in #dojo pointed me to ContentPane which does this (see the create function earlier).

Anyway, this served as a good reminder to me that not testing because you think something will be hard to test is stupid. Once you try and prove it is hard to test, then you have some legs to make excuses on, until then, kick yourself in the ass and go test some more :-)

Along the way I started cleaning up the JS test harness (re-namespaced into something less likely to have collisions!) I've been mucking with. I've been n the insane side of busy lately so pretty much everything not directly involved with $goal had fallen by the wayside for a bit. Fun to have some breathing room to pick things up!

0 writebacks [/src/javascript/dojo] permanent link

Dojo Deferred and DFS

A while back I talked about the "DataFlowSequence" which Tim and I came up with for dealing with concurrent XHRs. To recap, if you needed a result not yet available you would register interest in it and when it became available it would be yielded to your callback:

var dfs = new skife.ajax.DataFlowSequence()

// register callbacks

dfs.when("first", function(args) {
    log.debug("first received: " + args.first)
});

dfs.when("last", function(args) {
    log.debug("last received: " + args.last)
});

dfs.when("first", "last", function(args) {
    log.debug(["full name:", args.first, args.last].join(" "))
});


// supply data

dfs.supply("first", "Sam")

dfs.supply("last", "Jones")

In the interim Dojo created a Deferred class based on MochiKit's Deferred class, which in turn is based on Twisted's Deferred.

Using Dojo's goes like this:

var d = new dojo.Deferred();

// register callback

d.addCallback(function(name) {
    log.debug("name: " + name)
});


// supply data
d.callback("Sam Jones");

Deferred has an additional registerable callback, an errback, which is called if Deferred#errback(value) is invoked instead of Deferred#callback(value).

var d = new dojo.Deferred();
d.addCallback(function(v) { log.debug("success! : "  + v) })
d.addErrback(function(e) { log.debug("error! : " + v) })


if (Math.random() > 0.5) { 
    d.callback("yea!") 
} 
else { 
    d.errback("boo!") 
}

This would be handled in the DFS via

var dfs = new skife.ajax.DataFlowSequence()

// register callbacks

dfs.when("success", function(args) {
    log.debug("success! : " + args.success)
});

dfs.when("error", function(args) {
    log.debug("error! : " + args.error)
});

if (Math.random() > 0.5) { 
    dfs.supply("success", "yea!") 
} 
else { 
    dfs.supply("error", "boo!")
}

Is fun stuff to hack around with. The Mochikit and Dojo Deferred implementations are almost certainly more performant than the DFS -- they can work with arrays instead of hashes, which should be much quicker. They optimize the common case, wanting one result, and can handle the less common case of wanting several by having deferreds waiting on deferreds. I think I prefer the DFS syntax for needing multiple results, though. I need to play more with deferreds waiting on the results of other deferreds. The single argument nature might be nice to work with, rather than using ~named arguments.

Wheee! Reminds me of a silly hack I have been wanting to do to optimize the syntax of all this. The string value of a JS function includes the declaration, which includes the parameter names. These could be parsed out fairly painlessly, allowing us to do the DFS like:

var dfs = new skife.ajax.DataFlowSequence();

dfs.when(function(first, last) {
    log.debug(["Hello,", first, last].join(" "))
});

dfs.supply("first", "John");
dfs.supply("last", "Flynn")

This would be much slower, but is kind of nice :-)

1 writebacks [/src/javascript] permanent link

Scratching a JavaScript Testing Itch

I've been using jsunit for unit and some integration testing of my javascript, and it is extremely slick -- but I wanted a few things it doesn't have, so in traditional open source fashion, I started scratching my itch :-)

I've used lots of XUnits, but not written one before, so it was a fun experiment. As this kind of library is very API driven I started with what I though it should look like, it started pretty traditional, something like:

function testSomething() {
  assert(1 == 1);
}

function testSomethingElse() {
  assertNotNull("hello world");
}

But this really bothered me -- it was filling up the global scope with tests, and required the assertion functions be global as well. I thought about taking a step from JUnit, et. al, and putting them in a class, but... this is JavaScript, it has no class.

Okay, what does it have. It has a nice prototype and open object system. Hmm, that lends itself to builders, maybe something like:

xn.test(function() {
  this.testSomething = function() {
    this.assert("Math doesn't work", 1==1);
  }
  
  this.testSomethingElse = function() {
    this.assertNotNull("Strings are null", "hello world");
  }
})

That looked interesting, so I wrote some code to make it work. Woot! It wasn't quite right though, so I poked around some more and eventually wound up veering away from the XUnit canonical form and making stuff that looks like this:

  dojo.provide("test.test");
  dojo.require("xn.test");
  
  xn.test.suite("A Test Suite Named Bob", function(s) {

    s.setUp = function(t) {
      // it is good to store test-specific state on the test object
      // but don't collide with the publicly exposed methods =)
      t.setupCalled = true;
    }

    s.tearDown = function(t) {
      t.log("tearDown invoked on %s", t.name);
    }

    /* actual tests */

    s.test("Verify that setup was called", function(t) {
      t.assert("Setup was not called", t.setupCalled);
    })

    s.test("assertNotNull passes for empty object", function(t) {
      // t.log("first test invoked");
      t.assertNotNull("Empty object is null!", {});
    });  

    s.test("Test which fails on purpose", function(t) {      
      // t.log("second test invoked");
      t.assertNotNull("This failure is on purpose", null);      
    });
    
  });

Along the way I went ahead and introduced a dependency on Dojo as I use it in almost all my javascript anyway. I also introduced suites (I am dearly tempted to rename them groups instead of suites as they are not really suites). All fun and good, and works well. Had some spiffy output going to report results. Yea!.

Now that I have something that I like the feel of the API for... tim to tackle one of the itches that drove me to write it -- handling arbitrary async calls. The first stab was to have explicit asynchronous test functions, something like this:

  s.test.async("Asynch test which succeeds", function(t) {            
    
    dojo.io.bind({
      url: "/Give-Me-A-404", // 404
      error: function(type, thing) {
        t.succeed();
      }
    });
    
  });
  
  s.test.async("Async test which should fail", function(t) {            
    
    dojo.io.bind({
      url: "/Give-Me-A-404", // 404
      error: function(type, thing) {
        t.fail("Failure is on purpose");
      }
    });
    
  });

This, while kind of pretty API-wise, was annoying to implement as the async function wan't actually on the Suite object, so it got ickers. Using Suite#testAsync looked awful (I believe strongly in aesthetic APIs), so, much muckiting about and I wound up with:

  s.test("Asynch test which succeeds", function(t) {            
    t.async();
    
    dojo.io.bind({
      url: "/Give-Me-A-404", // 404
      error: function(type, thing) {
        t.succeed();
      }
    });
    
  });
  
  s.test("Async test which should fail", function(t) {            
    t,async();
    
    dojo.io.bind({
      url: "/Give-Me-A-404", // 404
      error: function(type, thing) {
        t.fail("Failure is on purpose");
      }
    });
    
  });

Where it is the same test() function but adds an async() function to the test object which changes its behavior. The change of behavior is important because idiomatic XUnit says that if a test completes without failing an assertion then it passes. With an async test you need to tell it when the test completes via the succeed() function. Having to call this in normal tests would get old, fast.

In the first real use of this I realized I needed a timeout, no problem, have the async call take a timeout (I made the default pretty short, this is a unit test, after all):

  s.test("Async test which fails via a timeout", function(t) {
    t.async(100);
  });

Woot!

A lot of where the design veers from standard XUnit form it is to accommodate JavaScript idiosyncrasies. For instance, if we look at the test() function we see a variable being passed in! This variable is the instance of the test case, which is also, as it so happens, the value of this in the test function.

JavaScript's scoping bizarrity makes it much more practical to pass the test case in as an argument, that way you don't have to capture the correct value of this, or pass around this as a context to other calls when using anonymous functions. Handy. The same thing applies to suite.

Defining it as a Dojo module is also handy, it makes building a real suite pretty easy, you just require each of the "suites" -- which is why I think I'll rename it to group or even testCase, but I hate the camel casing, so will think a bit :-)

Reporting success and failure is easy to override, the default just creates a definition list and plugs results in, like so. Not pretty, but easy to make pretty as time goes on.

The API exploration test cases are all online, but not especially polished. We'll see where this goes. I kind of like the feel of how it is coming together. A couple things I definitely need to do though are changing failure to raise an exception rather than allow the rest of the test to continue, which will make stack trace generation for tracking down failures more useful; and add some more assertion helper functions (oh, and add the one-argument form, always including an explanation is annoying once you have stack traces to see where it came from).

All in all, a fun scratch!

3 writebacks [/src/javascript] permanent link

JavaScript as Compilation Target

Richard MacManus:

To create complex web applications, 'high-level language to Javascript' compilers have come into play ... As web applications grow more and more sophisticated, with lots of user interaction, it gets harder and harder to hand code it all in Javascript.

Hiding Javascript behind another language's layer of abstraction is like killing puppies.

5 writebacks [/src/javascript] permanent link

Dojo Widgets: (Slightly) Beyond Hello World

There are quite a few good bare-bones introductions to Dojo widgets out there, but not a lot beyond the Hello World stuff. Here is a bit, as I find it and figure some of it out...

The recent 0.3 release added an amazingly convenient method for creating new widgets, dojo.widget.defineWidget, which I'll use for all my widget making -- it is just that much nicer =)

dojo.widget.defineWidget("skife.sample.CommentPane", dojo.widget.HtmlWidget, {
  templatePath: "http://morphy.skife.org/src/dojo-samples/CommentPane.html",
  templateCssPath: "http://morphy.skife.org/src/dojo-samples/CommentPane.css"    
})

Is barebones usage fora widget which doesn't actually do much. This defines a widget named skife.sample.CommentPane which extends dojo.widget.HtmlWidget. It has both HTML and CSS templates. When the widget is defined by this method it will be registered under the name CommentPane, which is kind of inconvenient as it basically strips all the namespacing out. C'est la vie.

The method takes a couple more arguments though, so we could make it look more like

dojo.widget.defineWidget("skife.sample.CommentPane", dojo.widget.HtmlWidget, {
  templatePath: "http://morphy.skife.org/src/dojo-samples/CommentPane.html",
  templateCssPath: "http://morphy.skife.org/src/dojo-samples/CommentPane.css"    
}, "html", function() {
  this._secretIdentity = dojo.dom.getUniqueId();
});

The third argument, html declares what contexts the following argument applies to, I think. Frankly, I have only ever done html widgets, no SVG or otherwise for me yet, so I mostly leave it blank (it applies to anything) or use "html" to specify that it is html specific.

The last argument is a function to be used as a constructor. Kind of handy, though to be honest I have found I rarely use them when making dojo widgets. Weird as I love my ctors in most programming. It could be purely aesthetic, this function is both super handy and butt ugly if you use anything beyond the second argument =) Anyway, I digress.

Fully static templates are pretty boring, though. The next step up is some variable interpolation. If we change our template to be (the still boring, but hey)

<div class="skife-comment-pane">
  <h1>${title}</h1>
</div>    

we can interpolate some values. Now, Dojo's templating is pretty abyssmal, and for anything complicated I have been using TrimPath's JST instead. It is not tough to make JST play nicely with dojo's packaging and compressing system (though to make the compression not break you need to apply a minor patch) but for fairly simple stuff, Dojo's works great.

Dojo's template processor looks for a field called strings on the widget to pull values from. The keys are properties on the strings object, and the keys are, in fact, used as keys, there is no kind of expression language. To support a default title, let's change the widget definition to add a title field:

dojo.widget.defineWidget("skife.sample.CommentPane", dojo.widget.HtmlWidget, {
  templatePath: "http://morphy.skife.org/src/dojo-samples/CommentPane.html",
  templateCssPath: "http://morphy.skife.org/src/dojo-samples/CommentPane.css",
    
  /* Default Property Values */
    
  title: "I am a widget named Bob"
});

While the widget has a title our template cannot find it. To handle this we will start to make use of the widget's lifecycle hooks. The useful one here is postMixInProperties which is invoked after the initial properties are set on the widget. We'll use this to set up our strings for interpolation into the template.

dojo.widget.defineWidget("skife.sample.CommentPane", dojo.widget.HtmlWidget, {
  templatePath: "http://morphy.skife.org/src/dojo-samples/CommentPane.html",
  templateCssPath: "http://morphy.skife.org/src/dojo-samples/CommentPane.css",
  
  /* Default Property Values */
    
  title: "I am a widget named Bob",
    
  postMixInProperties: function() {
    this.strings = {
      title: this.title
    }
  }
})

This will be invoked before the template is processed, but after any properties have been set, so we can now change the title either by setting it in the declarative style:

  <div dojo:type="CommentPane" title="Stuff about Dojo" />

Or programmatically if we are creating our widget that way:

var widget = dojo.widget.createWidget("CommentPane", {
               title: "More Stuff about Dojo"
             });

In both cases the title provided will be used instead of the default. It is worth mentioning, as this caught me initially, that if you use the declarative style you must have a default value assigned to the property in order to have the one from the xml node copied over.

Variable interpolation, in 0.3, is flat. I have heard rumour that it will support traversal of object graphs a la JEXL in 0.3.1, and a much more powerful templating system is "on it's way, sometime."

Ajax, not DOM Scripting, is the hot shiznit right now, so let's go ahead and fetch some remote data for our CommentPane -- the existing comments. The first step is to put somewhere in the template for us to display these comments:

<div class="skife-comment-pane">
  <h1>${title}</h1>
  <dl dojoAttachPoint="commentListNode"></dl>
</div>

This uses the standard dojoAttachPoint to attach the dl DOM node to our widget. We still need to put something into it, however. To do this we hook into a different lifecycle hook, fillInTemplate and add another parameter to our widget, commentsUrl, which is a url to GET the existing comments from a remote service, in JSON format, looking like this.

dojo.widget.defineWidget("skife.sample.CommentPane", dojo.widget.HtmlWidget, {
  templatePath: "http://morphy.skife.org/src/dojo-samples/CommentPane.html",
  templateCssPath: "http://morphy.skife.org/src/dojo-samples/CommentPane.css",
    
  /* Default Property Values */
  
  title: "I am a widget named Bob",
  commentsUrl: "!",

  /* lifecycle hooks */
    
  postMixInProperties: function() {
    this.strings = {
      title: this.title
    }
  },
    
  fillInTemplate: function() {
    var list = this.commentListNode;
      
    dojo.io.bind({
      url: this.commentsUrl,
      mimetype: "text/json",
      load: function(type, data, evt) {
        dojo.lang.forEach(data.comments, function(comment) {
          var dt = document.createElement("dt");
          dt.innerHTML = dojo.string.escape("html", comment.author);
            
          var dd = document.createElement("dd");
          dd.innerHTML = dojo.string.escape("html", comment.text);
            
          list.appendChild(dt);
          list.appendChild(dd);
        })
      }
    });
  }
});

The fillInTemplate function is invoked after the template has been translated to a DOM tree, and the various nodes have been attached. We fetch a list of existing comments from a remote service and populate them into the DOM dynamically. The comments won't appear until after the widget renders, but that is okay, better to show the user something as soon as possible. Properly we should also include an error handler in this, making the bind look like:

dojo.io.bind({
  url: this.commentsUrl,
  mimetype: "text/json",
  load: function(type, data, evt) {
    dojo.lang.forEach(data.comments, function(comment) {
      var dt = document.createElement("dt");
      dt.innerHTML = dojo.string.escape("html", comment.author);
      
      var dd = document.createElement("dd");
      dd.innerHTML = dojo.string.escape("html", comment.text);
      
      list.appendChild(dt);
      list.appendChild(dd);
    })
  },
  error: function() {
    var dt = document.createElement("dt");
    dt.innerHTML = "Skife"
    
    var dd = document.createElement("dd");
    dd.innerHTML = "Unable to retrieve existing comments, sorry."
    
    list.appendChild(dt);
    list.appendChild(dd);
  }
});

The URL used to fetch the comments is a parameter so that this widget can be used in different places, providing it is just adding another param in either style of creating widgets, here is the declarative one:

<div dojo:type="CommentPane" 
     commentsUrl="http://morphy.skife.org/src/dojo-samples/existing-comments.json" 
     title="Stuff about Dojo" />

It is important to be able to add comments as well, but I'll leave that as an exercise for the reader, or until next time! If you cannot stand the wait, look at how the sample code in this works =)

All the code from this is available here if you want to muck around with it.

Update! Scott J. Miles was kind enough to post about upcoming changes to defineWidget which address my "aesthetically, it's butt ugly" comment. Without further ado, Scott's comments pulled out from the comments and properly formatted:

Scott J. Miles:

Also, the constructor function (init argument) can now be specified before the properties, hopefully improving readability.

dojo.widget.defineWidget("skife.sample.CommentPane", dojo.widget.HtmlWidget, function() {
  this._secretIdentity = dojo.dom.getUniqueId() 
},{
  templatePath: "http://morphy.skife.org/src/dojo-samples/CommentPane.html",
  templateCssPath: "http://morphy.skife.org/src/dojo-samples/CommentPane.css"    
});

If you don't like that either, there are actually two other ways to specify the constructor. I will have some examples online shortly.

5 writebacks [/src/javascript/dojo] permanent link

DataFlow Style Sequencing for Concurrent Ajax Calls

There is a somewhat misleading notion that you don't need to worry about concurrency in JavaScript because JavaScript itself is strictly single threaded. When you have concurrent ajax requests you do need to deal with concurrency. You can (for RFC compliant browsers) have two sockets open to the server at a time. Making use of them for concurrent requests, or having some operation dependent on UI input and a server response forces you to deal with not knowing what will happen when.

The most useful mechanism I have found is based on dataflow concurrency. It is far from a full dataflow machine, but it basically takes the part where functions can (and will) execute when all their arguments are available, but not until then. Tim and I recently extracted the techniques we had been using into a nice general purpose tool, the DataFlowSequence. Setting it up works like this (note I am using a different implementation then what we extracted, I did it to explore a different algorithm):

    var dfs = new skife.ajax.DataFlowSequence();

    dfs.when("firstName", function(args) {
      alert(args.firstName);
    });

    dfs.when("lastName", function(args) {
      alert(args.lastName);
    });

    /**
     * Note that we explicitly return the full name
     */
    var fullname_key = dfs.when("firstName", "lastName", function(args) {
      var full_name = args.firstName + " " + args.lastName;
      alert(full_name);
      return full_name;
    });

    /**
     * Note that this depends on the return value of the previous
     * function.
     */
    dfs.when("salutation", fullname_key, function(args) {
      alert(args.salutation + " " + args[fullname_key]);
    });

All that does is create a managed set of functions which need some data to execute. The common use case is where these are things which need to be fetched via ajax, and might come back in any order. The dfs.when("name", "anotherName", function(args) { ... }) calls say that the function passed in depends on name and anotherName. When these variables have been bound, this function will be executed. We found it was more convenient to bind them onto a single args object than to match them positionally, but you could pass them in positionally instead of binding them by name. Your choice.

In a number of cases we need the output of one function to be input to another so we have the DataFlowSequence return a key which can be used by other functions to depend on output from each other. It's worked well.

Now, to supply some things to these so they do some actual work...

dojo.io.bind({
  url: "/xn/rest/1.0/first-name",
  mimetype: "text/json",
  load: function(type, name, evt) {
    dfs.supply("firstName", name);
  }
});

dojo.io.bind({
  url: "/xn/rest/1.0/salutations",
  mimetype: "text/json",
  load: function(type, greets, evt) {
    dfs.supply("salutation", greets);
  }
});

dojo.io.bind({
  url: "/xn/rest/1.0/last-name",
  mimetype: "text/json",
  load: function(type, name, evt) {
    dfs.supply("lastName", name);
  }
});

We don't know what order these will come back in, but regardless, the functions dependent on a particular piece of data won't be called until that data is in fact available, and ones not dependent on it can be called as soon as what they need is there =)

To do a better demo, I wired the above with a some visual gewgaws and did a mechanical blogreader style web service (faked them out) with forms you can submit to have the service "respond" right here.

Given Name

Family Name

Full Name

Greeting

Source and unit tests are available of course =)

1 writebacks [/src/javascript] permanent link

Embracing JavaScript

I've, lately, had the opportunity to do a lot of JavaScript and Ajax hacking in anger. Summary, holy crap, JavaScript hacking has come a long way =)

The biggest thing, I think, is to embrace JavaScript. It has such a dirty feel from ten years ago that this sounds anethema to many, but it is worth it in the same way that you are better off embracing SQL, or HTTP rather than trying to shove them under a lossy abstraction layer.

The vast majority of tools available, even the really sexy ones, try hard to shield you from JavaScript. While I do, at the moment, much prefer hacking Ruby and Java (and I am so damned happy that Ruby is blazing a fiery trail across the sky as it makes it a much easier internal sell ;-) to JavaScript, it really does you a disservice.

The main disservice it does you, I think, is to force you to think in terms of the classic web. This is the wrong model. JavaScript doesn't buy you visual effects, it buys you a better way of building big web-based systems. Welcome to the top of the line scaling problem #2, you just got promoted.

JavaScript is a weird little language, squirrely, but really rather powerful (meaning expressive and useful). Use it. Embrace it. Go buy Professional JavaScript for Web Developers -- it is the best guide to the language I have found.

Tool wise, I have been extremely impressed by Dojo. For one-off effects, or for use with Rails (because of the tight RJS integration) I'd probably still use Prototype, but my tool of choice has definately become Dojo -- when working in JavaScript.

Alongside Dojo, I have been getting great mileage out of TrimPath's JavaScript Templates. This is a great little templating system for Javascript, which combined with JSON (which rocks, forget XML for JavaScript hackery), lets you push most of your work out onto the client. This also rocks. I haven't used their "Rails in JavaScript" thing, nor the SQL liek bit, those seem a bit too much of trying to be something they are not, and I think taking a Model2 approach in JavaScript is the wrong way -- you can do NeXT/Cocoa style MVC very easily, why go to Model2?

JST is easy to hook into dojo, add a line to the JST source:

dojo.provide("trimpath.template")

Tell dojo where to find the trimpath stuff:

// path is relative to dojo
dojo.setModulePrefix("trimpath", "../trimpath");

And you can treat it like any other Dojo bit:

dojo.require("trimpath.template")

Transitive dependency management is fantastic. Sure, for production you will probably compile up a file with everything you need, but during development this rocks, and for exporting services (if you happen to be building a platform for social software, for instance) and want to make life easy for your users (developers), it is even more useful.

Anyway, lots of fun JS hackery, probably going to be writing a lot more about it.

2 writebacks [/src/javascript] permanent link

Ajax Experience

Okay, I still thnk that "Ajax" is a kind of silly name, but it looks like Dion and crew are co-sponsoring a conference on it just down the street.

While I may not make the conference, hopefully I can make it to the bar and say hi to folks =)

2 writebacks [/src/javascript] permanent link