And *that* is why you need multiple test harnesses

I'm a devotee of black-box testing. Yes, unit testing has been all the fashion for years now, but I confess that I think it's often carried much too far: testing every input of puny little classes is a fine way to waste time for very little benefit. By contrast, a true black-box test that surrounds the outside of the system, pretends to be a real user as honestly as possible, and puts the system through its paces, is invaluable. When time is tight, I'll always work on the black-box first, because the bang for the buck is very high. So CommYou's black-box harness is pretty elaborate, pulling up multiple browsers and pretending to be several users operating in the real UI, with no cheating.

That said, black-box testing has its limitations -- in particular, reality sometimes rudely intrudes. Take, for instance, the new LJ integration. One of the things I want to test is that the social network synchronizes properly: when you change your flist on LJ, the next time you synchronize the same changes should show up in CommYou. Problem is, LJ is Very Very Slow about propagating those changes, even internally: the sad fact of the matter is that it can take an hour before LJ *admits* that your flist has changed, and there's not much CommYou can do to speed that up. So it's essentially impossible to automate these tests for real, because that could take an hour per test.

This, however, is why CommYou (like almost everything I write) is based on the intensely-pluggable "Ecology" architecture. Basically, I'm a fanatic about making the whole system pluggable and indirectly-referenced, with no static references to anything. It's a mild pain upfront, but times like this are when it pays off.

So yesterday's project was writing a new test harness, at the mid-level. The core of the system is all real, so it's higher-level than an ordinary unit test, but it talks efficiently to the APIs instead of the slower web pages, and it replaces the LJ communications with a little module in the Ecology that simulates LiveJournal. So instead of it taking an hour to propagate each flist change, it's instantaneous, and I can hammer the real system with dozens of flist changes in a couple of seconds. And now I have an API-level test harness for these fiddlier tests that need to fake out the internals of the system. Quite useful: I'll do more with that.

(End result: 3 bugs found, out of 300 lines of new or rewritten code to import LJ communities as well as friends. Not too bad...)

And *that* is why you need multiple test harnesses

I'm a devotee of black-box testing. Yes, unit testing has been all the fashion for years now, but I confess that I think it's often carried much too far: testing every input of puny little classes is a fine way to waste time for very little benefit. By contrast, a true black-box test that surrounds the outside of the system, pretends to be a real user as honestly as possible, and puts the system through its paces, is invaluable. When time is tight, I'll always work on the black-box first, because the bang for the buck is very high. So CommYou's black-box harness is pretty elaborate, pulling up multiple browsers and pretending to be several users operating in the real UI, with no cheating.

That said, black-box testing has its limitations -- in particular, reality sometimes rudely intrudes. Take, for instance, the new LJ integration. One of the things I want to test is that the social network synchronizes properly: when you change your flist on LJ, the next time you synchronize the same changes should show up in CommYou. Problem is, LJ is Very Very Slow about propagating those changes, even internally: the sad fact of the matter is that it can take an hour before LJ *admits* that your flist has changed, and there's not much CommYou can do to speed that up. So it's essentially impossible to automate these tests for real, because that could take an hour per test.

This, however, is why CommYou (like almost everything I write) is based on the intensely-pluggable "Ecology" architecture. Basically, I'm a fanatic about making the whole system pluggable and indirectly-referenced, with no static references to anything. It's a mild pain upfront, but times like this are when it pays off.

So yesterday's project was writing a new test harness, at the mid-level. The core of the system is all real, so it's higher-level than an ordinary unit test, but it talks efficiently to the APIs instead of the slower web pages, and it replaces the LJ communications with a little module in the Ecology that simulates LiveJournal. So instead of it taking an hour to propagate each flist change, it's instantaneous, and I can hammer the real system with dozens of flist changes in a couple of seconds. And now I have an API-level test harness for these fiddlier tests that need to fake out the internals of the system. Quite useful: I'll do more with that.

(End result: 3 bugs found, out of 300 lines of new or rewritten code to import LJ communities as well as friends. Not too bad...)