Extending your testing strategy beyond go-live

In a recent post on his software testing blog, Eric Jacobsen talks about responding to bugs which ‘escape’ into production:

For each escape, we ask the following questions:

  1. Could we do something to catch bugs of this nature?
  2. Is it worth the extra effort?
…

The answer to #2 is split between “yes” and “no”.  We may say “yes” if […] the extra effort is manageable. We may say “no” when a mechanism is in place to alert our team of the prod error; we can patch some of these escapes before they affect users, with less effort than required to catch them in non-prod environments.

The point of Citrulu is to enable you to go one better: by allowing you to monitor your website in a test-driven way (rather than simply looking for exceptions or timeouts), it blurs the distinction between the two and effectively allows you to extend your testing strategy beyond the launch date.

It also allows you to write tests for scenarios which can only arise intermittently as a result of features of your production environment and are therefore very difficult to test in development.

For example, on Swing Out London I display the latest tweet from the @swingoutlondon twitter feed. It worked fine locally, but before I had implemented any caching, the following test screamed out at me that I was exceeding the API limits on a regular basis:

  So I know that the twitter feed is working 
    On http://swingoutlondon.co.uk
      I should not see "Sorry, for some reason we can't load the latest tweet"

When my first few attempts at implementing caching were less than perfect, Citrulu let me know that all was not well, and when I did get it right, a few subsequent failures of this test allowed me to identify that the caching service I was using was going down intermittently.

Citrulu is currently in Alpha – Sign up for an invitation at citru.lu