Software Quality — the Whole not the Parts

Ben Higginbottom makes a significant point right at the top of this question about the Knight Capital fiasco:

Was the Knight Capital fiasco related to Release Management? On August 1, 2012, Knight Capital Group had a very bad day, losing $440 million in forty-five minutes. More than two weeks later, there has been no official detailed explanation of ...

Ben Higginbottom: Nightmare scenarios like this are not the result in the failing in any one discrete components that can be 'fixed' simply and sweetly by improving the process, but by small failures in multiple domains (I'm kind of reminded of a fatal accident investigation). So coding error, gap in the unit testing, weak end to end testing and poor post release testing coupled with a lack of operational checks when live. I can understand the desire for a 'silver bullet' fix, but in any complex system I've never known them to exist.

Sometimes, it's the whole, not the parts, that needs fixing.

A javascript unit testing framework in 50 lines – plus some assertions

Surprised by a TDD session at XP-Man in which we rolled our own unit test framework from scratch - using the growing framework itself to test itself as it grew - I added some basic assertions to my 50 line javascript test runner.

I suggest that assertions should be thought of as separate to a test runner. Assertions are at the heart of - and should grow into - your domain specific testing language. So you should think of assretions as owned by your application, not the testing framework. As your application grows, so domain specific assertions should grow. A testing framework should probably provide minimal assertions for the language it runs on: the 'primitive' true/false/equal/notEqual assertions which will be used to define more domain specific assertions; and doing the legwork of tedious but common cases, such as testing equality by value of complex objects and collections.

In the case of javascript, the below might do. The tricky thing for assertions is to come up with something that is fluent and not verbose. C# extension methods are perfect for this. In javascript one could add methods to Object.prototype, but that doesn't have the same scoping restriction as C# so feels too much like trampling all over everyone else's garden: it's poor namespacing and a failure of good manners.

/* Assertions for a test framework. http://www.cafe-encounter.net/p756/ */

function Assert(actual, assertionGender){
    this.actual=actual;
    this.notted=!assertionGender;
    this.Expected=  assertionGender ? "Expected" : "Did not expect";
}

function assert(actual){return new Assert(actual, true);}
function assertNot(actual){return new Assert(actual, false);}

Assert.prototype.IsTrue = function(msg){
    if( this.actual == this.notted ){ throw msg||this.Expected + " true.";}
};

Assert.prototype.Equals = function(expected,msg){
    if( (this.actual==expected) == this.notted ){
        throw msg|| this.Expected + " " + this.actual + " to equal " + expected ;
    }
};

Assert.prototype.EqualsByPropertyValue = function(expected,msg){
    var actualValue =  JSON.stringify( this.actual );
    var expectedValue = JSON.stringify( expected );
    if( (actualValue == expectedValue) == this.notted ){
        throw msg|| this.Expected + " " + actualValue + " to equal " + expectedValue + " by value";
    }
};

$(function(){
    new Specification("SpecificationsForPrimitiveAssertionPasses",{
        isTrueShouldBeTrueForTrue : function(){
            assert(true).IsTrue();
        },
        equalsShouldPassForEqualArguments : function(){
            assert( 1+1 ).Equals( 2 );
        },
        equalsByPropertyValueShouldPassForValueEqualArguments : function(){
            assert( { a: "a", b : function(){ return "b";}} )
                .EqualsByPropertyValue( { a: "a", b : function(){ return "b";}} );
        },
        assertNotShouldReverseTheOutcomeOfAnAssertion : function(){
            assertNot(false).IsTrue();
            assertNot(1+1).Equals(3);
            assertNot({ a: "a"}).EqualsByPropertyValue( { a: "b"});
        }
    }).runTests();

    new Specification("SpecificationsForPrimitiveAssertion_Failures",{
        isTrueShouldNotBeTrueForFalse : function(){
            assert(false).IsTrue();
        },
        equalsShouldFailForNonEqualArguments : function(){
            assert(1).Equals(2);
        },
        equalsByPropertyValueShouldFailForDifferingPropertyValues : function(){
            assert( { a: "a" } ).EqualsByPropertyValue( { a: "A" } );
        },
        equalsByPropertyValueShouldFailForDifferingProperties : function(){
            assert( { a: "a" } ).EqualsByPropertyValue( { aa: "a" } );
        },
        assertNotShouldReverseTheOutcomeOfIsTrue : function(){
            assertNot(true).IsTrue();
            assertNot({ a: "a"}).EqualsByPropertyValue( { a: "a"});
        },
        assertNotShouldReverseTheOutcomeOfEquals : function(){
            assertNot(1+1).Equals(2);
        },
        assertNotShouldReverseTheOutcomeOfEqualsByValue : function(){
            assertNot({ a: "a"}).EqualsByPropertyValue( { a: "a"});
        },
        assertTrueFailureWithCustomerMessageShouldOutputCustomMessage : function(){
            assert(false).IsTrue("This assertion failure should output this custom message");
        },
        assertEqualsFailureWithCustomerMessageShouldOutputCustomMessage : function(){
            assert(1).Equals(2, "This assertion failure should output this custom message");
        },
        assertEqualsByValueFailureWithCustomerMessageShouldOutputCustomMessage : function(){
            assert({a:"a"}).EqualsByPropertyValue( 1, "This assertion failure should output this custom message");
        }

    }).runTests();
});

A javascript unit testing framework in 50 lines

Last week at xp-man we did some TDD practise - by writing a tdd framework from scratch. The surprise was, how easy it was. So here, to my surprise, a very-bare-bones javascript unit test framework in 50 lines of code including the passing unit tests for the framework itself.

/*  www.cafe-encounter.net/p734/ */

function Specification( name, tests, outputContainer ){
    this.name= name;
    this.tests= tests;
    this.outputContainer= outputContainer || $('#testResults');
}

Specification.prototype = {
    name : 'UnnamedSpecification',
    tests : {},
    outputContainer : null,

    addMyselfToUI : function(){
        this.outputContainer.append( "<li><h3>" + this.name +"</h3><ul id='" + this.name + "'></ul></li>");
    },

    testDomId:function (test) {
        return this.name + "_" + test;
    },

    addTestToUI : function addTestToUI( test ){
        $('#' + this.name).append( "<li id='" + this.testDomId(test) + "'>" + test + "<input type='checkbox'/></li>");
    },

    markTestPassed : function( test ){
        $("#" + this.testDomId(test) + " input").prop("checked",true);
    },

    markTestFailed : function( test, ex ){
        $("#" + this.testDomId(test) + " input").prop("checked",false);
        $("#" + this.testDomId(test) + " input").after("<ul>Output:<li>" + ex.toString() + "</li></ul>");
    },

    runTest : function (test) {
        this.addTestToUI(test);
        try{
            this.tests[test]();
            this.markTestPassed(test)
        } catch(ex)  {
            this.markTestFailed(test, ex);
        }
    },

    runTests : function(){
        this.addMyselfToUI();

        for (var test in this.tests) {
            if (this.tests.hasOwnProperty(test)) this.runTest( test );
        }
    }
};

$(function(){
     new Specification("TestFrameworkSpecifications", {

        aTestShouldAddItselfInTheTestResults:function() {},

        aPassingTestShouldTickTheCheckbox : function() {},

        aFailingTestShouldUntickTheCheckbox:function () {
            throw "This thrown exception should appear in the output, indicating a failure";
        }
    }).runTests();
});

And the html doc to load and run it:

<html>
<head>
    <title>Minimal Javascript Test Framework</title>
    <script type="text/javascript" src="js/jquery-1.8.2.js"></script>
    <script type="text/javascript" src="js/wsl-testframework.js"></script>
</head>
<body>
<h1>Tests</h1>
<ul id="testResults">
</ul>
</body>
</html>

As written, I admit a dependency on jQuery, but you can replace that with document.write(). The obvious omission is a bundle of pre-defined assertions, but if you are developing a domain specific testing language then assertions should be part of that package, not part of the test running framework.

The obvious oddity is that the test suite demonstrates its test failure behaviour by failing a test; and you see that that is the correct result by inspecting the output.

This has led me to think that the TDD way to learn a new language is ... to write a TDD framework for it.