NUnit 2.4 was released a couple of weeks back and I’ve jumped on it. NUnit is traditionally a unit testing tool supporting the development process, but in my organization we’re using it almost exclusively for functional (black-box) testing. This latest version addresses a number of issues that previously made it difficult to do functional test without modifying the NUnit source.

By the way, I still believe NUnit is the best unit-testing tool out there–and it’s years ahead of competing products.

In my view, the big difference between unit-testing and functional test is this: in unit-testing, when a test fails, the product code is then fixed almost instantaneously–the NUnit tests are a developer’s tool and used to make sure his or her code is correct. But when used for product verification, passing and failing tests are merely used to show the state of the product.

If one of our NUnit tests suddenly starts failing, it is not fixed straight away. Instead, we raise a bug in our bug-tracking system and it then passes through the usual bug-fixing cycle. This means that a bug may not be fixed for weeks, months or even until the next major software release.

The problem, then, lies with reporting. I don’t want to set failing tests to Ignore; likewise, I don’t want them always to appear in our reports as failing. I want to be able to mark the tests as failing and I want to mark it with details of the bug I’ve raised. Even more importantly, because we run our tests on a range of platforms, I want to mark which platform the tests are failing on.

So for me, the big new feature of NUnit 2.4 is not the new assert syntax, it’s the new user-extensible attributes. The new PropertyAttribute allows me to attach information to my tests, and most importantly (and this is new) I get the property written out to the report XML.

What I can then do is use this information in my reports to colour test results as one of the following:

• Green when a test passes;
• Red when a test is failing for the first time;
• Orange when a test is failing but we’ve already raised a bug to track the issue;
• Blue when a test is passing but is marked as failing (i.e. the bug is fixed).

This is test heaven. The test analyst–who crucially may not even be a coder–can arrive in the office at 9am and have a report on his desk, and known immediately the state of the product.

So, on to the code. Here’s how I can mark my tests:

[Test]
[FailsOn(FailedPlatform.SqlServer2000)]
[Bug("#12345")]
[Priority(Priority.Core)]

This extra information appears in the XML report like this:

<categories>
<category name=”CorePriority” />
</categories>
<properties>
<property name=”FailsOn” value=”SqlServer2000″ />
<property name=”Bug” value=”#12345″ />
</properties>

Note that the Priority derives from the category attribute, so I can tell NUnit “I only want to run Core and High priority tests please.”

The final stage is to use some XSLT to transform this into a pretty report… and hey presto, we have a fantastic functional test system.

Many thanks to the NUnit guys for another superb release.