Custom Nunit runner for parallel testing part 2

Last week I wrote about my desire for an nunit runner that could execute tests in parallel batches. Specifically selenium tests that were using the nunit framework.
It turned out to be pretty easy to implement.

The Nunit framwork provides a threadedtestrunner class, and you can refine what is run by the use of filters, which can be lists of tests.

So I was able to discover every test in the assembly, and create batches based on the number of threads I wanted. Each batch became a filter passed to a threadedtestrunner.

This meant I didn’t have to write any complicated code, no thread stuff ,just use the nunit framework.
I was also able to pass a custome listener to each threadrunner so that I could be notified of test start/end etc. I had to write a little locking around my resultwriter so that threads didn’t write results in a mixed jumble ,but the whole thing took about 1 day to get mostly working, and another day to finish off.

I was able to get our tests down from about 50minutes to about 9 when using 10 selenium grid nodes on just my desktop. (ok so my desktop is quadcore beast with 8gb of ram so that helps)

Following from the success of this I came up with a simple way to run extra diagnostics for failing tests.
My listener keeps track of every test that fails. At the end of the run my launcher re-runs each failing test one by one. Before each test it puts a simple signal in a file. The test startup and teardown methods look for this signal, and if seen they perform extra steps for diagnostics. It’s not super elegant ,but it is simple and works reliably.

This brings the added benefit of running the failing test twice. If it passes the second time then it’s more likely a timing issue that isn’t happening when run solo. If it fails again that is more consistent.

Next steps here are to introduce extra diagnostics capture methods to gather more system info during the test. If I could I’d capture video of the selenium test whilst it runs ,but the only solution I’ve seen for that is a python library and I’m running in c#. Ideally selenium would provide some mechanism via there java rc server. The makers of selenium grid appear to provide this capability in their cloud hosts environment, but it’s not clear what they use to achieve it.

It’s a shame that nunit doesn’t more natively support signalling between the test runner and the test, or even between test and startup/teardown. But i guess it wasn’t really intended for complex test execution, just simple unit tests. Overall though I was plesantly surprised by how easy it was to create a custom runner to do what I wanted without doing much more than use standard framework classes in a slightly different order to the standard runner.