Custom Nunit runner for parallel testing part 2

Last week I wrote about my desire for an nunit runner that could execute tests in parallel batches. Specifically selenium tests that were using the nunit framework.
It turned out to be pretty easy to implement.

The Nunit framwork provides a threadedtestrunner class, and you can refine what is run by the use of filters, which can be lists of tests.

So I was able to discover every test in the assembly, and create batches based on the number of threads I wanted. Each batch became a filter passed to a threadedtestrunner.

This meant I didn’t have to write any complicated code, no thread stuff ,just use the nunit framework.
I was also able to pass a custome listener to each threadrunner so that I could be notified of test start/end etc. I had to write a little locking around my resultwriter so that threads didn’t write results in a mixed jumble ,but the whole thing took about 1 day to get mostly working, and another day to finish off.

I was able to get our tests down from about 50minutes to about 9 when using 10 selenium grid nodes on just my desktop. (ok so my desktop is quadcore beast with 8gb of ram so that helps)

Following from the success of this I came up with a simple way to run extra diagnostics for failing tests.
My listener keeps track of every test that fails. At the end of the run my launcher re-runs each failing test one by one. Before each test it puts a simple signal in a file. The test startup and teardown methods look for this signal, and if seen they perform extra steps for diagnostics. It’s not super elegant ,but it is simple and works reliably.

This brings the added benefit of running the failing test twice. If it passes the second time then it’s more likely a timing issue that isn’t happening when run solo. If it fails again that is more consistent.

Next steps here are to introduce extra diagnostics capture methods to gather more system info during the test. If I could I’d capture video of the selenium test whilst it runs ,but the only solution I’ve seen for that is a python library and I’m running in c#. Ideally selenium would provide some mechanism via there java rc server. The makers of selenium grid appear to provide this capability in their cloud hosts environment, but it’s not clear what they use to achieve it.

It’s a shame that nunit doesn’t more natively support signalling between the test runner and the test, or even between test and startup/teardown. But i guess it wasn’t really intended for complex test execution, just simple unit tests. Overall though I was plesantly surprised by how easy it was to create a custom runner to do what I wanted without doing much more than use standard framework classes in a slightly different order to the standard runner.

7 responses to “Custom Nunit runner for parallel testing part 2”

    • i’ll have to ask. technically the code is what i’m paid for, so not sure how my employer feels about sharing that.

    • To give people ideas? To answer questions? I wanted to write about what i’d been up to, but I’m not allowed to post the code.

  1. Create a tutorial blog post somewhere where you show the community how this is done by abstracting the parts of the code that would indicate it’s proprietary work.
    Bit lame to use an open source framework, get a nice parallel extension working for it that many people search to use for their projects and then throwing that kinda mumbo jumbo to the people that are interested. After all NUnit is open-source, give something back to the community if you figured out something nice! 🙂

    • I’m not sure what such a tutorial would contain beyond what the post already says. I guess it could be restructured in tutorial format. But the intention was to indicate the possibilities. As with a more recent post, the idea was more to show what can be achieved to hopefully help others justify working on such a thing.Regardless of my own desire to share code or not, its not my decision to make. Sure it would be nice if every company that makes use of open source frameworks were obliged to share back the tools the build around them, but according to my contract what code I write in my job belongs to my employer, not me.
      I guess I could have just not written about it, as you’re not the first to be frustrated to find reference to the thing your interested in but no concrete code. However it was still my hope that by knowing the possibility exists, and having evidence of the kind of improvement its possible to get, that it will inspire people to take the same path. Maybe unencumbered by contracts that preclude their sharing of code.
      I have asked about sharing, and even highlighted the benefits of being more active in the community around these tools. However the answer was no.