NUnit – multiple process execution to speed up test runs
I wrote ages ago about developing a tool to make nunit tests run in multiple threads part1 and part2. That was pretty good however it hit a slight issue in that if the tests weren’t good about the use of global statics, then things could get very messy.
So more recently I updated that program to split things into separate processes, instead of separate threads. This way I got better isolation between the main runner and the tests being run, and I got to work with tests that each wanted to mess with the spring container, as they are each in their own process so they don’t interfere.
Now the biggest challenge to running most of our ‘local in memory’ tests was that they were using a real sql server (somewhat heavy weight for what are technically ‘unit tests’ ) However a colleague recently altered those tests to use an in memory database, such that each new test fixture was playing in it’s own space, so it was no longer an issue to run multiple tests that attempted to update the same things at the same time.
Now this test bucket had been growing steadily in size and runtime, at the point that it took about 45 minutes to run all the tests in the bucket, we split it into 3 separate buckets based on projects, so that we could run them on 3 build agents. This obviously meant that we were using 3 times the machines, to keep the runtime down to an acceptable level. Again the runtime crept up until each of those 3 buckets was taking about 25 minutes. It is at this point that I started experimenting applying my multi process runner to these tests. until now I’d only applied it to the selenium tests written by my test team, but I’d kept the tool generically nunit, so it was time to play
Our standard build server is a 4 cpu win 2008 machine, with about 2gb of memory allocated, (these are all virtual) so I took the whole run, and used my parallel runner using 4 processes (one per core) and got the whole bucket (~1800 tests) to run in about 25 minutes. So it was no faster than the total elapsed time we already had, but using 1 build engine instead of 3. This was a pretty good result, but not good enough for me…
One issue I had was that the developer written tests are spread over a bunch of dlls, and the nunit framework only lets you give a single runner a single dll. The way I originally wrote the code was based on a single dll with all the tests in, and when I enabled it to work with lots, I basically just wrapped the whole logic in a giant foreach loop. This was pretty inefficient for dlls with relatively few tests in. So I refactored it to pre-load all the dll information, so that it would keep allocating work to free processes as long as there remain more dlls with tests in, rather than wait for a single dll to finish all its tests before starting on the next. making this change pulled down the runtime to around 22-23 minutes, a couple of minutes of runtime just for handling the process allocation logic a little more sensibly. (I know it doesn’t sound like much, but we are talking around 10% improvement here!)
At this point I asked our infrastructure team to hook me up with a build engine that has 8 cpus assigned, instead of the standard 4. This allowed me to boost the number of processes to 8, in theory double the cpu, double the speed, so I had high hopes. I was pretty happy with the results, the change brought the runtime down to about 13.5 minutes! now we’re talking. It’s not a liner improvement, but suddenly things are looking pretty sweet, we can use 1 build engine with the cpu allocation equivalent of 2 of the old build engines, but achieve half the runtime for the entire suite of tests compared to using up 3.
Now, drunk on the possibilities of speed, I scoured the code to figure out ways to squeeze a little more performance out. In doing this I spotted some logic I had put in place for the selenium tests, after launching a new process i actually sleep for 5 seconds to give it some time, this is because I found the selenium server would fail to create its temp folders if multiple processes talk to it in very quick succession (seems like a bug to me) but this sleep was totally unnecessary for the in memory unit tests, so I made it configurable, the existing tests could keep the sleep, but the new ones would reduce it to zero. I wasn’t sure how much impact this would really have, since after the initial launch of processes we could be waiting in other places.
However having made the change I wiped about another minute of the run time, now getting an average of 12.5 minutes per run. Again, 1 minute off may not sound like a big deal, but if you’ve read this far in a post about making tests go fast, I figure you are the kind of person that knows how much of a difference it can make to shave those extra seconds off of a test run. At this point we’re chasing the golden possibility of having a serious commit test bucket, running 1800 tests, taking less than 10 minutes. (not quite yet, but I’m so close I can taste it)
Unfortunately at this point I’m also out of tricks, I’ve looked the code over and there are no obvious places to make a difference. One great hope was a change made by one of the developers to persist a version of the in memory db and deserialise it into memory rather than recreate it from scratch each time. This seems like a great idea, and made a big impact when running the tests serially, but in practice it yielded no appreciable change to running them in parallel.
So I was chatting with the dev architect, and told him that this is as good as it gets, there are no more software improvements I can think of, the only thing left is throwing more brute power at it. So he did…
I came in the next morning to find an email showing the cpu load on a build agent with 24 cores! and 8gb ram!, the runner was pushing it pretty hard, and achieved a runtime of just 5min30seconds, it has to be said I was pretty damn happy with that result. Sure it’s not a linear improvement, but holy cow, we took a test run that took over an hour run serially on one build engine, and dropped it to barely over 5 minutes, the product takes longer than that to compile.
It’s possible that we won’t leave a build engine with that much cpu assigned all the time, however it is great to know that when we want to, we can turn the dial up to 11 and really help crank through the changes.
New Gadget: ASUS Transformer Prime
This week I got my new Asus Transformer Prime, and it is awesome.
There are many professionally written reviews all over the web, and all found it to be the best android tablet on the market right now. There has also been a fair amount of press around reported short comings and failures.
When reading xda-developers prior after making my pre-order and before it arrived, I was beginning to get a little nervous with reports of dead pixels, DOA units, units without serial numbers preventing updates, wifi being hopeless… seemingly many people were unhappy.
However, the thing I clung to was that all those professional reviews couldn’t have messed up, if the thing had real serious issues it would never have had such glowing write up.
When my unit finally arrived, I waited patiently to play. I was at work when the package arrived, and knew that if i opened the box to play I would get nothing done. So I waited.
Eventually the evening came and I opened up the box and plugged in. I did immediately try to power up, but got no response. So I left it to charge. In total it took about 50 minutes on charge before there was enough juice to respond to attempts to power up. I realised that most (all?) claims of ‘DOA’ were likely people freaking out that it didn’t respond once plugged in, not realising the amount of time that might be required on charge before it would respond.
Of course the first thing I checked for was dead pixels, and was relieved to find one. Then I checked serial number and that was fine to. I let it download all my apps (I have an android phone so it synced all the apps installed there to my tablet)
Once that had finished I went checking for the Android 4 update (Ice Cream Sandwich) and found it waiting, so pretty much immediately I hit the upgrade path and picked up the new OS level, and a camera driver update.
After the update and reboot, I was playing wth ICS to see what was what, when I realised that the wifi did keep dropping out on me, and re-establishing. Perhaps this issue was a real one, and at the rate it was happening it would have been a pretty serious issue. But then it occurred to me that I’d had no such problems pre-upgrade, and my instincts said to try a clean power down and back up. (my phone sometimes has the GPS go mental, tracking my position about 100 yards offset from my position, a reboot always clears it up) As suspected, a reboot cleared up the problem, and a week later I can say I’ve had no issues with wifi dropping. Some say the range is poor, I can’t say I’ve noticed, I get a good stable connection everywhere at home and at work, so I guess it’s good enough.
After a week of paying I can honestly say I love it, all the awesomeness of a tablet, but with the addition of a keyboard (on which I’m typing this) which is a little small, but pretty good for writing long form content without messing around with the on screen keyboard. Also the battery life is at least as good as claimed. I have to say I’ve not pushed it that hard, I charge it every night, since that is the habit I’m in with my phone. However today, I sat in bed this morning reading engadget’s distro magazine, my wife and I watched the 2 hours of ‘star wars uncut’ (pretty cool, check it out) and now I’m writing this. The tablet shows 82% charge and the keyboard is at 52%. if I tried this on any laptop I’ve ever owned it would be on its last legs by now. And my phone would likely be looking in serious need of a top up after that much video played, and streamed over wifi.
One complaint that all the reviews made, was not specific to this tablet, but rather the continuing issue with all android tablets is ‘not enough tablet apps’.
I can now appreciate what they’re saying, but I think it’s not what you might think. Every android app runs, and just expands to fill the screen. For many that is fine, I’m using noteeverything right now, and it works great at tablet size. However when you use those apps that are ‘tablet optimised’ you do feel like you get such a nice experience, that it is a little disappointing using apps that don’t make the most of the large area and resolution. So it’s not so much that you can’t get apps to do stuff, just that some of them leave you wanting more from the interface once you’ve seen how good it can be. I guess as those reviewers noted, it’s just a case of time.
Obviously you can’t really give a fair review of any device like this based on just one week. time will tell. However so far I love it, have had really none of the
reported issued, and find it to be every bit the amazing device that the early reviews reported. If you are on the fence, get off it and go buy one, you won’t regret it.
CraftSuppliesUSA artist’s sketch pencil
I wrote this post entirely using the wordpress for android app on my phone, which then decided to lose everything I’d written ;-( So here I wrote again thinking that I may never trust the android app again.
Previously I wrote about having made an oak book stand for my wife’s Christmas present. In addition to that I also decided to make her an artists sketch pencil.
We originally saw these at a craft faire in Winchester, as part of a large selection of pens being sold by a woodturner. Kat and I were both drawn to the unusual nature of these sketch pencils. Basically the mechanism is a 4 jaw ‘chuck’ which you operate by pushing the end plunger to open up the jaws. They take a 5mm lead in various colours, and when released, the 4 jaws grip the lead securely.
The guy selling them mentioned that you can’t get the kits in this country and they have to be imported. I forget now much he sold them for, but they were not cheap. I often feel a little pang of guilt at this kind of time, since I’ve normally shown quite an interest in the goods being sold by a woodturner, but only ever out of interest to go away and make them myself, never to actually buy them.
Having remembered them I did a little search and found them sold by Craft Supplies USA Woodturners catalog, unfortunately having to get them from the US makes things a little costly due to the shipping, so I ended up buying a couple of those sketch pencils and a couple of their ‘tool box’ variants. The tool box variant have little hex ‘nuts’ at the centre and plunger end in order to stop the pencil from rolling around when put down.
In construction I had a couple of mishaps from not really reading the instructions on drilling the blank (how hard can it be to drill a hole through piece of wood?) Basically I had the drill speed way to slow, and didn’t leave enough spare length on the blank to allow me to stop without drilling all the way through. So on two separate blanks I managed to blow out the base of the blank as I got to the end of the drilling (the bottoms literally exploded into fragements ;-() This left me not very happy. However after taking a little more time to read up on it, I sped up the drill, made myself a better jig for supporting the blanks, and went more carefully and all was well.
The tool box variant was a little bit strange, in that the rear hex nut is not a push fit into the barrel, rather having turned the barrel you need to trip a few mill off the rear end, back down to the brass tube, then glue the nut onto the brass tube. This also means that there is no snug fitting around the plunger (as there is on the artists variant) The finished product is still cool, but I feel it would be nicer with a push fitting that provides something close to the rear fitting of the artists sketch pencil.
I think this made a really nice gift and I was happy with how it turned out. A colleague at work expressed an interest in them, but I think the cost shipping would make them quite difficult to make and sell at a reasonable price. That said I do also have my eye on the soft touch kit for making stylus’ that work with capacitive touch screen tablets (my transformer prime should arrive tomorrow) so maybe I could order another batch of kits. If I convince people to buy them from me I might offset the cost of making my own…
Adjustable oak book stand
This christmas my wife and I agreed that we would make our gifts for each other. Given we made the decision a month or so before christmas it allowed plenty of time to figure out a plan, order supplies, and have some weekends to actually start making..
Very quickly I knew I wanted to include some artistic element, and obviously I wanted to make it personal. A few months back Kat had drawn a design of her name such that it had rotational symetry (looked the same upside down) and it struck me that taking that design and somehow rendering it in wood was a nice idea.
For a while I toyed with various thoughts about simply cutting the design out and mounting it as a purely artistic piece, but I really prefer to make functional things. I like the idea that what I make will enter every day usage filling some role.
Then I had a flash of inspiration, Kat often writes longhand in notebooks, then later refers to them whilst typing into her computer. She once made a comment about the slight inconvience of not having the book leant up so it was easier to switch views between monitor and notbook. So I decided to make a book stand, with her name design embossed on the front.
Here it is
It supports 3 positions
The last one maybe a little too upright since a heavy book pulls it over forwards, but for a few sheets or something light it’s fine.
The notches in the feet were cut on my bandsaw whilst they where still a single piece of wood, then I split it lengthwise to get the two feet.
The dowels were hand turned on the lathe to be the right diameter to drill convienient holes for them. They are not yet glued in place as I thought I might want to be able to adjust things, but actually it might be better to just fix them so they stay centred.
There are two little retaining arms that can swing up to hold pages back. That are just held by a nail, the head of the nail is a tight fit in the arms, and a slightly loose fit in the base of the stand to allow them to turn.
The patern on the front was created by first cutting out a stencil, then drawing around that onto the wood in pencil. Then I used a burn tool to scortch the outline, before finally using a burr tip ina rotary tool to make the texture. It was my first attempt at texturing like this and I’m not altogether happy with it. It looks ok at a distance, but it might have worked better if I had coloured it. I did consider doing so, but lacked a fine brush and didn’t want to ask Kat for one since it would have given away elements of the gift.
It’s finished in woodwax, which I like for things intended to be handled. Hopefully it will last well, and live up to it’s intended use to make it easier to reference notebooks whilst typing.
New workshop!
So I figured it’s about time I wrote about my new workshop. In truth I’ve not spent a huge amount of time building or making in it yet. I’ve mostly been doing setup stuff to get myself in a position to make something.
When we were looking for a new place to live, I was the cause of some serious complications. I was in a house with a large garage (a little under 7m x 4m) and whilst I did keep the car in it, there was plenty of room for me to have a substantial workshop. This meant the only places even open to consideration to move to required a garage, and even a single garage ran the risk of not really being enough.
When we were looking around this place, I knew it had a garage in a block, and a shed. As we were shown around the house I was favorably impressed, but still wondering about the workshop possibilities. First we saw the garden and the shed. It’s a fairly large shed as far as such things tend to go, not terribly tall, and in need of a little TLC, however it did have power and I was musing about whether I’d be able to make a workshop out of it. When the previous tenant (who was showing us around) said ‘Well of course there is also the *double* garage’
oh…it’s a DOUBLE garage in a block…well, suddenly things started to look more promising.
And so it was, whilst it is in a block, and there is no power directly to it, it is close enough to be able to solve that problem with a long extension lead. Obviously the extension was picked for its ability to handle the load placed by my machines, and with thermal cut off etc.
The double garage is almsot exactly 5m by 5m, and this time there was plenty of parking around so I had no intention of wasting space with the car 
25 square meters of workshop space… but what to do with it? when we moved in all my stuff got summarily dumped in random ways into the garage, filling the space, such that it was impossible to get around it all.
Before really starting to move everything to clear the space I decided to make a sketch of roughly how I could lay out the room.
With the sketch in place and a reasonable idea of where I wanted things to end up, I set to shifting everything around to try to get a sense of order. It took a lot of effort to bring sanity to the space, but eventually I’d cleared most things off to the edges of the space, shelving filled and tools setup. Leaving just the central islands (lathe and workbench) to complete.
However before that I’d need light!, I had been relying on sunny days and leaving the doors open to get things moved around, but a quick test with my 1 angle poise lamp with the doors shut showed that it would not be anything like enough, even with a few I figured I really needed to install some strip lights.
In the absence of a permanent power supply, I decided to buy 2 strip lights and wire everything as you would for a regular lighting circuit, with 2 lights spaced evenly on joists over the 2 main working areas of the workshop, I wired both into a junction box, linking in a single switch and power supplied via a regular plug with a 3Amp fuse which just plugs into the extension. These two lights make a HUGE difference, immediately I can close the doors and feel like I’m in a workshop rather than just a garage with tools in.
That done I turned my attention to my two planned islands in the middle of the workshop. Where as in my previous garage I bolted my workbench and lathe to the wall, here I didn’t want to do that, partly to avoid too many holes in the walls that I merely rent, and partly because bolting to the wall has some serious limitations.
Yes you get stability, a rock solid surface to work with. (Though the lathe still caused vibrations through it’s mounts when turning an uneven/off-centre piece.) However you can’t work with large items on the workbench, and whenever I tried to do hollowing on the lathe, I was butting my chisel handles against the wall, limiting my movement.
So I was keen to have both as islands in the centre of things, with supporting machinery arrayed around the outside.
Another benefit, particularly for the lathe has been the ability to create a clamp to hold my dust extractor hose in place behind the lathe bed, positioned to capture as much dust as possible in turning and sanding operations. This is really easy then to switch back and forth between the lathe and the bandsaw.
So far I’ve not done any serious operations on either the workbench or the lathe, so I can’t see how much either will move/vibrate. I’ve put in a few bracing pieces to stiffen the structures of both, but it’s possible that the workbench will be too light, under the force of certain activities may be inclined to move. Only time and use will tell.
I’m really happy with the workshop, I feel like I’ve got lots of space to work in, I also found enough space to set up my dart board, which spent the last 6 years gathering dust. So when I’m waiting for glue to cure, or just in need of some contemplation time I can throw a few darts, listen to a podcast. Generally chill out in my man-cave 
