Monday 26 April 2010

Programing Tools : The Test Farm

In my discussions about programing tools, I've talked about there being a farm of build machines, so that builds can run in the background, and so that you always have a working build environment when you need one. The same has to be true of test.

Automated test is something which is more valuable to the software industry than anybody who doesn't have experience of it can imagine. It doesn't replace the test team, but it means they can focus their energies on finding new and better bugs rather than ensuring that old favourites don't rear their ugly heads.

As a developer, there are many things that stop me working. Sometimes its a problem, somthing I have to think about, mull over and ultimately solve. But sometimes I'm blocked by the fact that in order to do something, I first have to do something which is a hassle. Every time I am asked to test on a different platform, I feel that hassle. And I put off the move... on a bad day, I go to the warm (slightly tacky, a smelling of cheap perfume) embrace of the internet.

I wan't this hassle gone.

I want to say "Run this test in this environment" to my dev suite, and tlet the software find an appropriate machine.

And for that I need a test farm - just like build farm, it needs to have a number of daemon equipped machines, which are ready to run the tests I give them (and which ideally are able to clean themselves up totally - they really need to be able to reimage back to a newly installed state if I so wish)

But I also want wrriting tests to be easy. I must be able to go to my dev environment and say "now I want to write a test". It should give me a place to write that test, and appropriate tools to help me produce a working testable system.

Early stage test development is just a matter of setting appropriate applications running, and ammassing output. But I also later need to be able to write code to verify the output is correct.

And once I have tests that work, I need to be able to run them on every platform (perhaps scheduled overnight), and I need to be able to submit them, so that my tests enter the range of tests available to everybody.

Once I have those features, writing tests will be the standard way I choose to begin to develop my software. And as a company we will end up with many many tests.

The advantage of many many tests is overnight testing runs.

Overnight testing runs are really important. And they shouldn't just run overnight. They should run whenever there is spare time available on the test farm, testing every development build the build farm has generated. And they should report to the owner of a tree whenever a test failure is found.

On identifying a test failure, I need to know the history of the issue - has this failure occured before? On my branch? On the main tree? I need the system to binary chop for me, and find the point where the failure first occured. The test system, like the buiold system, needs to be fully aware of the version control system. And these failures should find their way into the issue tracking system. Ideally without my help - because if tests are failing, there is an issue - either with the test or with the application under test.

the test system should also be able to make decisions about the meaing of test failures - if a test begins to fail, that is important. If a test regularly fails intermittantly, that is less important. Tests which always pass need to be tested less often. Tests which regularly pass, but which also regularly break are the most important - every test suite has them, those tests which you recognise are the ones which regularly fail when you change something - the tests which are like a canary in a coal mine sniffing out impending doom. A test suite should be able to identify them and run them the most often, to identify problems as soon as they hit.

Oh, and the test team? Do they all go an sun themselves on the sunny sunny beaches of Acapulco? Maybe. But they have a new role. Because now they not only have a set of tools which make writing tests easier, they also have hundreds of tests being developed for them by developers... tests which migh themselves be buggy. They have a new range of applications to QA. But as tests are simpler applications, programs which can be held in ones head all in one go, they are more able to find the hidden defects, and ensure that the tests provide the QA the main application requires.

[Incidentally, I've ignored UI testing in this article. Thats because, frankly I know nothing about it. Testing that buttons work can be scripted. Testing that an application is usable will take more work. All I can say is that there is scope for manual tests which can be run by an individual with a script and be fed back into the same system. I don't quite know how this would work... consider it a reseash problem, and if you have any ideas, do get in touch]

No comments:

Post a Comment