Creating a master test plan not only consists of creating a draft MTP, doing a product risks analysis and creating the test strategy. You also have to estimate the project. In this post I describe how I do this.

In January I asked Pradeep Soundararajan (@testertested) how he estimated test projects. He posted an blog on it which you can find here. I asked him this question because I am intrigued about the use of Rapid Software Testing (RST) when testing software. But one of the things I thought it lacked was an approach to estimate the test project. Why? Because RST execute tests in an more informal approach. You ‘explore’ the software to give an advice about the insight of the software quality. To me this means you execute a lot of tests with the use of a lot of oracles and other documentation to get the most information about the test object (or System under Test). I think this approach has an added value to scripted testing. As I said in an earlier post to me testing consists of checking, exploring and accepting. But still I had this question on how I should plan the exploring. Please read Pradeep’s post about this to get some more information.

The following question of course is: How do I estimate a testing project? There is a glitch here. In Holland we most of the time follow a formal checking process and the most estimated I do are based on this checking. For this I use different methods. I’ll explain two of these methods here. Why only these two? Because these two are the ones I use the most and they complement each other (I’ll never use only one of these methods to estimate the testing). These methods is based on metrics from other projects I have done in the past and ratios from colleagues of mine.

At first I use simple ratios to estimate the testing work. These ratios come from the book TMap® NEXT and I have proven them in past projects. In past projects I’ve collected metrics since I became a testing co-ordinator at the various clients I’ve worked for. This collection is now a database consisting of almost two dozen projects I’ve done with all the metrics I need (or think I’ll need in the future). From these metrics I extracted a few calculation rules. Like how many time is spend per testing phase, amount of defects per test object, amount of follow up time on these defects and these rules are per quality attribute that have been tested. I also have collected additional information but I don’t use these jet. My colleague Rob Baarda has a lot more metrics on a lot more projects and he is using them successful in various estimates. But the ratios that are given in TMap® NEXT and that I use are for a system with good, but complex documentation:

• Preparation 6%;
• Specification 54%;
• Execution 21%;
• Completion 2%, and
• Control and Setting up and maintaining infrastructure 17%.

For a system test with an inadequate documentation:

• Preparation 21%;
• Specification 33%;
• Execution 24%;
• Completion 5%, and
• Control and Setting up and maintaining infrastructure 17%.

As you can see these ratios are especially for checking the software with scripts. When you add another ratio to this you can extrapolate the time spend on testing and of a testing phase. This ratio is the relationship between development and testing. In general I say that development vs. testing is 60 to 40. I’ve proven this ratio in organisations that don’t have a lot of quality control in place. When quality control of the project is better this ratio can go down, but when quality control lacks even the most basic quality checks I will raise the ratio on development vs. testing. With these ratios I can extrapolate the amount of hours spend on testing. And from there use the ratios stated above on how much time is spend per phase.

When I have this estimation I do another to check if I’m correct. With the use of my collected metrics I can extract a factor on how many time is spend on the different phases in the testing project per risk class, see my Lean MTP (part 2) post for more information about risk classes. With these factors I can give a more precise estimation on the testing activities. I go through the global requirements and look for test cases per object part. These object parts I already acknowledged when I was doing the product risks analysis. The test cases shall not be the precise amount, but a estimation of them. These estimated test cases are hold against the risk class of the object part, the percentage of test cases I can reuse from former projects (test ware) and hands on experience of the testers. This all together leads to an estimation on the time spend on specifying the test cases.

To estimate the other phases I also have to look at other things. Like for the preparation phase I look at the quality of the documentation. If the quality of the documentation is OK, or the past experience is that the quality of the documentation is normally OK I add some time per type of documentation. This time is raised based on the quality of the documentation. Every document has its own time for preparation. And when evaluations are implemented I also calculate this (more time spend in reading and evaluating the documentation and less time spend in execution).

Within the execution also take into account the amount and experience of testers in the project. Experienced testers shall execute the test cases faster or can test the object parts that have a higher risk class. The estimated amount of defects is needed here. This percentage is needed to estimate the amount of rework that is done in the testing and development.

With all this together I now can estimate the amount of time spend on control and setting up and maintaining infrastructure. The estimation of control is dependent on the lead time of the testing, the amount of testers and the experience of the testers. The estimation for control and setting up and maintaining infrastructure is the most difficult to estimate. So for these two I use the earlier ratio of 17% of all testing activities.

The test completion phase varies per client. Some clients have a formal project completion policy some end the project after it is shipped successfully. That’s why I most of the time also use the ratio of 2-4% of all testing activities. When a formal project completion is done with conserving the test ware there should be more time spend on this phase.

This is how I estimate a project. Part of this is because in the Netherlands most testing is done formally and by checking the scripts. Checking is done most of the time on a ‘Friday afternoon’. In new projects I’ll use Pradeep’s estimation for exploring the system.

How do you estimate your testing project?

1. Trackbacks Says:
   September 10th, 2010 at 23:04
   • Using metrics to estimate your testing | Testing Excellence