Creating a master test plan not only consists of creating a draft MTP, doing a product risks analysis and creating the test strategy. You also have to estimate the project. In this post I describe how I do this.

In January I asked Pradeep Soundararajan (@testertested) how he estimated test projects. He posted an blog on it which you can find here. I asked him this question because I am intrigued about the use of Rapid Software Testing (RST) when testing software. But one of the things I thought it lacked was an approach to estimate the test project. Why? Because RST execute tests in an more informal approach. You ‘explore’ the software to give an advice about the insight of the software quality. To me this means you execute a lot of tests with the use of a lot of oracles and other documentation to get the most information about the test object (or System under Test). I think this approach has an added value to scripted testing. As I said in an earlier post to me testing consists of checking, exploring and accepting. But still I had this question on how I should plan the exploring. Please read Pradeep’s post about this to get some more information.

The following question of course is: How do I estimate a testing project? There is a glitch here. In Holland we most of the time follow a formal checking process and the most estimated I do are based on this checking. For this I use different methods. I’ll explain two of these methods here. Why only these two? Because these two are the ones I use the most and they complement each other (I’ll never use only one of these methods to estimate the testing). These methods is based on metrics from other projects I have done in the past and ratios from colleagues of mine.

At first I use simple ratios to estimate the testing work. These ratios come from the book TMap® NEXT and I have proven them in past projects. In past projects I’ve collected metrics since I became a testing co-ordinator at the various clients I’ve worked for. This collection is now a database consisting of almost two dozen projects I’ve done with all the metrics I need (or think I’ll need in the future). From these metrics I extracted a few calculation rules. Like how many time is spend per testing phase, amount of defects per test object, amount of follow up time on these defects and these rules are per quality attribute that have been tested. I also have collected additional information but I don’t use these jet. My colleague Rob Baarda has a lot more metrics on a lot more projects and he is using them successful in various estimates. But the ratios that are given in TMap® NEXT and that I use are for a system with good, but complex documentation:

• Preparation 6%;
• Specification 54%;
• Execution 21%;
• Completion 2%, and
• Control and Setting up and maintaining infrastructure 17%.

For a system test with an inadequate documentation:

• Preparation 21%;
• Specification 33%;
• Execution 24%;
• Completion 5%, and
• Control and Setting up and maintaining infrastructure 17%.

As you can see these ratios are especially for checking the software with scripts. When you add another ratio to this you can extrapolate the time spend on testing and of a testing phase. This ratio is the relationship between development and testing. In general I say that development vs. testing is 60 to 40. I’ve proven this ratio in organisations that don’t have a lot of quality control in place. When quality control of the project is better this ratio can go down, but when quality control lacks even the most basic quality checks I will raise the ratio on development vs. testing. With these ratios I can extrapolate the amount of hours spend on testing. And from there use the ratios stated above on how much time is spend per phase.

When I have this estimation I do another to check if I’m correct. With the use of my collected metrics I can extract a factor on how many time is spend on the different phases in the testing project per risk class, see my Lean MTP (part 2) post for more information about risk classes. With these factors I can give a more precise estimation on the testing activities. I go through the global requirements and look for test cases per object part. These object parts I already acknowledged when I was doing the product risks analysis. The test cases shall not be the precise amount, but a estimation of them. These estimated test cases are hold against the risk class of the object part, the percentage of test cases I can reuse from former projects (test ware) and hands on experience of the testers. This all together leads to an estimation on the time spend on specifying the test cases.

To estimate the other phases I also have to look at other things. Like for the preparation phase I look at the quality of the documentation. If the quality of the documentation is OK, or the past experience is that the quality of the documentation is normally OK I add some time per type of documentation. This time is raised based on the quality of the documentation. Every document has its own time for preparation. And when evaluations are implemented I also calculate this (more time spend in reading and evaluating the documentation and less time spend in execution).

Within the execution also take into account the amount and experience of testers in the project. Experienced testers shall execute the test cases faster or can test the object parts that have a higher risk class. The estimated amount of defects is needed here. This percentage is needed to estimate the amount of rework that is done in the testing and development.

With all this together I now can estimate the amount of time spend on control and setting up and maintaining infrastructure. The estimation of control is dependent on the lead time of the testing, the amount of testers and the experience of the testers. The estimation for control and setting up and maintaining infrastructure is the most difficult to estimate. So for these two I use the earlier ratio of 17% of all testing activities.

The test completion phase varies per client. Some clients have a formal project completion policy some end the project after it is shipped successfully. That’s why I most of the time also use the ratio of 2-4% of all testing activities. When a formal project completion is done with conserving the test ware there should be more time spend on this phase.

This is how I estimate a project. Part of this is because in the Netherlands most testing is done formally and by checking the scripts. Checking is done most of the time on a ‘Friday afternoon’. In new projects I’ll use Pradeep’s estimation for exploring the system.

How do you estimate your testing project?

Hi friend J.,

Thank you for the opportunity to test your new application. It’s pleasure to see such a product from a totally new organization with a young entrepreneur like you. With your product you can support a lot of people that need your services, easily by asking it via internet. As you already know, we found some security defects, here are some examples.

ordering a negative amount of products gives a lot of credits in the next step you can sell them.

XSS is possible in the different fields, create popups and execute all possible scripts

An other one is insecure communication by using the “http” protocol in stead of “https“.

You can avoid these vulnerabilities in your application, but therefore you have to know that they exist and what the effects of these vulnerabilities are. In this letter I will give you a short advice from my point of view. How to deal with these requirements and how to give the right criteria to your supplier in India.

First of all be sure you have the right persons around you. Application development and especially security is not as easy as it looks! A lot of security features are things you can’t see. It’s not like functionality with a button on the left or on the right site. It’s all about how to validate, on the server or client side, about the difference between data and logic, about how to process and about how to store your confidential information.

To be a successful young entrepreneur with an IT component you must be sure the software you use works fine. In your situation is the application the most important part of your organization in combination with the experts. Testing the application at a professional manner is one of the things that is needed to get informed about the quality of the application. But if you want to test it, it’s needed to know the expectations.

Building a application without security is like building a new office without requirements of the lock at the doors and windows. If you meet a good builder that gives you a good advice he construct the building with the right locks. But if he doesn’t deliver them to you it`s your mistake.

Some hints to improve the application security of your product:

• Use the OWASP top 10 to be aware of the vulnerabilities and threats that can occur in your application. This top 10 describes the most important threats that can cause danger in the application. Be aware that 990 other vulnerabilities can follow. These then are the most important.
• Give the supplier the assignment that the application must pass all the checks/requirements as written in the ASVS level 1 and 2 (both static and dynamic test). If it fulfills these requirements your application is safe enough for this type of business.
• Analyze the process of your business model (with for example the sub-contractors) and see what weaknesses there are in the process. The application aspect is important but if the people aspect is bad, for example if people are giving there passwords to other persons, your total security is weak.
• If you have a new project or new version of the application send it to an expert so he can test it. Crowdtesting is a good solution for these problems. With crowdtesting experts can test it in an early phase of the process.

And dear J., having the correct requirements is the biggest challenge. If we, as testers, have to test an application it’s not possible without any background information and documentation. That’s why we send you the first 10 defects. But to give the right orders to your supplier you have to define requirements about performance, security and functionality (and more).

Requirements are the foundation of your application. Compare it with the vision and goals you define for your new organization. It all starts with a brilliant idea, a dream, a wish. From this little starting point you have defined the organization, the product.

J., this is my advice from my point of view, it’s up to you to use it or not, but if you have any question for now or in the future please let me know. Let us drink a good glass of South-African wine, in the near future so you can teach me how to be a young entrepreneur starting a new company.

Let me finish this letter with telling you know that you’re not the only one with these defects in the application. Many big insurance companies, banking and other types of organizations have these problems. But to be successful you have to solve these things. Otherwise people earn the money with your application instead of you as the owner.

Kind regards,

Andréas Prins.

In 2008 I was attending a session for the Sogeti Test in Gouvieux. France. There I learned about standard testware. Standard testware should help companies to upgrade there outsourcing abilities for their clients. But the question was how this could be accomplished. My answer to this is by ‘tagging’.

According to Wikipedia a tag is “In online computer systems terminology, a tag is a non-hierarchical keyword or term assigned to a piece of information (such as an internet bookmark, digital image, or computer file). This kind of metadata helps describe an item and allows it to be found again by browsing or searching. Tags are generally chosen informally and personally by the item’s creator or by its viewer, depending on the system.” In this blog we use tags as you can see in the right column.

With tags you add metadata to a piece of information. This information can be a test case, test scripts, test charters or whatever you my need to test an application, the testware. A lot of times this testware has already been made by someone else. This can then be reused for the same type of test. Sure you need to customise the test cases, but the most of it is reusable.

The problem with this testware has always been a standard type of notation for it. With a standard type of notation it’s easy to understand and can it be searched better. But standardising the documentation can be a time consuming task. And at the end of a project there is no time! Let alone to change all the test cases into a different standard.

I would not recommend this for a company to be the policy for testware. But you still have this information available. How can I ‘see’ it? By using tags you can make the information (testware) accessible for others. When you add testware to a, let’s say, testware repository in whatever notation you want, you could add tags about the test cases use, client, product, project, test level, type of business, quality attribute, etc. Like when I add test cases from a governmental client I could use the tag ‘government’, a tag for the project, a tag for the test level, maybe ‘FAT’, tags for object parts and/or test design techniques.

How do you implement this? It can be done easy ‘on the fly’ and more procedural. I would advise the last, but the simplest is the one I’ll explain here.

Let’s say you have a couple of test scripts for testing an application. A test script per object part, different scripts per test level and separate regression tests. These scripts can all be created in Excel (see tmap.net for an Excel template for a test script). First you add the test scripts to a directory where it’s agreed to manage the testware, like this.

All the test scripts are added to the directory (I created this on a Dutch version of Windows XP, but you see what I do).

Next step is to ‘tag’ the scripts. By viewing the ‘properties’ of the file you can add this information on the ‘Summary’ tab.

Under ‘Categories’ you can add the different tags with a semicolon (;) between the different tags.

Like above. I entered the tags ‘FAT’, ‘final’, date of creation in ‘May 2009’, test object and the quality attribute (‘functionality’). When this information is entered it’s easy to do a search on the different tags. And it’s even easier when a tag cloud is used to search through the different scripts.

When a tester starts a new project or client he can search through the testware database to find any scripts that can help in understanding the system and create test scripts faster without the reinvention of them…

I recommend that a company saves all its testware in one place (using a testware management process) and adding tags on them to help in future test projects.