There are many discussions at conferences and on the web about the use of formal test design techniques. In my opinion the use of only formal test design techniques in a project is poison for the creativity of testers. If a test manager decides to use a certain amount of techniques he covers a certain amount of risks. The unexpected part of the application isn’t covered. And this part contains also some important risks. Especially because they aren’t described.

This post contains some reasons why the use of only formal test design techniques isn’t very useful. It also shows you what you’re missing, the tests you’re missing are based on creativity.

Most test design techniques are a step by step approach. Every time you use a certain technique you create the same type of test cases. Everyone that uses a technique will make the same kind of test cases. This is caused by the basic principles of the technique. Some of these techniques are described in this blog. The fact they are always the same for a specific technique is because you`re executing some steps that are always the same. This is why you cover always a similar part of the application. That’s why test design techniques only covers the risk we can imagine before we start testing. During testing a lot of new risks will appear because we learn from the behavior of the application. With only a prescribed approach we miss these new discovered risks.

The testers creativity disappears if he uses only the formal techniques. They describe what to do first and next and all the other steps. Just following the steps is enough to reach the agreed test coverage. To cover the predefined risks searching for exceptions isn`t needed anymore. For these repetitive you don`t need any creativity just a book and follow the steps.

Test design techniques are a step by step approach, manual testing isn’t needed in this case. Every time you do exactly the same to create the script and to execute a script. Because this is always the same you can automate these steps. The understanding and insight people can have from an application is far more than a stupid tool has. The creativity of people can be used during manual testing, the tools can do the repetitive work. An example is security testing, tools can do the amount of standard tests. For example checking all the possibilities for SQL-injection. Tools can do the work for you. But you need also some insight to hack for example the flows of the application, or to switch between procedures.

If there are five steps to authenticate yourself you can skip some of them. In the first one you fill the fields with your name and birth date for example. Instead of going to the second screen you go directly to the fifth screen (you have to know the URL), try do fill the fields in the last step and in press the OK button. I`ve tested a couple of applications where this occurred. A tool will never find this you need your brain and creativity.

Let me explain why we need creativity during testing. Two important fields of testing need creativity to find the bugs and flaws in the software. These two are security and usability, specific these two can’t be caught with standard test design techniques.

The use of only the standard techniques is like playing traditional chess, it`s fun but there is more if you see the 3D board. The third dimension is like the hidden features in the application and this contains the important risks.

In particular with security testing most of the bugs and flaws you’ll find are the exceptions in the software. With testing the functionality of an application you’re focused on the part that is described. With security testing you try to find all the special features you can use without any rights of with more rights than granted. These bugs and flaws are always different. And mostly are the risks from these bugs not known in the beginning of a project. How do you cover these without any creativity?

Usability testing is the other one that needs a lot of creativity, testing usability means that you need to know who your users are. You need to know what background they have and how they will use the application. I haven’t found a test technique that fits this. Usability testing is always focused on finding an optimal situation, and when you’ve reached this situation there`s always a better onein front of you. A predefined approach wouldn’t work in this case.

In my opinion formal test design techniques are needed to cover the known part of an application, if you complement this with a specific time for creativity you can cover (almost) the total application. Therefore you can use the Friday afternoon for the tests based on peoples creativity. In my opinion this isn`t enough for security for example because that needs a lot more time. But this Friday afternoon can be a good beginning of a change in your project. (Don`t read this boss:  my motivation and inspiration for executing the standard scripts isn`t high enough on Friday afternoon.) A creative test in which you can do whatever you want show`s you a lot defects.

Combine the formal and creative tests

Before you start a project you have to create a test strategy. This can be the standard test strategy with a plus. This plus is the creativity that covers a whole new part. Use a strategy that consists of two elementary parts:

• Formal test design techniques, to cover an import part of the risks
• Creativity and experience to find the exceptions

With this approach the design techniques are not a deadly poison for creativity but are they complementary to each other. In this case is the tester with a lot creativity the added value during software testing.

1. Thomas Says:
   October 9th, 2009 at 09:43

   Andréas,

   You should see Johan’s presentation on monday. I think you agree with him on a lot of points…

2. Michael Bolton Says:
   October 10th, 2009 at 04:51

   * Formal test design techniques, to cover an import part of the risks
   * Creativity and experience to find the exceptions
   

   Plus a third element: The skill and the initiative to alternate between the two productively.

   In rapid testing, we talk about this as dynamically managing your focus. Using quick tests, rather than careful tests (and vice versa). Using a diversity of techniques–function testing, domain testing, stress testing, flow testing, scenario testing, claims testing, user testing, risk testing, and automatic testing. Paying attention to what you’ve been told to address anticipated risks, vs. deliberately ignoring what you’ve been told in a search for new risks.

   Formal techniques only “kill” creativity when you decide not to use them creatively, when you only use one or two of them, or when perform formal approaches to the exclusion of other approaches.

   —Michael B.

3. Rik Marselis Says:
   October 12th, 2009 at 06:49

   The starting point of test design techniques has always been that there is not too much creativity from the tester. So that’s no problem for as far as I can see.
   However the mistake many test manager make is they only apply one technique and do that all the time.
   A good test strategy and/or a good master test plan uses a product risk analysis to distinguish risk classes and assigns various test design techniques based on risk. I always promote to use a formal test design tecnique COMBINED with Exploratory Testing. The formal technique helps you to reach a coverage that’s needed to assure the client that his software meets the required quality level, and the Exploratory Test takes care of the parts and features of the system that would not be covered by a formal test design technique.
   And always keep in mind that Functional testing, Usability Testing and Security Testing are differents aspects of testing and also all should be in the test strategy.

4. Justin Hunter Says:
   October 12th, 2009 at 21:39

   Ewald and Andreas,

   Good post again.

   Michael Bolton articulated my main reaction better than I could: “Formal techniques only “kill” creativity when you decide not to use them creatively, when you only use one or two of them, or when perform formal approaches to the exclusion of other approaches.”

   I think we all agree that a “check your brain at the door and just execute these scripts and confirm that the result you see when doing so matches the expected result you see printed” approach is bad (and unfortunately all to common in the testing community).

   I also agree that a “paint by numbers” approach to test design can lead to a set of sub-optimal tests. Your article does a good job of pointing this out.

   Even so, it is my contention that testers can go beyond a “paint by numbers” approach and add creativity into the formal test design approach itself. Slow, lumbering, incompetent testers will only think of obvious values to input into a test design tool; more creative, insightful, lateral-thinking, experienced and knowledgeable testers will be able to input more comprehensive values into a test design tool (and go beyond basic black box testing into gray box testing)… and in doing so, they would create a set of test cases that do a superb job of maximizing coverage in a minimal number of tests, even for things like security testing and User Interface testing.

   If we assume for the moment that testers will perform “brain-engaged” testing, I personally don’t see any reason why they wouldn’t be able to test things like Usability Testing concurrently with the execution of pre-scripted tests that have been put together using test design techniques.

   In short, my views are: (1) we can go much further with formal test design techniques than people traditionally do, (2) people should be more creative in thinking about the value inputs into test generating tools, (3) some ideas about input won’t come until testing is underway which is OK with a flexible approach, and (4) yes, these test design methods will need to be supplemented by other testing methods.

   - Justin

5. andreas Says:
   October 13th, 2009 at 06:12

   @michael thanks for the extra input, I totally agree that the skill to alternate them very important is. If you define your strategy without changing it during the test phase you miss the point.

   @Justin, I think it`s hard to train people in the aspects you mentioned. We can learn them the step by step approach and give them techniques to test more creative. But an important part is experience and training on the job. Maybe some testing communities should start an intensive coaching on the job principle to learn them how to use tools, creativity etc.

6. Vikas Says:
   June 15th, 2011 at 09:25

   Hey. I am veteran manual tester, I have 2 years of exp too,
   now i want to learn Qtp would you please suggest me any web page or study material to analyse Qtp, Load runner
   please reply me
   or send study material to vikaspagrotra@yahoo.com

   Thanks

7. Vikas Says:
   June 15th, 2011 at 09:25

   Hey. I am veteran manual tester, I have 2 years of exp too,
   now i want to learn Qtp would you please suggest me any web page or study material to analyse Qtp, Load runner
   please reply me
   or send study material to vikas.pagrotra@yahoo.com

   Thanks

8. Trackbacks Says:
   February 5th, 2012 at 23:39
   • test innovation and the future » Blog Archive » Find a challenge: Test
   • uberVU – social comments
   • creativity model