An user acceptance test doesn’t exist!
Nov20
Is the user acceptance test we perform, a real UAT? This question comes in my mind thinking about the three aspects of software testing, checking, exploring and accepting, that are mentioned in this post. Is the user acceptance test a test where we really execute an user acceptance test? I think it isn’t, because most of the user acceptance tests I know are executed by test experts. In this post I’ll show you some thoughts and feelings around this test type and maybe we can conclude what is missing.
Before you go on with reading the post it is important to have a personal answer at some questions. Because we’re interested in your opinion about this topic.
- What is in your opinion (the definition of) a user acceptance test (UAT)?
- What is the goal of this test?
A definition can be something different compared to what happens in the real world.
What I see in practice (in the Netherlands) is that companies ask for test experts. These test experts come and do preparations for an user acceptance test. So far so good. What happens after this is that the test experts also execute these test cases (of the UAT). This is strange! Okay, the oracle the testers used are the requirements and therefore they made the assumption that these requirements represent what the users wants…
Two things are important in this case:
- Do the requirements really represent the user needs?
- Is a test expert really able to accept the software?
Definitions of the UAT
Before we give a answer at these questions, that, in my opinion, represent a big part of the problem, let’s check some methods about the definition of an UAT.
TMap NEXT® has the following definition of the users acceptance test.
“A test carried out by the future user(s) in an optimally simulated production environment, with the aim of demonstrating that the developed system meets the requirements of the users.”
In the TMap NEXT definition the future user plays an important role, I think this is really important to take into account. Because only the user that can accept software.
The second part is the aim of an UAT. This aim is: demonstrating of it meets the user requirements. Is demonstrating equal to accepting the software? If you give a demonstration of a particular tool and you’re a ‘good’ consultant you can mislead the audience. Because in your demonstration you can show a very small part, the thing the users wants to see, and skip all other stuff.
What I miss in the demonstrating part is, you can do this without a test executed by the users. A consultant can demonstrate it to show them that all requirements are met (this can happen if companies buy “commercial-off-the-shelf” software).
A third remark I have by this definition is: some test expert make the assumption that the requirements that are in the documentation are really the user requirements. Mostly these requirements are incomplete and do not represent the real needs of the user.
ISTQB has the following definition of acceptance testing (they don’t have a specific one for user acceptance).
“Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. [After IEEE 610]”
Remarkable isn’t it that the glossary of ISTQB doesn’t make a difference in acceptance testing. Because in this case all acceptance tests are equal to each other.
Is an UAT really a formal test? Can this be done with formal test design techniques? Do you need them to accept the software? What if a very special situation, that is essential for the process, can’t be caught in a particular test case? You can`t test this in an acceptance test. So you can`t accept the software.
“With respect to” is really the most ambiguous word in this definition. Is it really needed to check all the oracles that are mentioned in the definition? The user needs are really important, but how can I do a formal test if these things aren`t written down in designs?
What is wrong with these definitions? If you read only these definitions without any good explanation and examples there is a chance you`ll misinterpret the meaning of them.
Therefore some additions:
- Only the user/business can do the preparation and execution of an UAT
- Requirements that are described in documentation are not always the right requirements
- Requirements does not always represent what the user required
- The aim of an UAT is accepting , not demonstrating, accepting really means in my opinion: After the test people are really willing to use the software, because it has added value for them or for the overall process.
- There is a difference in for example a user acceptance and a functional acceptance test.
- An UAT is a more informal test often only the time box for preparation and execution is formal.
- During testing we have to respect each other always with or without an UAT;)
So I guess I have to change my mind about the user acceptance test. What I can do as test expert is: facilitate the process, do some planning activities, some defect management and more of these supporting activities.
Respect to all!
September 4th, 2010 at 00:13