Tag Archives: test design technique

Apple: Use a test technique/A test of Time!

Posted on by
1

This post was originaly posted by me on blog.sogeti.com

At the beginning of 2011 the hottest technology topic in the news was the alarm clock bug in Apple’s mobile appliances – iPad, iPhone and iPod. For January 1 and January 2 in this New Year, the Alarm Clock application wasn’t functioning. The same problem happened a few months ago when Daylight Saving Time was set in  October 2010.  The issue is with the ‘repeating alarms’ – a software bug in Apple’s software, iOS.

Courtesy of http://www.flickr.com/photos/helionorio

Changing the time

I would like to help the test manager at Apple with the necessary testing  he/she needs to get executed to find this fault in their other software. Because the bug appears with specific time changes; Daylight Saving, New Year, etc; . it looks like an error in specific Equivalence Classes.

OK, but what is an Equivalence Class? According to TMap NEXT®, it’s “the entire value range of a parameter partitioned into classes, in which the system behaviour is similar (or equivalent)”. Other terms used to refer to the design of test cases based on equivalence classes are ‘equivalence partitioning’ and ‘domain testing’.

Why should I use equivalence classes?

Well, as it happens at these specific time changes, you can set these time changes in a value range of a parameter, a class. Within this class, the behaviour is similar; it generates a bug so the alarm doesn’t go off.

How should they test this?

Apple should be using a test technique called the Data Combination Test (DCoT),  a versatile technique for testing of functionality both at the detail level and at an overall system level. In the embedded world, this technique is also known as the “Classification Tree Method”, was developed by Grochtmann and Grimm.

The fact that domain expertise is usable as a test basis also makes this technique suitable for situations in which specifications are incomplete or out of date, or even unavailable altogether. Time changes!

Model-Based Testing, now!

Posted on by
10

The crisis is getting less, but the current economic situation hides the issues on test resourcing. There is a shortage of good testers in the labor market. Many companies hire the expertise from other companies, but they also have problems with delivering quality testers. The result: the development of applications is delayed or the risks at implementation are increased. Last year a hype going around in the world test. An amazing innovation: Model-Based Testing (MBT). Using MBT it’s possible to derive models into generated test cases. An application, the test cases, and optionally, there is the possibility to automate the test cases for execution. Model-Based Testing provides the opportunity to automate the process from test specification to test execution. Thereby creates an acceleration in the preparation of test cases. It is also possible to use the applications, for specifying and executing test cases, to generate information that provides insight into the testing process. As a result MBT makes the time consuming task of test specification less dependent on the amount of testers.

So this seems a panacea for the future test market. We specify all test cases with MBT! MBT off course is one of the things that makes testing applications easier and faster in the future. But how does MBT actually work and what can you do with it?

Operation
The operation of MBT is very complex, regarding the various tools that generate test cases, but the method of MBT is much simpler. Model-Based Testing is an approach based on designing test cases derived on a model of the test object. These test cases are then, where possible, automatically specified against the test object. The challenge with this approach is to include the creation of a formal model in which the operation of (part of) the application is represented. The creating of this model is done by hand. When this model is finished it can be read by a tool that enables the creation of test cases. An additional advantage is that MBT allows the creation of standard test cases. Something which is an added value for the ability to offshore or outsource test execution. MBT also claims is has the opportunity to check the quality of the requirements in an early stage in the development process.

Functional requirements, like derived from use cases, can serve as a starting for the modelling of test cases. The result is an UML (Unified Modeling Language) model, to let the model meet with UML the model  is interchangeable with multiple applications on the market available. These applications model with all the conventions of UML 2.0 to comply. This tooling can also help improve the designing of the models.

As a first step the test object will be designed in an object (UML) Class Diagram. The Class Diagram consists of the system and business attributes. This is a separate test model, not the model created in the Analysis Phase. The reason that if a model from the Analysis Phase is turned into a test model, this:

  1. A test model consists of other topics in the classes compared to a development model, for instance test data and the use of booleans;
  2. It would cost more time to adjust than to create a new development, and
  3. The same errors that are in the development model will be incorporated in the test model, allowing the test to find no defects in the Analysis Phase, since the same model is used.

Tip: When modelling appoint some functional ‘keywords’ serving test automation, which can be developed more quickly.

While modelling a UML workflow model is developed which, a State Diagram. This model shows the behaviour of the application, how it is operating with data within the application. This model explains the various decision points and paths in the test object. The Object Model describes any required (test) data for the various decision points and path. In OCL, Object Constraint Language, it is possible to process any business rules and requirements into the model, this adds the ability to trace back to the requirements.

Next these four models can be exported to a MBT tool. This MBT tool automatically generates multiple test cases, possibly including traceability to requirements (depends on the MBT tool and the use of OCL). The combination of the three different UML diagrams (Class Diagram, State Diagram and Object Model) and the OCL is needed for the creation of test cases. The application creates test cases by taking ‘paths’ through the system on which the test cases are based. Think about test paths. During the generation of test cases the application can automatically check the consistency of the test cases made. The application shows the test cases that are not. These test cases can’t be derived by the system and thereby gives a message. This error is the result of a modelling error in the models and should therefore be resolved.

Manual execution
Some MBT tools generate test cases into readable steps with the possibility to perform them manually. Once the keywords are implemented, the test cases can be executed. After the creation of the test cases MBT tools often have an option to export it to multiple formats, such as HP Quality Center or MS Word, like this example.

MBT_testcase

Learning curve
To generate the test cases the first step is to convert the requirements into UML models. This is the biggest challenge, new testers need to master in 3 UML diagrams, namely the Class Diagram, State Diagram and Object Model, and OCL programming language. The latter will entail the most challenges, this as most testers are naturally not developers. In the OCL the business rules are derived from the requirements, but also have the possibility for traceability to the requirements. It can also be chosen to hard code some information into the OCL which normally are put into a data chart.

Customisation
As indicated above, it is possible to adapt test cases  by adjusting the models. As a modification of the test basis is first manifested in the requirements it is possible to quickly change the models based on such a change . In the UML model the resulting changes are reflected into the test cases (when generating them form a MBT tool).

Coverage
Besides being able to specify test cases it is also important which test design techniques (TDT) a MBT tool can cover. This is something important when matching the test cases with the established testing strategy. By assigning the test design techniques: the test strategy is realized. The standard coverage of Model-Based Testing is the test design technique Process Cycle Test (PCT). This is because of the use of the State Diagram when modelling. In the PCT test intesity can be changed by increasing of lowering the test depth. It also shows that the tool not only covers the quality characteristic functionality, but also suitability.

The models can be adapted to cover further test design techniques. The Object Diagram and/or OCL lend themselves to cover the data required for various test design techniques. In the data the needed information for the necessary test situations on boundary value analysis and equivalence classes can be incorporated. This creates the possibility for the coverage of Data Combination Test (DCoT), Semantic (SEM) and Syntactic Test (SYN), Decision Table Test (DTT) and Elemantory Comparison Test (ECT).

MBT_qualityattr

However, if one or more of the TDT are to be chosen though should be given on how to model the UML to create the needed logical test cases. In the Data Combination Test, Semantic and Syntactic Test it can be suffice with a proper input of the various test data used for the test object.

Conclusion
By using models to test (multiple) releases a improvement of time could be made in regard to specifying test cases compared to manual specification of test cases. By ‘loose’ use of MBT an added value can be achieved due to the findings of errors in draft versions of the requirements. The problem lies in learning how to create the UML models and especially the OCL. This is an unexplored territory for most testers and could be a separate specialization within testing. If you succeed automatic test cases generations with automates test case execution it will be possible to achieve a further improvement.

Test Design Technique – Program Interface Test

Posted on by
4

The next episode in this ongoing story about a test design techniques. At my client we need to test the interfaces between programs. This post is about the test design technique to use, the Program Interface Test (PIT).

“The PIT is a test design technique that is mainly used in the unit integration test“

The program interface test (PIT) is a test design technique that is used to test the interfaces between the various programs and/or modules. After testing the independent function of each program, this technique is used to verify whether the programs still function correctly after integration with real data flows.

The PIT is a test design technique that is mainly used in the unit integration test.

When two programs in a application work with each other the integration of these two needs to be tested. When executing test cases created with PIT it is verified whether program 1 and 2 function correctly together. If there is a program 3 or more, these are replaced with stubs and drivers. The integration between program 1 and 2 is done with real data.

“The goal of the PIT is to test the interfaces between the various programs and/or modules“

The goal of the PIT is to test the interfaces between the various programs and/or modules. And thus check if the various programs interpret the data flows the same way. This is because at these points in the programs many defects occur, differences between the two programs. These types of misinterpretations and incorrect execution are dealt with during this test. The program interface test focuses especially on the interfaces and not on the correctness and completeness of processing.

“The PIT covers the quality attribute functionality“

As with the DCoT, the PIT covers the quality characteristic (ISO 9126 or TMap) functionality of the test object. But also connectivity between the two programs.

Like with UCT and DCoT there are 5 steps that need to be taken:

Steps UCT

“The PIT identifies the data flows in the integration“

The PIT determines that the test situations are identified by the data flows. There are two type of data flows; the direct data flows and the indirect data flows. A direct data flow is when a sender program transports the data directly to a receiving program without storing this data. An indirect data flow is when the sender program stores the data in a file and the receiving program collects the data from this file.

So first we need to identify the data flows (both direct and indirect) that are involved in the integration between the two programs. After this the attributes per data flows are identified. At last the domain of each attribute is determined and the validity rules for other attributes that are present are determined.

“Test situations with PIT are created by equivalence classes“

With the data attributes the equivalence classes is the basic technique to be used. Testing doesn’t have the option to include all possible attributes from the domains in the test cases. That is why the equivalence classes are used.

Determining the equivalence classes for each data flow is done by:

  • If an attribute is not mandatory, then the empty set is also an equivalence partition;
  • If the domain contains only a few values, then each value is an equivalence partition (single element);
  • If the domain of the attribute contains a large number of numeric values, take: smallest, greatest and 0 (if within the domain);

If the domain of the attribute contains a large number of alphanumeric values, take: minimum number of positions, maximum number of positions and the other values.

Data Flows

“The logical test cases creation is done in two steps“

Creating logical test cases consists of two steps. First choose a representative from each equivalence class and second combine the representative into test cases.

In step one the a representative from each equivalence partition is chosen. When an equivalence class consists of a single attribute value, there is no choice. When an equivalence class contains several attribute values a choice should be made. Preferably, choose an attribute value that borders on another equivalence partition to be the representative and not one represented in the validity rules. This because these validity rules should be verified in other tests

When representatives are chosen, the representatives should be combined into test cases.

“The logical test cases will almost automatically be the physical test cases“

With the logical test cases in place the physical test cases should be derives simple. These logical test cases will almost automatically also be the physical test cases. Only descriptions must be made with the test cases as actions and checks. These actions are the actions for the sender programs to the test cases. The checks of the receiving program are also added. These checks are the determination of the expected result of the interface in the receiving.

“Test script derives from the starting point and physical test case“

The starting point for the physical test cases now needs a little more information. The initial data sets needed for the test are described, this also comes in handy for reasons of control and reuse. It helps if this data is provided with a meaningful name to recognize it. After this the test script can be build.

testscriptPIT