Model-Based Testing, now!
Oct05
The crisis is getting less, but the current economic situation hides the issues on test resourcing. There is a shortage of good testers in the labor market. Many companies hire the expertise from other companies, but they also have problems with delivering quality testers. The result: the development of applications is delayed or the risks at implementation are increased. Last year a hype going around in the world test. An amazing innovation: Model-Based Testing (MBT). Using MBT it’s possible to derive models into generated test cases. An application, the test cases, and optionally, there is the possibility to automate the test cases for execution. Model-Based Testing provides the opportunity to automate the process from test specification to test execution. Thereby creates an acceleration in the preparation of test cases. It is also possible to use the applications, for specifying and executing test cases, to generate information that provides insight into the testing process. As a result MBT makes the time consuming task of test specification less dependent on the amount of testers.
So this seems a panacea for the future test market. We specify all test cases with MBT! MBT off course is one of the things that makes testing applications easier and faster in the future. But how does MBT actually work and what can you do with it?
Operation
The operation of MBT is very complex, regarding the various tools that generate test cases, but the method of MBT is much simpler. Model-Based Testing is an approach based on designing test cases derived on a model of the test object. These test cases are then, where possible, automatically specified against the test object. The challenge with this approach is to include the creation of a formal model in which the operation of (part of) the application is represented. The creating of this model is done by hand. When this model is finished it can be read by a tool that enables the creation of test cases. An additional advantage is that MBT allows the creation of standard test cases. Something which is an added value for the ability to offshore or outsource test execution. MBT also claims is has the opportunity to check the quality of the requirements in an early stage in the development process.
Functional requirements, like derived from use cases, can serve as a starting for the modelling of test cases. The result is an UML (Unified Modeling Language) model, to let the model meet with UML the model is interchangeable with multiple applications on the market available. These applications model with all the conventions of UML 2.0 to comply. This tooling can also help improve the designing of the models.
As a first step the test object will be designed in an object (UML) Class Diagram. The Class Diagram consists of the system and business attributes. This is a separate test model, not the model created in the Analysis Phase. The reason that if a model from the Analysis Phase is turned into a test model, this:
- A test model consists of other topics in the classes compared to a development model, for instance test data and the use of booleans;
- It would cost more time to adjust than to create a new development, and
- The same errors that are in the development model will be incorporated in the test model, allowing the test to find no defects in the Analysis Phase, since the same model is used.
Tip: When modelling appoint some functional ‘keywords’ serving test automation, which can be developed more quickly.
While modelling a UML workflow model is developed which, a State Diagram. This model shows the behaviour of the application, how it is operating with data within the application. This model explains the various decision points and paths in the test object. The Object Model describes any required (test) data for the various decision points and path. In OCL, Object Constraint Language, it is possible to process any business rules and requirements into the model, this adds the ability to trace back to the requirements.
Next these four models can be exported to a MBT tool. This MBT tool automatically generates multiple test cases, possibly including traceability to requirements (depends on the MBT tool and the use of OCL). The combination of the three different UML diagrams (Class Diagram, State Diagram and Object Model) and the OCL is needed for the creation of test cases. The application creates test cases by taking ‘paths’ through the system on which the test cases are based. Think about test paths. During the generation of test cases the application can automatically check the consistency of the test cases made. The application shows the test cases that are not. These test cases can’t be derived by the system and thereby gives a message. This error is the result of a modelling error in the models and should therefore be resolved.
Manual execution
Some MBT tools generate test cases into readable steps with the possibility to perform them manually. Once the keywords are implemented, the test cases can be executed. After the creation of the test cases MBT tools often have an option to export it to multiple formats, such as HP Quality Center or MS Word, like this example.
Learning curve
To generate the test cases the first step is to convert the requirements into UML models. This is the biggest challenge, new testers need to master in 3 UML diagrams, namely the Class Diagram, State Diagram and Object Model, and OCL programming language. The latter will entail the most challenges, this as most testers are naturally not developers. In the OCL the business rules are derived from the requirements, but also have the possibility for traceability to the requirements. It can also be chosen to hard code some information into the OCL which normally are put into a data chart.
Customisation
As indicated above, it is possible to adapt test cases by adjusting the models. As a modification of the test basis is first manifested in the requirements it is possible to quickly change the models based on such a change . In the UML model the resulting changes are reflected into the test cases (when generating them form a MBT tool).
Coverage
Besides being able to specify test cases it is also important which test design techniques (TDT) a MBT tool can cover. This is something important when matching the test cases with the established testing strategy. By assigning the test design techniques: the test strategy is realized. The standard coverage of Model-Based Testing is the test design technique Process Cycle Test (PCT). This is because of the use of the State Diagram when modelling. In the PCT test intesity can be changed by increasing of lowering the test depth. It also shows that the tool not only covers the quality characteristic functionality, but also suitability.
The models can be adapted to cover further test design techniques. The Object Diagram and/or OCL lend themselves to cover the data required for various test design techniques. In the data the needed information for the necessary test situations on boundary value analysis and equivalence classes can be incorporated. This creates the possibility for the coverage of Data Combination Test (DCoT), Semantic (SEM) and Syntactic Test (SYN), Decision Table Test (DTT) and Elemantory Comparison Test (ECT).
However, if one or more of the TDT are to be chosen though should be given on how to model the UML to create the needed logical test cases. In the Data Combination Test, Semantic and Syntactic Test it can be suffice with a proper input of the various test data used for the test object.
Conclusion
By using models to test (multiple) releases a improvement of time could be made in regard to specifying test cases compared to manual specification of test cases. By ‘loose’ use of MBT an added value can be achieved due to the findings of errors in draft versions of the requirements. The problem lies in learning how to create the UML models and especially the OCL. This is an unexplored territory for most testers and could be a separate specialization within testing. If you succeed automatic test cases generations with automates test case execution it will be possible to achieve a further improvement.
October 5th, 2009 at 10:19
Is this some concrete tooling that you describe? Because model-based testing is neither necessarily based on UML nor does it necessarily require the combination of class diagrams, etc. as you specify…
October 5th, 2009 at 11:48
Ben,
I know there are options that don’t use models based on UML, but I think the most used are UML based. And these are the ones I tried to derive test cases. What tools do you think about without UML?
Regards,
Ewald
October 5th, 2009 at 15:47
Hi there,
I used in 2007 Visio 2003, and worked on some of the diagrams you write about in your blog, meaning class and state diagrams. I did this only for documentation. I didn’t come up with diagrams because of, for example, unit testing.
I was wondering if you could give some examples of ocl queries, regardless of whether these queries are generated by a tool or utility, or crafted by a programmer.
Best regards,
Tonci.
October 6th, 2009 at 05:30
Tonci,
Fot those things I would like to recommend the blog of a collegue of ours: http://robkuijt.nl/. Rob has more of these things you asked for lined out and I think he can help you further.
Regards,
Ewald
October 6th, 2009 at 07:08
In fact, i would argue that most model-based testing tools are based on lower level models like FSMs, mealy machines, labeled transition systems, input/output transition systems or SDL and the methods have a history of up to 30 years. Well known tools for these techniques are Torx, TGV, or Autolink, but there are a lot of more. It’s always been one specific research branch of the formal methods community. I haven’t tested the tool, but i supposed you could also count the Microsoft Spec Explorer into this category of tools.
From your article i’m not even sure whether you talk about model-based testing or model-driven test development which are two entirely different methods. For model-based testing, you model the system and derive test cases from the system model (chinese postman transition tours etc.) whereas in model-driven test development, you model the test cases and you merely have a slightly different level of abstraction for the test specification and mostly specify graphically. However, you don’t derive entire test suites, but it is mostly a 1:1 relationship, for example, one sequence diagram corresponds to one test case, i.e., one concrete topolical sorting despite its partial order nature as everything else would mostly represent just a different scheduling as opposed to a test case that actually tests something different functionally or structurally.
Also, i still wonder what kind of tooling you use with what you describe in the text, because you obviously need some kind of restricted UML profile for UML to make sense for code generation. The semantics of non-restricted UML is much too open for code generation otherwise. I’m asking solely out of interest, because i simply don’t know any good MBT tooling for UML and you obviously have something specific in mind when you say that what you describe uses only class diagrams, state diagrams, and object diagrams. Other tools might make use of sequence diagrams…
October 6th, 2009 at 10:15
Ben,
I agree that a lot of MBT tools don’t use UML. Look for instance at Cover at http://robkuijt.nl/. This one is based on pseudo code.
I am writing about Model-Based Testing. The original piece of text I wrote when checking out Smartesting Testdesigner at http://www.smartesting.com/index.php/cms/en/explore/products. This tool is the most hyped in the Netherlands about it. And most people I know work with this tool or the tool by Rob Kuijt.
I will look into your other comments shortly and will reply here (maybe even a new post)
Regards,
Ewald
October 7th, 2009 at 16:01
Hi Ewald,
Just wondering how you see the future of Model-Based Testing for software build using Model-Driven Development.
A while ago I have written an article on quality in Model-Driven Engineering – http://www.theenterprisearchitect.eu/archive/2008/06/07/quality-in-model-driven-soba-development – but I am not a testing expert.
October 16th, 2009 at 11:26
@Johan,
Ik have to get back on that one. It’s not so simple and you have a good, but long post.
Ewald
February 24th, 2010 at 10:40
@Johan.
Ik took me a while to look into this. But I had the answer already. When using MDE to create applications the model is the most critical part. This model has to be reviewed, like this: http://www.testingthefuture.net/2009/02/evaluating-models/.
I hope this helps you. Mendix is already doing a great job.
-Ewald