The cloud model of testing

Cloud computing is an emerging phenomenon that offers enormous advantages, such as shorter time to market, flexible computing capabilities and limitless power, but the cloud market, still in a very early stage, continues to grow and evolve.

But cloud computing is much more than technology, cutting costs or getting more agile. The cloud model is a new business model, a new way of thinking and doing IT as a business instead of a technology! The cloud business model. More modular, incremental, selective, collaborative and with a value proposition based on financial liquidity and operational flexibility [Forrester, 2010]. This has a lot of (side) effects, also on software testing…

New model for testing

The way we do business around testing will change also. Not the way we test, but how we deliver it. The market will be more focused on value driven, agile, modular and flexible solutions from test service providers. These providers need to create more ‘building blocks’ around how they deliver services: testing. Testing will also become a service, Software Testing as a Service (has a nice buzz ring to it).

Software Testing as a Service

Software Testing as a Service (STaaS) was coined by my colleague Leo van der Aalst in 2008, but his ideas didn’t go as far as I would think of today. In Leo’s proposal all testing was done by a service provider on demand at the scale as needed by the client. My idea goes further.

Software testing needs to be even more flexible, from the test manager to the test engineer, from the system test environment to the performance test environment, from the security scan to the full usability test and from audits to end-2-end tests. And everything can be done separate and fully integrated with one other. This creates a full cloud model for testing. With the possibility to be fully standardized, agile and elastic to help the client when needed; a full service consisting of several building blocks.

Execute testable specifications: Simulation Testing

Software development should become cheaper! And with this, testing should become cheaper! That’s a line I’ve been hearing a lot the last few years, but even more now. But How can this be achieved?

I’ve been blogging quite a few posts (like here, here, here and here) about finding defects in an earlier stage in the development process. One of the most important activities that can be done to find these defects, is to integrate evaluations (like testing) in the quality strategy of software development projects. But as all systems are becoming more and more complex, this emphasis on finding defects early is also growing. And as Boehm told us: the earlier errors are found, the cheaper it is to fix them. But are there other options than evaluations to do this?

Yes, there are. Advancing technologies make this possible, even now. How? By using models! Models can be used to generate test cases automatically (MBT), evaluate the software and can help with understanding the functionality of the software. But it’s also possible to use it for other practices. In embedded software it’s a common use to use models for testing the software in an early stage of the project. This way errors are solved early before the software is implemented in the appliance, like a VCR, refrigerator, car or even an airplane. But how can we do this with non-embedded software?

With Model-Based Design (MBD)! The next figure is an example of an embedded system. But it’s also to create a figure like this for a non-embedded system. With using Object Oriented (OO) languages a model can be create, which can be executed.

Within the embedded world, various hardware components are processed in the design process. OO can also do this, by using different functions as the components. These (standard) functions can be incorporated into a model and connected with each other. As a result a model of a component arises which then can be simulated. And the functional correctness of the component can be determined.

Using models gives us (testers and developers) insight into the dynamics of the development process and algorithmic aspects of a system by simulation them. Therefore models can be used:

  • as executable and testable specifications;
  • to communicate (system) requirements and interface definitions;
  • as a model of the entire system when provided with virtual prototypes;
  • automatic code generation for software development (algorithm or logic), and
  • automatic test case generation for testing the software.

Execute the functional design

As a tester I find the first option particularly interesting; ‘as executable and testable specifications’. With embedded software this is maybe easier to understand. There models are used to simulate the hardware or real world, in order to test the software of the embedded system. But with OO ‘normal’ software applications, such as administrative systems, can also be executed as a testable specification!

When using models it is now possible to simulate the (functional) design and visualise and debug the specification. In this simulation it is possible to integrate both the initial input of the test as well as the expected result, thus creating a functional test. Within the embedded world this is called model-in-the-loop testing (MIL). A term we can use also in non-embedded testing.

Development testing
A full exploration of a (functional) design based on testing scenarios by using simulations is a way to test requirements. Within these testing scenarios diverse subjects can be included, such as input parameters and environmental factors. And therefore covering all the possible input and parameters of a component or unit and this , subsequently, leads to an increased coverage of the development tests.

Best results can be achieved when the testing of the models (this is called simulation testing or virtual testing) is done parallel with the design process. When the models are created, the tests are focused on the design aspects and are simulated in the model. Since the design is still in a draft phase, you are not only able to adjust the (functional) design, but also the tests. And, as a result, a design model (at unit level) is created that enjoys the confidence of analysts, developers and testers.

Integration of the components

With MBD and simulation testing it is possible to continually test and validate the design. The next step is to integrate the validated models with other validated models and check the integration of the different components. The previously developed testing scenarios at an unit level can also be integrated with each other to use as a comprehensive integration test.

This can be done the same way as an end-2-end test of a built system. But when integrating various subsystems (models), it is important to have a good overview of the various requirements of each individual component. Or else the complexity will only get harder.

Reuse test cases

In the end a fully tested system can be built with good confidence. All system components and subsystems can be built, (re)tested and integrated in larger systems.

And with tooling it’s possible that the tests of the simulation can be reused during the  testing of the actual code and software. The use of identical test cases allows a comparison between the model and reality simpler (eg. Test Driven Development).

Simulation Testing

By using Simulation Testing (modelling and executing these models), it is possible to visualise and debug designs without any (newly) written code. Design errors can be removed quickly from the documentation. Moreover, a large part of the (development) testing can be performed based on simulations. By using simulations for checking the development process, models offer a different technique to find defects in the design at the earliest possible moment.

Augmented Testing, a touch of genius?

I just read this quote by E.F. Schumacher: “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage — to move in the opposite direction.” Let’s have a touch of genius because our future systems are becoming more and more complex. Those systems need to be integrated with other systems. This in turn will lead to an even more increased complexity. This increased complexity also has its effect on testing. How can we handle this increased complexity?

We will need more information about the system(s) we’re testing, the client’s real wishes about this system and the business processes that control the system. With this information we can estimate what is needed for testing and make the correct adjustments to our testing process. But what information is needed?

Most of the information we need, is already available, but it needs to be interpreted correctly. For example information about the connections, business processes, users, testers, developers, used techniques and earlier testing. But we need to streamline this information. And when we have this information we can use this in a real-time view of the system with perhaps all its defects and missing items; Augmented Testing.

With Augmented Testing it’s an option to use this information as a layer over the System under Test (SuT) so all the information about the system under test while testing is available in real-time. By creating a structure that interlinks all this known information with each other and even added information this information can be made available. When this information is readily and in real-time available it can be ‘projected’ over the SuT. 

augmentedtesting

 Click image to enlarge.

This Augmented Testing can be the touch of genius we testers need!