Tag Archives: structured testing

Apple: Use a test technique/A test of Time!

Posted on by
1

This post was originaly posted by me on blog.sogeti.com

At the beginning of 2011 the hottest technology topic in the news was the alarm clock bug in Apple’s mobile appliances – iPad, iPhone and iPod. For January 1 and January 2 in this New Year, the Alarm Clock application wasn’t functioning. The same problem happened a few months ago when Daylight Saving Time was set in  October 2010.  The issue is with the ‘repeating alarms’ – a software bug in Apple’s software, iOS.

Courtesy of http://www.flickr.com/photos/helionorio

Changing the time

I would like to help the test manager at Apple with the necessary testing  he/she needs to get executed to find this fault in their other software. Because the bug appears with specific time changes; Daylight Saving, New Year, etc; . it looks like an error in specific Equivalence Classes.

OK, but what is an Equivalence Class? According to TMap NEXT®, it’s “the entire value range of a parameter partitioned into classes, in which the system behaviour is similar (or equivalent)”. Other terms used to refer to the design of test cases based on equivalence classes are ‘equivalence partitioning’ and ‘domain testing’.

Why should I use equivalence classes?

Well, as it happens at these specific time changes, you can set these time changes in a value range of a parameter, a class. Within this class, the behaviour is similar; it generates a bug so the alarm doesn’t go off.

How should they test this?

Apple should be using a test technique called the Data Combination Test (DCoT),  a versatile technique for testing of functionality both at the detail level and at an overall system level. In the embedded world, this technique is also known as the “Classification Tree Method”, was developed by Grochtmann and Grimm.

The fact that domain expertise is usable as a test basis also makes this technique suitable for situations in which specifications are incomplete or out of date, or even unavailable altogether. Time changes!

A testers view: Revision control

Posted on by
3

A hot item in the development life cycle is software versioning and/or revision control. According to Wikipedia software versioning is “the process of assigning either unique version names or unique version numbers to unique states of computer software”. And revision control is “the management of changes to documents, programs, and other information stored as computer files. It is most commonly used in software development, where a team of people may change the same files”. With software versioning and revision control tracks and provides control over changes in the documentation and source code. How do testers do this and what is their take on it?

Revision control is not only done in ICT. It is also done in engineering, business and law. Examples like “contract redline” and “legal blackline” are some of the earliest forms of revision control, and are still employed with varying degrees of sophistication.

Defects hide in different versions
As teams design, develop, test and deploy software, it is extremely common for multiple versions of the same software to be deployed in different environments. This is also true for developers to be working simultaneously on updates of the same software. Bugs and other issues (defects) with software are often present in certain versions. Therefore, for the purposes of locating and fixing defects, it is vitally important to be able to retrieve and run different versions of the software to determine in which version(s) the defect occurs.

At the simplest level, designers and developers could simply retain multiple copies of the different versions of the document or program, and label them appropriately. This simple approach has been used on many large software projects. This requires a lot of self-discipline on the part of author of the document, and unfortunately often leads to mistakes. Consequently, applications that automate some or all of the revision control process have been developed.

‘Practice what you preach’
Revision control also has multiple effects on testing. As said before defects can be ‘hidden’ between versions and testers need a stable test base for creating there test cases. Therefore testers are highly aware of revision control of their test base and source code. As a test manager I’m always indulged in the revision control of a project and when there is none I explain the risks it implies. But do we testers also take this so highly when it comes to our own revision control of our test cases? Most test plans we create we see as a document so we follow the control of this, but with test cases we should ‘practice what we preach’!

So why is it so important for testers to be involved in the revision control? TMap® NEXT tells us it is because testers need to know:

  • Which version of the test scripts is required. As a tester it is important to know which version of the various components is installed. In practice this is most important on the version of the software, usually the test object. Depending on the version, certain requirements or not built thus resulting that some scripts could not be needed;
  • Which release is shipped to the customer. The test takes place at a particular release version, and therefore the release advice should cover that release.

But testers should also get involved in their own revision control. How do I think this should be done? At first there is a difference in approach between development testing and system and acceptance testing.

Document a change in test cases
With system and acceptance testing test cases cover certain versions of the documentation (requirements or designs). In a perfect (testing) world the this test base should be frozen, but unfortunately this is more exemption rather than common. And of course there is also Agile. Test cases are based on a version of the documentation/requirements and these test cases can vary. This because test cases aren’t right the first time, documentation changes, but also progressive insight. Therefore it is needed to link test case versions with documentation/requirement versions.

With Agile, and thus iterations, the same applies. Within iterations there are pieces functionality (and non-functionals). The pieces of functionality will be yielded for testing in certain iterations. Thus revision control is needed on the iterations and the test cases. When certain functionality is passed to another iteration this should also be changed in the version of the test case.

Unfortunately many tools don’t support versioning standard. However, there are many add-ons to obtain the same result.

Source code and unit test case together
With development testing revision control is a little bit more complex. You could continue like with system and acceptance testing and link every version of source code with a version of a test cases. Unfortunately you are dependent on the approach of revision control of the source code, this usually differs from the revision control of documentation. Ideal is to create a direct link with the source code and the unit tests. When the source code is revised you directly change the test case. This is the simplest when it’s in the same version all together.

Some tooling is available for this.

Meta information in tags
The thing I use for my test projects is a lot like the post I wrote about test ware management. I use tagging on the test cases my team creates. We tag them with test case version, documentation version, requirement (version), date, author, quality attribute, system part, iteration, etc. As a result I have the advantage that all the meta information is clear and stored with the test cases. This hads always been done on system and acceptance testing.

Other options to include revision control in the testing process are also available. You could think of always using ‘track & trace’ in the documentation. This way the changes compared to the other version are always vissible to others. And make test revision control part of overall revision control from a QA point of view.

Tip: When a new version of the software or application is released a new version of the test documentation should also be released.

Tag! You’re it…

Posted on by
4

In 2008 I was attending a session for the Sogeti Test in Gouvieux. France. There I learned about standard testware. Standard testware should help companies to upgrade there outsourcing abilities for their clients. But the question was how this could be accomplished. My answer to this is by ‘tagging’.

According to Wikipedia a tag is “In online computer systems terminology, a tag is a non-hierarchical keyword or term assigned to a piece of information (such as an internet bookmark, digital image, or computer file). This kind of metadata helps describe an item and allows it to be found again by browsing or searching. Tags are generally chosen informally and personally by the item’s creator or by its viewer, depending on the system.” In this blog we use tags as you can see in the right column.

With tags you add metadata to a piece of information. This information can be a test case, test scripts, test charters or whatever you my need to test an application, the testware. A lot of times this testware has already been made by someone else. This can then be reused for the same type of test. Sure you need to customise the test cases, but the most of it is reusable.

The problem with this testware has always been a standard type of notation for it. With a standard type of notation it’s easy to understand and can it be searched better. But standardising the documentation can be a time consuming task. And at the end of a project there is no time! Let alone to change all the test cases into a different standard.

I would not recommend this for a company to be the policy for testware. But you still have this information available. How can I ‘see’ it? By using tags you can make the information (testware) accessible for others. When you add testware to a, let’s say, testware repository in whatever notation you want, you could add tags about the test cases use, client, product, project, test level, type of business, quality attribute, etc. Like when I add test cases from a governmental client I could use the tag ‘government’, a tag for the project, a tag for the test level, maybe ‘FAT’, tags for object parts and/or test design techniques.

How do you implement this? It can be done easy ‘on the fly’ and more procedural. I would advise the last, but the simplest is the one I’ll explain here.

Let’s say you have a couple of test scripts for testing an application. A test script per object part, different scripts per test level and separate regression tests. These scripts can all be created in Excel (see tmap.net for an Excel template for a test script). First you add the test scripts to a directory where it’s agreed to manage the testware, like this.

All the test scripts are added to the directory (I created this on a Dutch version of Windows XP, but you see what I do).

Next step is to ‘tag’ the scripts. By viewing the ‘properties’ of the file you can add this information on the ‘Summary’ tab.

Under ‘Categories’ you can add the different tags with a semicolon (;) between the different tags.

Like above. I entered the tags ‘FAT’, ‘final’, date of creation in ‘May 2009’, test object and the quality attribute (‘functionality’). When this information is entered it’s easy to do a search on the different tags. And it’s even easier when a tag cloud is used to search through the different scripts.

When a tester starts a new project or client he can search through the testware database to find any scripts that can help in understanding the system and create test scripts faster without the reinvention of them…

I recommend that a company saves all its testware in one place (using a testware management process) and adding tags on them to help in future test projects.