Hi friend J.,

Thank you for the opportunity to test your new application. It’s pleasure to see such a product from a totally new organization with a young entrepreneur like you. With your product you can support a lot of people that need your services, easily by asking it via internet. As you already know, we found some security defects, here are some examples.

ordering a negative amount of products gives a lot of credits in the next step you can sell them.

XSS is possible in the different fields, create popups and execute all possible scripts

An other one is insecure communication by using the “http” protocol in stead of “https“.

You can avoid these vulnerabilities in your application, but therefore you have to know that they exist and what the effects of these vulnerabilities are. In this letter I will give you a short advice from my point of view. How to deal with these requirements and how to give the right criteria to your supplier in India.

First of all be sure you have the right persons around you. Application development and especially security is not as easy as it looks! A lot of security features are things you can’t see. It’s not like functionality with a button on the left or on the right site. It’s all about how to validate, on the server or client side, about the difference between data and logic, about how to process and about how to store your confidential information.

To be a successful young entrepreneur with an IT component you must be sure the software you use works fine. In your situation is the application the most important part of your organization in combination with the experts. Testing the application at a professional manner is one of the things that is needed to get informed about the quality of the application. But if you want to test it, it’s needed to know the expectations.

Building a application without security is like building a new office without requirements of the lock at the doors and windows. If you meet a good builder that gives you a good advice he construct the building with the right locks. But if he doesn’t deliver them to you it`s your mistake.

Some hints to improve the application security of your product:

• Use the OWASP top 10 to be aware of the vulnerabilities and threats that can occur in your application. This top 10 describes the most important threats that can cause danger in the application. Be aware that 990 other vulnerabilities can follow. These then are the most important.
• Give the supplier the assignment that the application must pass all the checks/requirements as written in the ASVS level 1 and 2 (both static and dynamic test). If it fulfills these requirements your application is safe enough for this type of business.
• Analyze the process of your business model (with for example the sub-contractors) and see what weaknesses there are in the process. The application aspect is important but if the people aspect is bad, for example if people are giving there passwords to other persons, your total security is weak.
• If you have a new project or new version of the application send it to an expert so he can test it. Crowdtesting is a good solution for these problems. With crowdtesting experts can test it in an early phase of the process.

And dear J., having the correct requirements is the biggest challenge. If we, as testers, have to test an application it’s not possible without any background information and documentation. That’s why we send you the first 10 defects. But to give the right orders to your supplier you have to define requirements about performance, security and functionality (and more).

Requirements are the foundation of your application. Compare it with the vision and goals you define for your new organization. It all starts with a brilliant idea, a dream, a wish. From this little starting point you have defined the organization, the product.

J., this is my advice from my point of view, it’s up to you to use it or not, but if you have any question for now or in the future please let me know. Let us drink a good glass of South-African wine, in the near future so you can teach me how to be a young entrepreneur starting a new company.

Let me finish this letter with telling you know that you’re not the only one with these defects in the application. Many big insurance companies, banking and other types of organizations have these problems. But to be successful you have to solve these things. Otherwise people earn the money with your application instead of you as the owner.

Kind regards,

Andréas Prins.

In 2008 I was attending a session for the Sogeti Test in Gouvieux. France. There I learned about standard testware. Standard testware should help companies to upgrade there outsourcing abilities for their clients. But the question was how this could be accomplished. My answer to this is by ‘tagging’.

According to Wikipedia a tag is “In online computer systems terminology, a tag is a non-hierarchical keyword or term assigned to a piece of information (such as an internet bookmark, digital image, or computer file). This kind of metadata helps describe an item and allows it to be found again by browsing or searching. Tags are generally chosen informally and personally by the item’s creator or by its viewer, depending on the system.” In this blog we use tags as you can see in the right column.

With tags you add metadata to a piece of information. This information can be a test case, test scripts, test charters or whatever you my need to test an application, the testware. A lot of times this testware has already been made by someone else. This can then be reused for the same type of test. Sure you need to customise the test cases, but the most of it is reusable.

The problem with this testware has always been a standard type of notation for it. With a standard type of notation it’s easy to understand and can it be searched better. But standardising the documentation can be a time consuming task. And at the end of a project there is no time! Let alone to change all the test cases into a different standard.

I would not recommend this for a company to be the policy for testware. But you still have this information available. How can I ‘see’ it? By using tags you can make the information (testware) accessible for others. When you add testware to a, let’s say, testware repository in whatever notation you want, you could add tags about the test cases use, client, product, project, test level, type of business, quality attribute, etc. Like when I add test cases from a governmental client I could use the tag ‘government’, a tag for the project, a tag for the test level, maybe ‘FAT’, tags for object parts and/or test design techniques.

How do you implement this? It can be done easy ‘on the fly’ and more procedural. I would advise the last, but the simplest is the one I’ll explain here.

Let’s say you have a couple of test scripts for testing an application. A test script per object part, different scripts per test level and separate regression tests. These scripts can all be created in Excel (see tmap.net for an Excel template for a test script). First you add the test scripts to a directory where it’s agreed to manage the testware, like this.

All the test scripts are added to the directory (I created this on a Dutch version of Windows XP, but you see what I do).

Next step is to ‘tag’ the scripts. By viewing the ‘properties’ of the file you can add this information on the ‘Summary’ tab.

Under ‘Categories’ you can add the different tags with a semicolon (;) between the different tags.

Like above. I entered the tags ‘FAT’, ‘final’, date of creation in ‘May 2009’, test object and the quality attribute (‘functionality’). When this information is entered it’s easy to do a search on the different tags. And it’s even easier when a tag cloud is used to search through the different scripts.

When a tester starts a new project or client he can search through the testware database to find any scripts that can help in understanding the system and create test scripts faster without the reinvention of them…

I recommend that a company saves all its testware in one place (using a testware management process) and adding tags on them to help in future test projects. 

Sometimes testers make some assumptions, but hopefully they avoid them. This series of posts will show you some well-known pitfalls and assumptions that tester make. This third post describes assumptions about testing during lifetime. (post 1, post 2)

The application react the same in production as in the test environment
Some people assume that if you finished testing in the test environment everything is ready to go into production. Some times this is possible if they use for example the hardware of the test environment as a production machine. This works fine with new projects. But what about releases? What if they move it to another machine?

Very often the application is copied or configured again on a different machine. This machine has other characteristics, like hardware and other configuration. As you can imagine, the application reacts different to this environment. It can happen that new defects are caused by the configuration.

To test this difference it’s useful to have a regression test set. A set with test cases you used earlier during testing. Most of the time it isn’t needed to test all the functionality again. But a representative test set can cover these risks. If there are parts of the application that needs more configurations you have to test them with more depth.

How many of you testers also plan a retest in production? Are you in the office, the weekend of the big bang, when they go live with the new application? Sometimes we underestimate the change of the environment at the software quality!

Once an application is tested it doesn’t mean it’s tested for his lifetime
Are you really finished with testing when the application is production? Regression testing is still needed during the lifetime of the application. The quality attribute Security is an example of aspect that needs a lifetime testing.

Each day new threats occur; most of them don’t have much impact at each application. But the bigger threats have more impact and retesting must be done. Think about, Cross-site Request Forgery (CSRF), Click Jacking and the latest one beginning this year Cross-site History Manipulation (XSHM). These vulnerabilities must be checked in the application. A regression test each half year for security reasons can be very useful. Also a check of for example, the web server is updated with the newest version the last half year.

A couple of months ago I tested an application that has an Apache web server latest update June 2004. The site http://www.milw0rm.com/search.php gives the vulnerabilities in this version (and a lot of other software).

A test that often works fine is
- Go to the highest level in the domain: “www.mysite.net/”
- Add “aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bbbbbbbbbbbbbbbbbbbbbbbbba.aaaaaaaaa
aaaaaaaat”,
- Send this URL to the server: “www.mysite.net/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.bbb
bbbbbbbbbbbbbbbbbbbbbba.aaaaaaaaaaaaaaaaat“.

Tip: Testing is needed throughout the total lifetime of the application!