Certification of software can be useful
If you have a large network of applications connected to each other, exchanging critical information to each other and the software is of different vendors, certification of the software can be very useful. Take for example the electronic health record (EHR), all kinds of vendors are sending and receiving information to and from each other. This data is very critical in the meaning of life and dead of people.

For this type of software certification is useful. As a part of the assessment is the execution of a test set to validate of the software works according certain standards. To be sure that each tag in for example an SOAP message has the same meaning. But there is a little problem…

The pesticide paradox
If you’re using always the same test set to verify the software and you don’t change the test set you will not find any new bugs except for regression. This is according to the ISTQB source the pesticide paradox.

But what if you’re executing with for example the EHR an end-to-end test after you’ve certified the software? What if you find new bugs while you’re exchanging the information between systems of different vendors? What if you find issues when you’re live? Why haven’t you found them during the certification process?

If the test set of the certification process doesn’t change, you’re not able to find more issues in the software. Even if the software works according to certain standards this doesn’t mean the software works correct in each situation, it only works good enough according to the certification standard.

Bad certification if the test set doesn’t change
Does that mean that if you do not change the test set of the certification procedure of the software that the certification is bad? I don’t think so; it’s good enough according to the standards.
But failures in the software can lead to failures related to the dead of people why not improving the certification procedure?

If your software is certified with a certain version of the certification you know it works according to that certification. But this doesn’t mean it works fine in every situation.
If we give our software certificates must we change continuously our test set to improve the quality?

Continuous improvement of the set of qualifications
That’s why it’s important to change continuously the test set, make it bigger or better to cover more situations. If there occur bugs during the end-to-end test among different vendors, if there are new issues from production in for example the hospital, why not adding them to the test set?

If you improve the test set, change the certification, bugs from the past will not occur anymore in that situation because they are covered. And yes, existing software must be recertified maybe every half year.

Adding new tests to the test set cost time and money, but if you automate the test execution it’s not that much work. It shall give you more profit because the certification takes place before the end-to-end test with multi vendors and more important it takes place before you’re in production.

What I want to say is, learn from these important lessons learned. An extra test can save a life or a least a failure and money!