The Basic Principles of Testing

The previous sections addressed software testing, whereas the following section summarize the basics of testing in general. These are guidelines that have developed over decades of testing experience.

1. Principle #1:
   Testing shows the presence of defects, not their absenceTesting establishes the presence of defects and reveals the faults that cause them. Depending on the effort made and the thoroughness of the tests involved, testing reduces the probability of leaving undiscovered faults in the test object. However, testing does not enable us to prove that a test object contains no faults. Even if an object doesn’t fail during testing, this is no proof of freedom from faults or overall correctness.
2. Principle #2:
   Exhaustive testing is impossibleWith the exception of extremely simple or trivial test objects, it is impossible to design and perform a complete test suite that covers all possible combinations of input data and their preconditions. Tests are always samples, and the effort allocated to them depends on the risks they cover and the priority assigned to them.
3. Principle #3:
   Early testing saves time and moneyDynamic and static testing activities should be defined and begun as early as possible in the system’s lifecycle. The term “shift left” implies early testing. Early testing reveals faults at an early stage of the development process. In a software context, this helps to avoid (or at least reduce) the increasingly costly repair of faults later in the development lifecycle.
4. Principle #4:
   Defects cluster togetherGenerally speaking, defects are not evenly distributed throughout a system. Most defects can usually be found in a small number of modules, and this (estimated or observed) clustering effect can be used to help analyze risk. Testing effort can then be concentrated on the most relevant parts of the system (see also principle #2 above).
5. Principle #5:
   Beware the pesticide paradoxOver time, tests become less effective the same way insects develop resistance to pesticides. If you repeat tests on an unchanged system, they won’t reveal any new failures. In order for your tests to remain effective you need to check your test cases regularly and, if necessary, modify them or add new ones. This ensures that you test previously unchecked components and previously untested input data, thus revealing any failures that these produces. The pesticide paradox can have a positive effect too. For example, if an automated regression test reveals a low number of failures, this may not be the result of high software quality but rather due to the ineffectiveness of the (possibly outdated) test cases in use.
6. Principle #6:
   Testing is context-dependentTests need to be adapted to the proposed purpose and the surrounding environment of the system in question. No two systems can be effectively tested the same way. Testing thoroughness, exit criteria, and other parameters need to be defined uniquely according to the system’s working environment. An embedded system requires different tests than, for example, an e-commerce system. Testing in an agile project will be very different from that in a project based on a sequential life-cycle model.
7. Principle #7:
   Absence-of-errors is a fallacyEven if you test all requirements comprehensively and correct all the faults you find, it is still possible to develop a system that is difficult to use, that doesn’t fulfill the user’s expectations, or that is simply of poor quality compared with other, similar systems (or earlier versions of the same system). Prototyping and early involvement of a system’s users are preventive measures used to avoid this problem.