Defect and Fault Terminology

The test basis as a starting point for testing

A situation can only be classed as faulty if you define in advance what exactly is supposed to happen in that situation. In order to make such a definition, you need to know the requirements made of the (sub)system you are testing as well as other additional information. In this context, we talk about the test basis against which tests are performed and that serves as the basis for deciding whether a specific function is faulty.

What counts as a defect?

A defect is therefore defined as a failure to fulfill a predefined requirement, or a discrepancy between the actual behavior (at run time or while testing) and the expected behavior (as defined in the specifications, the requirements, or the user stories). In other words, when does the system’s behavior fail to conform to its actual requirements?

Unlike physical systems, software systems don’t fail due to age or wear. Every defect that occurs is present from the moment the software is coded, but only becomes apparent when the system is running.

Faults cause failures

System failures result from faults and only become apparent to the tester or the user during testing or at run-time. For example, when the system produces erroneous output or crashes.

We need to distinguish between the effects of a fault and its causes. A system failure is caused by a fault in the software, and the resulting condition is considered to be a defect. The word “bug” is also used to describe defects that result from coding errors, such as an incorrectly programmed or forgotten instruction in the code.

Defect masking

It is possible that a fault can be offset by one or more other faults in other parts of the program. Under these circumstances, the fault in question only becomes apparent when the others have been remedied. In other words, correcting a fault in one place can lead to unexpected side effects in others.

Not all faults cause system failures, and some failures occur never, once, or constantly for all users. Some failures occur a long way from where they are caused.

A fault is always the result of an error or a mistake made by a person—for example, due to a programming error at the development stage.

People make errors

Errors occur for various reasons. Some typical (root) causes are:

• All humans make errors!
• Time pressure is often present in software projects and is a regular source of errors.
• The complexity of the task at hand, the system architecture, the system design, or the source code.
• Misunderstandings between participants in the project—often in the form of differing interpretations of the requirements or other documents.
• Misunderstandings relating to system interaction via internal and external interfaces. Large systems often have a huge number of interfaces.
• The complexity of the technology in use, or of new technologies previously unknown to project participants that are introduced during the project.
• Project participants are not sufficiently experienced or do not have appropriate training.

A human error causes a fault in part of the code, which then causes some kind of visible system failure that, ideally, is revealed during testing (see figure 2-1: Debugging, see below). Static tests can directly detect faults in the source code.

System failures can also be caused by environmental issues such as radiation and magnetism, or by physical pollution that causes hardware and firmware failures. We will not be addressing these types of failures here.

Fig. 2-1The relationships between, errors, faults, and failures

False positive and false negative results

Not every unexpected test result equates to a failure. Often, a test will indicate a failure even though the underlying fault (or its cause) isn’t part of the test object. Such a result is known as a “false positive”. The opposite effect can also occur—i.e., a failure doesn’t occur even though testing should reveal its presence. This type of result is known as a “false negative”. You have to bear both of these situations in mind when evaluating your test results. Your result can also be a “correct positive” (failure revealed by testing) or a “correct negative” (expected behavior confirmed by testing). For more detail on these situations, see section 6.4.1.

Learning from your mistakes

If faults and the errors or mistakes that cause them are revealed by testing it is worth taking a closer look at the causes in order to learn how to avoid making the same (or similar) errors or mistakes in future. The knowledge you gain this way can help you optimize your processes and reduce or prevent the occurrence of additional faults.