Evaluating Test Reports

The result of every systematically performed test run is a test (execution) log²³.

This documents (per test case and/or testing step) the actual behavior observed during the test, the expected behavior, and any deviations between the two²⁴.

Analyzing test reports

When analyzing a defect report²⁵, you have to decide which of the deviations between expected and actual behavior constitutes a defect. This is a yes/no classification, and you need to watch out for the following cases and potential misclassifications²⁶:

• True positive
  The test object is faulty and the test case correctly reports a deviation between the expected and actual behaviors. The test result is classed as “failed”²⁷, or “red”²⁸.
• False negative
  The test object is faulty but none of the test cases reveals this. This may be because the corresponding test data combination was left out of the test case design or wasn’t executed during the test run. It may also be because the corresponding test case doesn’t exist, or the fault in question cannot be revealed on the current test level or using the current testing technique.
• False positive
  Analysis reveals that the test object behaves correctly but the test case still flags a deviation between the expected and actual behaviors. This may be due to an outdated or badly designed test case, an incorrectly configured test object, badly implemented test automation, or incorrect test execution on the part of the tester.
• True negative
  The test object behaves correctly with regard to the corresponding test case and the test case confirms that expected = actual behavior. The test case is classed as “passed” or “green”.

Figure 6-3 illustrates these four cases:

Fig. 6-3True and false test results

If the issue is due to the test object²⁹ (i.e., a “true positive”) a corresponding defect report is created. If the defect has already been reported, this may be a duplicate. You then have to check whether the new report provides new insights that enable you to further narrow down the issue and its causes. If this is the case the original report has to be updated accordingly and, to avoid multiple reports of a single defect, no new report is created.

In the case of a “false positive”, this can be due to a defect in another part of the system or in the system environment. For example, a test case may fail because a network connection goes down or some other timeout kicks in. The resulting behavior is thus not due to the test object itself, even if the failure appears there. In cases like this the tester should, if possible, assign the issue to the causal component³⁰ in order to minimize the number of false positive findings.

If in doubt make a report

Time will often be too tight to decide which of the four cases listed above applies to each entry in a defect report, especially where you suspect “false negative” or “false positive” results. In these cases, you will often have to perform additional subsequent analysis, and it is often preferable to create a defect report in order to ensure that this analysis isn’t overlooked.

This is also true for any other significant unexpected event that occurs during testing—for example, a symptom that infers a faulty test object but that isn’t explicitly addressed by a test case.