Evaluation and Reporting

Test and project management can evaluate the defect database to provide an overview of the current state of defect correction and the resulting product quality.³⁶

Input for test and project management

Test management uses evaluation results and defect-based metrics to adapt the test plan (for example, to add tests for particularly fault-prone components, or for confirmation tests for corrected components), but also as a basis for conclusions and recommendations relating to product approval and release.

Project management uses correction progress reports to judge whether the planned release schedule is still valid or whether a release needs to be postponed. As well as localizing faults, determining how many failures have been found, checking how many of these have been corrected and how many issues are still open, trend analysis is an important part of the report evaluation process too. Trend analysis provides an estimate of how many defects are likely to be reported in future and helps to evaluate whether the number of defects is increasing or the incidence curve is flattening out.

Tips for improving test coverage

Defect report data can also be used to improve test coverage. For example, a comparison of the data for different test objects may show a test object with relatively few defects—this could, however, be due not to more careful programing, but rather to missing test cases.

Input for process improvement

This data can also provide input and ideas for general process improvement. For example, if it turns out that a large number of defects are caused by misunderstood requirements, you can use this finding as a good reason to initiate measures for improving the requirements engineering process.

Case Study: Extended exit criteria for the VSR-II system test

The exit criteria for the VSR-II system test need to reflect testing progress and the current level of product quality. The test manager extends the exit criteria accordingly to include the following defect-based metrics:

• All Class 1 defects are “done”
• All defects with “patch” priority are “done”
• The number of “new” defect reports per week is stagnating or decreasing