Test cycle monitoring is all about collecting information and feedback relating to all test activities. Information can be collected manually or automatically and are then used to assess testing progress and evaluate whether the exit criteria or in agile projects the “definition of done” have been fulfilled. For example, these could be fulfillment of product risk coverage, requirements, or acceptance criteria.
Ideally, test activity and success monitoring is measured using the metrics defined in the test plan. The test manager can then use these metrics to assess testing progress in relation to the test execution schedule and the available budget, and to assess the appropriateness and effectiveness of the testing techniques in use. Test metrics are differentiated as follows:
Side Note: Categories of test metrics
- Defect-based metrics
The number of identified faults and/or defect reports (per test object) for each release, dependent on the class of defect and defect status (, table 6-4) and possibly in relation to the size of the test object (in lines of code) or test duration, or similar. - Test case-based metrics
Number or percentage of test cases with a specific status—for example, specified, planned, blocked, run, passed/failed - Test object-based metrics
Covered requirements, code coverage, dialog coverage, covered installed versions, platforms, and so on - Cost-based metrics
Accumulated testing costs, the cost of the next test cycle in relation to the expected usefulness (prevented cost of failure and/or reduced product or project risk) - General progress
For example, the percentage of completed tasks in relation to those planned
Our Tip
- The test manager should only use metrics that can be regularly, reliably, and easily gathered and evaluated. This is always the case with test tools that gather metrics automatically. If a metric proves too difficult to gather or doesn’t provide useful data, it should be removed from the testing strategy or replaced by a more appropriate one.
Leave a Reply