Test Management Tools

Test management

Test management tools offer ways to register, catalog, and manage test cases and their priorities. They also enable you to monitor the status of test cases—in other words, you can record and evaluate whether, when, or how often a test has been performed and the results it delivered (pass/fail).

They help test managers to plan and manage hundreds or thousands of test cases. Some tools also support project management tasks within the testing process (such as resource planning and scheduling).

Today’s test management tools support requirements-based testing too, with requirements either entered directly or imported from a requirements management tool. Once imported, every requirement can then be linked to the test cases that test whether it has been fully and correctly implemented. The following example illustrates the process.

Case Study: Requirements-based testing for the VSR-II DreamCar module

The team working on the VSR-II DreamCar module uses epics to describe its functionality [URL: Epic]. The vehicle configuration and price calculation functionality is described by the following epic:

• Epic: DreamCar – Vehicle configuration and price calculationThe user can configure a vehicle (model, color, extras, and so on) on a screen. The system displays all available models and equipment variants and displays the corresponding list price in real time. This functionality is implemented in the VSR-II DreamCar module.

The detail of the epic is filled out with user stories. The following user stories are assigned to the DreamCar epic:

• User Story: Select vehicle model
  • The system displays all available base models
  • The user can select one of the displayed base models
• User Story: Configure vehicle
  • The system displays all available options for the selected base model (equipment options, special editions, extras)
  • The user can select the desired options (for example, red pearl effect paint, the OutRun sports package, the OverDrive music system, and the Don’tCare autopilot system). Options that are generally available but not for the current configuration are grayed out.
• User Story: Calculate vehicle price
  • The list price for the selected base model and configuration options is calculated and displayed

To check whether a user story has been correctly and fully implemented, the team defines corresponding test cases and links them to the corresponding user story using a test management tool. The test cases defined for the configure vehicle user story are as follows:

• Test Case: Base model with one option
• Test Case: Base model with multiple options
• Test Case: Base model with non-selectable option
• Test Case: Base model with equipment package option
• Test Case: Base model with no options

Requirements Management

Requirements management tools save and manage all kinds of data relating to requirements. They also enable you to prioritize requirements and track their implementation status.

These are not test tools in the narrow sense but are nevertheless very useful when it comes to deriving and planning requirements-based tests —for example, based on the implementation status of individual requirements. Most requirements management tools can swap data with test management tools, which can then create links between requirements, test cases, test results, and defect reports.

Traceability

This in turn enables full traceability for every defect or defect report (solved or unsolved), the relevant requirements, and with regard to which test cases have been executed. The verification and validation status for every requirement is then fully transparent, so requirements that have been incorrectly implemented or overlooked (i.e., not covered by any test cases), or those with gaps in their specifications can be easily identified.

Defect management

Defect management tools are a must for every test manager. As described in these tools register, manage, distribute, and statistically evaluate reported defects. High-end defect management tools include parameterizable defect status models and enable you to specify the entire process, from discovery of a defect through correction to regression testing. This way, the tool can guide everyone involved through the defect management process.

Configuration management

Configuration management tools too are not test tools in a strict sense. What they do is enable you to manage multiple versions and configurations of the software being tested, including different version of test cases, test data, and other product and test documentation. This makes it easier to determine which results from a test sequence were produced by which version of a test object.

Continuous Integration

In agile and iterative/incremental projects, integration and integration testing of modified components are usually highly automated. As soon as a block of code is checked in to the integration environment, all of the necessary integration steps and test processes are performed completely automatically (see also [Linz 14). This process is called continuous integration. Tools that control and perform this process are called continuous integration (CI) tools.

Tool integration

Test management tools require high-performance interfaces to a range of other tools. These include:

• A general export interface for converting data into a format that is generally readable for other teams and company departments (for example, in a spreadsheet).
• An interface to the requirements management tools for importing requirements or linking them bi-directionally. This enables requirements-based test planning and traceability of the relationships between requirements and test cases. Ideally, the test status of every requirement can then be tracked by the users of test management and requirements management tools.
• An interface between test management and test execution tools (test robots) for the provision of scripts and starting automated testing. The results are automatically returned and archived.
• An interface between test management and defect management enables seamless confirmation test planning (i.e., a list of the tests that are necessary to verify which defects have been corrected in a specific version of a test object).
• Test management tools can be linked to continuous integration (CI) tools so that tests planned using the test management tool can be executed as part of the CI process.
• Code files can be flagged with version numbers in the configuration management tool. A test management tool that can read such data can then link test cases or defect reports to the corresponding files and file versions.

Generating test summary reports and test documentation

Test management and defect management tools usually offer comprehensive analysis and reporting functionality and can, to an extent, generate test documentation (test schedule, test specifications, test summary report, and so on). The format and content of such documentation is usually configurable.

Test data can be evaluated quantitatively in a variety of ways. For example, you can analyze how many test cases have been run and how many of those were successful, or the frequency with which a particular class of defect is identified. These kinds of data help to judge and manage overall testing progress.