Testing Strategies

Test Planning

A task as complex as testing requires meticulous planning on an operational level and on a strategic level.

The starting points for strategic planning are provided by company testing policy and testing guidelines (if they exist), or by a generic company testing strategy. The test manager needs to transpose such generic guidelines to create a concrete testing strategy for the project at hand that fits the specific objectives, constraints, and risks of the project, the type, importance and risk level of the product being tested, and that conforms to any existing quality assurance plans³. A test manager’s strategic planning involves the following tasks:

Tasks involved in strategic test planning

• Specify the test object(s)
  Identify which components, modules, neighboring systems, and interfaces (will) make up the system to be tested. Decide which of these objects need to be tested and which can or should be excluded from the testing process.
• Draft the test objectives
  Analyze, define and draft the specific testing objectives and the criteria you are testing against for each test object and the entire system.
• Customize the testing process
  Customize the testing process foreseen by company policy or the project development model to suit the requirements of the project at hand. Define which test levels are necessary (based on the nature of the test object and the test objectives you have defined). Discuss and define coordination and cooperation between testing and other project activities.
• Select your testing methods and techniques
  Select and specify the overall testing approach and the testing techniques that are suitable and/or necessary to achieve the defined test objectives for the individual test objects and the entire system. Determine and select the training courses necessary to implement or improve the operation of the selected testing techniques. Analyze the type and scope of potential test automation options.
• Specify the required infrastructure
  Analyze which test environment(s) are required, which are already available, and which need to be extended or built from the ground up. Check which tools can or must be used to support the testing process. Document and acquire the stipulated tools and other resources required by the test team.
• Define test metrics
  Select and describe the metrics you will use to control the testing process. Specify the measurement techniques, evaluation criteria, and limit values⁴, as well as the required reactions to test results and test exit criteria.
• Specify the reporting system
  Specify which documents and proofs are to be created and maintained (test schedule, test logs, defect reports, test reports, and so on). Specify who is responsible for creating, maintaining, publishing, presenting, and archiving reports. Provide templates and tool configurations that implement the required structure and degree of detail.
• Plan costing and overall testing effort
  Provide an initial estimate of the overall testing effort and the corresponding resources (including personnel) required to execute the planned testing strategy. Calculate the overall cost of testing and secure the necessary budget. Update estimates and costs during the project.

The result is your testing strategy

The result of all this strategic planning is a testing strategy.⁵ The test manager uses the specified strategy to determine the testing objectives and constraints for the current project and documents them as part of the overall test plan. A well-formulated test strategy will include justification for the decisions it contains. It will usually also contain references to alternative approaches that were considered and could still be used if the decisions made (for example, regarding the available resources) should prove to be incorrect or unworkable.

Part 3 of the ISO 29119 standard [ISO 29119]⁶ provides a reference structure. A testing strategy can be worded generally so that it can be used for multiple members of a product family or for all test levels within a project (a “master test plan”). However, a test plan can also be drafted for a single test level (a “level test plan”), for individual test objects, or for concepts or strategies that apply to a specific type of test (for example, usability or performance tests). These kinds of strategies are optional and are usually used as an addition to the master test plan in critical projects.

Coordinating and adapting a strategy

Creating a strategy obviously requires comprehensive research, as well as intense discussion and negotiation between all parties to the project. In the course of this coordination process, the test manager will have to continually adapt, slim down, or refine the original test plan until a consensus is reached about how to proceed. This will sometimes require reducing the scope of useful measures, or even discarding them completely. Such decisions should always be based on systematic risk weighting and analysis.

As a project progresses, the entire team will continually gain insight into and increasingly precise information about the product and its related technologies, and can then see whether these work well with the planned project strategy.⁷ This is especially relevant to testing activities and the strengths and weaknesses of the testing strategy.

The results of tests that have already been executed and feedback from pilot customers help to identify the kinds of issues that testing has revealed and which defects have slipped through the net. They can also be used to analyze which test levels or test types contribute most to testing effectiveness, and how much effort is involved.

Feature planning changes continuously too. Features are always being adapted or discarded, and new or changed requirements are constantly added during the course of a project. This makes it essential to regularly assess and update project and product risks.

Strategic test planning is a continuous task

Operative and strategic planning are continuous tasks that the test manager has to perform through every phase or iteration⁸ of a project. A test manager has to remain critical and continuously update the testing process, the testing strategy, and the operative test plan whenever the situation changes.