Selecting a Testing Strategy

Selecting an overall testing strategy is one of the most important decisions that a test manger has to make. The overall objective is to establish a set of measures that maximize the effectiveness and efficiency of testing within the predefined project constraints. Or, viewed the other way around, the overall objective of testing is to minimize project and product risks.

Developing a strategy

The addressed the topics that need to be covered by strategic test planning (i.e., the “what”). The addresses the “how” of selecting or developing appropriate measures. Test strategy development can be broadly divided into two basic avenues of thought:⁹

• Decision-making and design freedom
• Available knowledge of the project and the product

Preventative vs. reactive approach

The moment at which testers first become involved in a project has a significant effect on whether certain elements of a testing strategy are actually workable and therefore whether they are to be seriously considered. There are two basic types of situation in which these kinds of decisions are made:

• Preventative
  Testers are involved from the very start of the project. The test manager can intervene and shape the testing strategy proactively with a view to optimizing the processes involved and reducing costs. Error mitigation measures (such as a design review), an early start for testing activities (such as specifying tests), and evaluating interim results (for example, using static analysis techniques) all contribute to avoiding faults, or at least discovering them as early as possible in the development process. This reduces defect density during dynamic testing, thus reducing costs and contributing to the stability and reliability of the product. A preventative testing strategy is often mandatory for safety-critical applications.
• Reactive
  If testers become involved later on it is usually too late to create a preventative strategy. They then have to react to the existing situation or unplanned situations and other unforeseen circumstances. One reactive technique that is often quite successful is exploratory testing. This is a heuristic approach to testing that involves “exploring” the test object and designing, executing, and evaluating the resulting tests on the fly.

Our Tip When should testing begin?

• You should take a preventative approach to testing whenever possible. Cost analysis clearly shows that testing and evaluation should begin as soon as possible within the project and continue throughout all phases of the project’s lifespan.

Case Study: Test planning for the VSR-II project

For VSR-II, test planning and test specification begins immediately following release of the initial requirements document. At least one roughly worded test case is designed for each requirement, and the results are reviewed by representatives from the appropriate business departments, by the developers, and by the designated system testers. The review reveals that many of the requirements are vague or incomplete and that some of the test cases are faulty or inadequate.

Simply writing down and discussing meaningful test cases with the developers has helped to identify a whole raft of issues long before any tests are actually performed or any program code written.

Analytical vs. heuristic approach

On an additional level, the available knowledge of the project and the product affects which testing strategy elements you should consider. There are two very different approaches that we can consider here:

• Analytical
  The testing strategy is based on systematic data analysis. The factors that directly influence strategic decisions (in whole or in part) are determined and their mutual relationships are modeled mathematically.
• Heuristic
  The testing strategy is based on in-house or external expert experience and/or on “rules of thumb”—either because no data is available, modeling is too complex, or simply because the required know-how is not available.