Testing Effort and Costs

The cost of implementing a testing strategy significantly influences the testing strategy itself. The test manager therefore has to estimate the effort involved in performing the planned testing activities and the costs this entails. The resulting cost forecast details either the budget that has to be applied for or the degree to which the planned testing activities under- or overrun the available budget.

The cost of testing always has to be balanced with the test objectives and the risks that testing is designed to reduce. For low-risk components, you can reduce the number of test cases or halt testing completely, while high-risk components usually imply the acceptance of greater costs.

Factors that influence testing effort

The factors that influence testing effort and therefore the cost of testing are varied and difficult to quantify. The most important factors are:

• The maturity¹⁷ of the development process
  • The number and frequency of changes to the software, the complexity of changes and corrections, and the number, seriousness, and frequency of failures
  • The maturity of the testing process, configuration management, defect management and change management, and the degree of routine and discipline in complying with these
  • Time pressure due to unrealistic estimates or planning
  • The overall stability of the company
• Software quality and testability
  • The number, seriousness and distribution of faults within the software
  • The quality, conciseness, and up-to-dateness of project documentation and other information that is relevant to the testing process
  • The size, type, and complexity of the system and its environment
  • The complexity of the application domain
• Test infrastructure
  • The availability of testing tools
  • The availability of the test environment and test infrastructure
  • The availability and recognition of the testing process, testing standards, and testing techniques
• Team and co-workers
  • Team experience and know-how with regard to testing knowledge, testing tools, the test environment, and the test object
  • Team experience of similar product and projects
  • Team cooperation and the quality of the relationships among testers, developers, management, and the customer
• Quality objectives
  • Planned test coverage
  • Planned reliability of the product in use
  • Required test documentation¹⁸ and its degree of detail
  • Legal and regulatory conformity (for example, with regard to system security, access security, and reliability)
• Testing strategy
  • Test objectives (derived from the quality objectives) and resources for achieving them, including the number and scope of the planned test levels
  • Selecting appropriate testing techniques
  • Test scheduling (start and finish, number of iterations)

Side Note: The test manager’s influenceprofiles

The test manager cannot directly influence many of these factors. From a test manager’s viewpoint, the situation usually looks like this:

• The maturity of the software development process cannot be influenced in the short term, so you simply have to accept it as it is. This factor can only be changed in the longer term through an appropriate process improvement program.
• The software’s testability depends heavily on the system architecture. Improving testability usually requires comprehensive changes to the system architecture and is therefore difficult to achieve in the short term. In the medium and long term, a well-structured development process (with appropriate reviews) will lead to a better overall software structure that is easier to test.
• The test infrastructure is often left over from previous or similar projects. Within the constraints of the testing budget, the test manager can certainly influence whether and how the test infrastructure can be developed to increase testing productivity. If the test environment has to be built from the ground up, be aware that this usually requires a lot of extra time.
• The test manager can influence the short-term structure of his team and the level of qualification within it simply by selecting the right personnel. In the medium term, he can increase the team’s know-how by offering appropriate training.
• If the quality objectives have already been defined by the customer and/or other stakeholders the test manager may be able to exert some influence by consulting with the stakeholders and prioritizing the objectives. If the test manager is able to set his own objectives they will become part of the overall testing strategy.
• The “test object’ and “test objectives” elements of the testing strategy can be prioritized too, while “testing techniques” are generally freely selectable and scalable.
• The most important variable that a test manger can adjust and control during test planning is the overall extent of test execution.