Software automation tool selection, purchase, and maintenance costs money, while acquiring appropriate hardware and training staff to use them uses further resources. The cost of rolling out a complex tool to a large number of workstations can quickly run into six figures. As with any other investment, the planned amortization period of a new testing tool is a critical point when deciding whether to go ahead with a rollout.
Cost-benefit analysis
In the case of tools that automate test execution (test robots), it is relatively simple to estimate the savings compared with running tests manually. The effort involved in programing automated tests has to be subtracted from these savings, so a cost-benefit analysis usually delivers a negative result after a single test run. It is only once you have performed multiple automated regression test runs (see figure 7-2) that you really begin to save money with each test execution cycle.
Fig. 7-2Test case lifecycle
The balance becomes positive following a number of regression test runs, although it is still tricky to accurately estimate the amortization period. You will only effectively cover your costs if you can program test cases that are regression test-capable. If this is the case, a capture/replay tool can begin to make real savings after just three test cycles. However, this kind of calculation only makes sense if manual testing is a real alternative. Many tests (for example, performance tests) cannot be performed manually anyway.
Evaluate the influence on overall testing effectiveness
However, evaluating testing effort only doesn’t tell the whole story. You need to consider how a tool increases overall testing effectiveness too—for example, when more failures are revealed and thus corrected. In the medium term, this can reduce the development, support, and maintenance costs of a project, providing a much more attractive potential for overall cost savings.
Potential benefits of introducing a test tool
To summarize, the potential benefits are:
- Tools cannot replace the creative side of testing, but they can support it. Using tools creates better test cases and test data, and increases the overall effectiveness of the testing process.
- Automation reduces testing effort (or increases the number of test cases that can be performed using the same amount of effort). However, using more test cases doesn’t automatically equate to better testing.
- Tools simplify the collection, evaluation and communication of data relating to test progress, achieved coverage, and failures and their distribution among the test objects. These data help you to paint an objective picture of the quality of the software being tested.
Potential risks of introducing tools
Introducing new tools into the testing process also involves risks. These include:
- The expectations attached to a tool can be unrealistic.
- It is all too easy to underestimate the time it takes and the costs involved in introducing a tool. The effort involved in using the new tool can be underestimated too.
- A tool cannot be used effectively (or cannot be used at all) because a defined testing process is either not established or, if it exists, not well practiced.
- The improvements and changes in the testing process required to realize the tool’s potential are more extensive than anticipated.
- The effort involved in maintenance and version control for test cases, test data, and other test documents is greater than expected.
- Using the tool in situations where manual testing is simpler and more economical.
- The new tool does not work well enough with existing tools.
- A tool’s manufacturer goes out of business or no longer supports the product.
- The open source project developing the tool is discontinued.
- New versions of a tool no longer support older technology that is still used in the system being tested.
- Tool support is poor or is no longer provided.
Leave a Reply