The Cost of Testing vs. The Cost of Defects

Cost-benefit relation

If the scope of checks and tests is reduced (or some tests are simply dropped), the number of undiscovered defects will increase accordingly. These then become part of the shipped product and increase the level of product risk and the likelihood of incurring additional costs:

The cost of a flawed product

• Direct costs
  These are costs that arise from defects that occur when the software is in use (which the manufacturer may be liable for). These can be the cost of incorrect calculations (lost data, accounting errors, damage to hardware or other plant components, personal injury); the costs involved when software-driven machines fail; the cost of installing new product versions and staff training; and so on. These kinds of costs are easy to overlook but can be significant—for example, consider the time and effort involved in installing a new software version at all customer sites.
• Indirect costs
  These are costs to the manufacturer that occur when the customer is not happy with the product. These can be contractual penalties, reduced payments due to non-fulfillment, or increased hotline and support costs. These kinds of issues can damage a manufacturer’s image and even lead to a loss of licensing for safety-critical products.
• Cost of defect correction
  These are the costs related to the effort involved in analyzing and correcting defects, regression testing, delivery and installation of the amended software product, re-training users, delays in releasing new products due to development staff involved in defect correction, and reduced competitiveness.

Risks due to product defects

The probability of these costs occurring and to what extent (i.e., the degree of risk involved in product defects) can be estimated using risk analysis. The level of risk depends on the type and scope of the product, the customer’s industry (contractual and legal stipulations), the number and type of defects, and the number of affected installations/end-users. There are also significant differences between custom and off-the-shelf software.

Early defect detection cuts costs

However large (or small) the risks, it is essential to identify the defects that cause them as early as possible. The costs of correcting defects increases significantly with every stage of the development process:

• An undiscovered fault (such as a poorly defined requirement) that occurs early on can lead to after-effects in the development process that multiply with each development phase.
• The later a defect is discovered, the more corrections are required to mitigate it. You may have to rework parts of previous project phases (requirement definitions, product design, coding) or, in agile projects, repeat some already completed tasks (which delays progress and postpones some objectives until later iterations).
• If the software is already installed at the customer site, the risk of additional direct and indirect costs increases. The potential cost and consequences of failure in safety-critical software (plant control, traffic control, medical applications, and similar) can be catastrophic for a manufacturer.