Testing and Risk

“Risk” is one of the best criteria to use when selecting and prioritizing test objectives, testing techniques, and test cases.

Risk = (expected loss in case of accident) x (the probability of the accident occurring)

Risk is defined as the product of the expected loss in the case of an accident and the probability of that accident occurring.

The “loss” in this case is the consequence and cost of the product failing. The probability of such a loss occurring depends on how the software product in question is used (i.e., its usage profile).

In practice, the parameters “loss” and “probability” can seldom be precisely defined and are usually estimated. A rough guess or screening using ranges or classes¹³ is usually sufficient. When evaluating and handling risk, we need to distinguish between project and product risks.

Project risks

Project risks can jeopardize the success of a project, or even kill it completely. Such risks include:

• Supplier issues such as when a sub-contractor is unable to fulfill a contract. This can cause many types of delay, up to and including legal disputes.
• Organizational issues. One example is when required resources become scarce—for example, if there aren’t enough people available with the skills required by the project, or if necessary training is planned but doesn’t take place. Political or interpersonal issues can hamper your work too. Testers may not adequately communicate their needs and/or their test results, or a lack of cooperation between departments or competing projects creates conflict. Decisions regarding cost reductions can also represent a risk to a project.
• Weaknesses in the development process such as inefficient defect management or a lack of adequate configuration management can undermine a project.
• Technical issues
  Incorrect, incomplete, ambiguous, underrated, or unrealizable requirements, and late changes to requirements can easily cause the entire project to fail. If new technologies, tools, programming languages, or other methods are introduced without employing people with appropriate experience, they can be detrimental to the project rather than delivering the desired improvements. If interim results (such as design documentation, program code, or test cases) are of poor quality or contain defects that remain undiscovered, this too can put the project at risk. Testing itself involves technical risks, such as the late deployment of the test environment, non-functioning automation processes, or inadequate test data.

Product risks

Product risks are those that results from (or can result from) issues with the deliverable product. These include:

• The product doesn’t perform its intended function or doesn’t work at all.
• Features don’t display the planned functionality, or non-functional characteristics are missing or incomplete (such as poor performance or usability). These kinds of risks are also referred to as quality risks.
• The system architecture is not scalable according to the growing needs of the application (for example, increasing number of users or connecting additional systems).
• Poor data quality, perhaps due to an earlier migration or conversion of system data.
• Use of the product causes physical damage or endangers human life.

Risk management

Project management needs to introduce systematic risk management¹⁴ that identifies risks at an early stage and introduces appropriate countermeasures that either nullify the risk or at least reduce its effects to a controllable level. The following risk management steps need to be taken regularly:

• Risk analysis that anticipates potential risk factors
• Prioritize identified risks
• Plan and/or introduce measures that reduce or obviate identified risks
• Create contingency plans in case a (serious) risk occurs

Testing reduces risk

Testing is one of the most important anti-risk measures. Testing delivers data on existing, real-world issues and risks, as well as on troubleshooting success (or failure). Testing reduces the uncertainty inherent in risk factors, and helps you to estimate levels of risk and identify new risks.

A test manager’s job is therefore to minimize or eliminate product risks by identifying faulty product functionality as early as possible, and by ensuring that any corrective measures taken actually work¹⁵.

Project risks (for example, sick leave) cannot be addressed directly by testing. However, testing can reduce the negative effects of project risks that are due to the occurrence of product risks (for example, an increased number of coding errors as a result of time pressures due to a developer sick leave).

Risk-based testing

A risk-based testing strategy reduces and combats product risks right from the start of the project. Such a strategy uses data on identified risks to plan, specify, prepare, and execute tests. In this case, all the most important elements of the testing strategy are risk-based:

• The techniques used to derive test cases, the test levels, and the types of test (for example, security or accessibility testing)
• The extent of testing—i.e., the number of test cases and the number of product configurations (variants or releases) to be tested
• Test case prioritization (see below)

If risk analysis hasn’t already taken place as part of the general project risk management process, the test manager will need to analyze product risks and prioritize them according to the level of knowledge and insight of the project stakeholders.

Evaluation of the probability and effects of a particular risk has to take place in the context of how the product is to be used. In other words, the evaluation takes place from the end-user’s viewpoint with a view to the product’s potential usage scenarios.¹⁶

In addition to testing itself, other activities (such as additional training for inexperienced designers) can be employed to further reduce risk.

Risk-based test prioritization

Risk-based testing involves prioritizing test objectives and test cases according to the assumed probability of a risk actually becoming an event. This ensures that high-risk components or features are tested earlier and more thoroughly than other parts of the system. Serious issues that require a high degree of remedial effort that is likely to delay the project are thus identified early on.

If you distribute your testing resources evenly, critical and non-critical test objects will be tested equally thoroughly. This means that critical objects are not sufficiently tested while resources are wasted on non-critical objects. Risk-based testing prevents this from happening.