Test Analysis

“What” do we need to test?

Test analysis involves determining what exactly needs to be tested. For this purpose, the test basis is examined to see whether the documents to be used are sufficiently detailed and contain testable features in order to derive test conditions. The degree to which the test conditions need to be checked is determined by measurably defined coverage criteria.

The following documents and data can be used to analyze the test basis and the planned test level:

Analyzing the test basis

• Requirements that specify the planned functional and non-functional system or component behavior. The requirements specifications include technical requirements, functional requirements, system requirements, user stories⁹, epics¹⁰, use cases, and similar work products or documentation. For example, if a requirement doesn’t specify the predicted result and/or system behavior precisely enough, then test cases cannot be simply derived. A reworking is necessary.

Analyzing documentation

• Design or implementation data that gives rise to specific component or system structures. System or software architecture documents, design specifications, call graphs, model diagrams (for example, UML¹¹ or entity relationship diagrams¹²), interface specifications, and similar materials can be used for the test basis. For example, you will need to analyze how easily interfaces can be addressed (interface openness) and how easily the test object can be divided into smaller sub-units in order to test these separately. You need to consider these aspects at the development stage, and the test object needs to be designed and coded accordingly.

Check the test object

• You need to investigate the individual components or the system itself, including the code base, database metadata, database queries, and other interfaces. For example, you need to check that the code is well structured and easy to understand, and that the required code coverage is easy to achieve and verify.

Consider risk analysis

• Risk analysis reports that cover functional, non-functional, and structural aspects of the system or its components need to be investigated too. If potential software failures create serious risks, testing needs to be correspondingly thorough. Testing can be performed less formally for software that is not mission-critical.

Potential documentation errors

The cornerstone of the entire testing process is the test basis. If the test basis contains defects, you cannot formulate “correct” test conditions and you won’t be able to draft “proper” test cases. You therefore need to analyze the test basis for defects too. Check whether it contains ambiguities, or whether there are gaps or omissions in the descriptions of functions. You need to check the documentation for inconsistencies, imprecision, contradictions, and for repeat and/or redundant passages. Any defects or discrepancies you find should be corrected immediately.

The discovery and removal of defects from the test basis is extremely important, especially if the documentation hasn’t been reviewed. Development methodologies such as behavior-driven development (BDD, [URL: BDD]) and acceptance test-driven development (ATDD, [URL: ATDD]) use acceptance criteria and user stories to create test conditions and test cases before coding begins. This approach makes it simpler to identify and remedy defects at a much earlier stage in the development process.

Prioritizing tests

Once the test conditions have been identified and defined, you need to prioritize them. This ensures that the most important and high-stakes test conditions are tested first. In real-world situations, time restrictions often make it impossible to perform all the planned tests.

Traceability is important

At the planning stage, you need to ensure that there is unambiguous bi-directional traceability between the test basis and the results of your testing activities (see above). This traceability has to precisely define which test condition checks which requirement and vice versa.

This is the only way to ensure that you can later determine how thoroughly which requirements need to be tested and—depending on the test conditions—using which test cases.

Choosing test techniques

It is useful to consider which test technique to use at the analysis stage. Each technique has its own system for reducing the likelihood of overlooking test conditions and helping to define these precisely. For example, using an equivalence partition (or “equivalence class”) test ensures that the entire input domain is used for the creation of test cases. This prevents you from forgetting or overlooking negative input in your requirements definitions. You can then define your test conditions to cover negative input data.

If you use an experience-based testing technique you can use the test conditions defined during analysis as objectives for your test charter. A test charter is a kind of “assignments” that, alongside traditional test objectives, provides potential ideas for additional tests. If the test objectives are traceable back to the test basis, you can evaluate the achieved coverage (for example, for your requirements), even when you are using an experience-based technique.