Boundary Value Analysis

A useful addition

Boundary value analysis provides a useful addition to test cases determined using equivalence partitioning. This is because faults often occur at the boundaries of equivalence partitions where error-prone special cases occur. Tests with boundary values often reveal previously undiscovered defects. Boundary value analysis can only be applied if the data set in an equivalence partition is numerically ordered or can be (at least partially) defined using some kind of order. The boundaries themselves have to be clearly identifiable. The test cases include the precise boundary value and the two neighboring values (inside and outside the partition). Where floating-point numbers are involved, you have to select a suitable degree of computational accuracy that equates to the smallest available increment in each direction.

Each boundary therefore yields three test cases. If the upper boundary of one partition coincides with the lower boundary of the next, the corresponding test cases coincide (see figure 5-3).

Fig. 5-3Coincidental boundaries in neighboring equivalence partitions

In many cases, there is no “real” boundary value, as the boundary value actually lies within one of the neighboring equivalence partitions. In such cases, it is usually sufficient to test the boundary using two values that lie just outside the equivalence partition (but within the neighboring partition) on either side of the boundary.

Case Study: Boundary values for calculating discounts

Four valid equivalence partitions were determined and corresponding representatives selected for the discount calculation (see table 5-1). Equivalence partitions 3 and 4 are specified with vEP₃: 20000 < x < 25000 and vEP₄: x ≥ 25000. To test the boundary between the two partitions (25000), we use the values 24999 and 25000 (using whole dollar values for simplicity). The value 24999 lies within vEP₃ and is the highest possible value in this partition. The value 25000 lies within vEP₄ and is the lowest possible value. The values 24998 and 25001 lie “deeper” within their respective equivalence partitions and therefore provide no useful additional results.

Side Note: Two tests or three?

When are the two test cases 24999 and 25000 sufficient and when do we need to add a third case (25001) to test the boundary value 25000?

A closer look at the way the query is formulated helps to answer the question. In our example, we assume that the corresponding place in the program includes the query if (x < 25000)¹¹. Which boundary values can we use to check whether this condition has been correctly implemented? Using the test values 24999, 25000, and 25001 returns the logical values true, false, and false, and the corresponding program paths are executed. Testing the value 25001 appears to provide no additional benefit as 25000 already returns false, and therefore facilitates the switch to the neighboring equivalence partition. Running the flawed query if (x ≤ 25000) returns the logical values true, true, and false. Here too, we can do without testing 25001, as 25000 already returns the wrong result and reveals the fault. If we run the completely erroneous query if (x <> 25000), we receive the results true, false and true, so in this case we have to test 25001 as well to reveal the faulty query. The values 24999 and 25000 deliver the expected results (i.e., the same as when the query is correctly formulated).

Note: if (x > 25000) returns false, false, true; if (x ≥ 25000) returns false, true, true; and if (x == 25000) returns false, true, false. These queries return two or three deviations from the required logical results, so again we only need to execute two test cases (using 24999 and 25000) to reveal the fault(s).

Table 5-8 illustrates the situation with a list of all the queries and the corresponding logical values returned by the boundary values.

Table 5-8Three boundary values used to test the query

You always have to decide when two values are sufficient and when you need three to test a boundary value. The defective code if (x <> 25000) can be revealed by a code review, as the query tests non-equality instead of a boundary value (if (x < 25000)). However, this fault is still easy to overlook. Only a boundary value test with three values at every boundary is certain to reveal every defective implementation of the query.

Case Study: Integer input

The integer input example described in yields five additional test cases, giving us a total of 12 test cases with the following input values:

{“f“,

MIN_INT-1, MIN_INT, MIN_INT+1,

-123,

-1, 0, 1,

654,

MAX_INT-1, MAX_INT, MAX_INT+1}

The test case with the value -1 tests the maximum value in the equivalence partition EP₁: [MIN_INT, …, 0[ and also the minimum under-run of the lower boundary (0) in the equivalence partition EP₂: [0, …, MAX_INT]. From the point of view of the latter equivalence partition, the value -1 lies outside the partition. Note that for technical reasons, values above the upper boundary or below the lower boundary can sometimes not be used as concrete input values.

Our example only lists the input variables. However, each test case must also include the expected behavior of the test object (i.e., its output) according to the test oracle, and the corresponding pre- and postconditions.

Is the testing effort justified?

Here too, you need to decide whether the effort involved in testing every boundary and its neighboring values is justified. You could leave out the test cases with the representatives of equivalence partitions that don’t test boundaries. In our example, these are the test cases with the input values -123 and 654. Because the maximum and minimum values in the equivalence partition already have their own test cases, it is assumed that test cases with input values taken from the middle of the partition won’t provide any additional insights. In our example, these are the values MIN_INT+1,1 and MAX_INT-1.

No boundary values in the set

If we use the input value “potential buyer” in the example above, we cannot determine boundaries for the input value range. Mathematically speaking, the input value has to be selected from one of the four elements of the set (employee, student, trainee, senior). There are no identifiable boundaries and it makes no sense to order the member of the set according to their age. Students can be of any age, so it is more likely that a person’s job status is more conclusive.

Boundary value analysis can of course be applied to equivalence partitions populated with output values.

Test Cases

The same way that test cases can be derived from equivalence partitions, the valid boundaries within an equivalence partition can be combined to build test cases. Invalid boundaries have to be tested separately and cannot be combined with other invalid boundaries.

In the example described above, values from the middle of an equivalence partition do not need to be tested if both boundary values within the partition have already been earmarked for use in test cases.

Case Study: Boundary value tests for calculate_price()

If we apply boundary value analysis to the valid equivalence partitions for testing the calculate_price() method, we end up with the boundary-based test data shown in table 5-9:

Table 5-9Boundary values for the calculate_price() method’s parameters

Considering only the boundary values that lie within the equivalence partition, we have 4+4+4+9+4 = 25 boundary-based representatives. Two of these (extras: 1, 3) are already covered (Test cases 1 and 2 in table 5-7). This leaves us with the following 23 boundary values for input in further test cases:

baseprice: 0.00, 0.01¹³, MAX_DOUBLE-0.01, MAX_DOUBLE

specialprice: 0.00, 0.01, MAX_DOUBLE-0.01, MAX_DOUBLE

extraprice: 0.00, 0.01, MAX_DOUBLE-0.01, MAX_DOUBLE

extras: 0, 2, 4, 5, 6, MAX_INT-1, MAX_INT

discount: 0.00, 0.01, 99.99, 100.00

Because these are all valid boundary values, they can all be combined to form test cases (see table 5-10).

Table 5-10Additional test cases for the calculate_price() method

The expected results of a boundary value test are not always clearly defined in the test object’s specifications. An experienced tester therefore has to define appropriate target behavior using the available test cases:

• Test case 15 checks all the valid lower boundaries for the calculate_price() method’s parameters. The test case appears to have no direct reference to reality¹⁴. This is due to the imprecise specification of the method that doesn’t specify lower and upper boundaries for the parameter values (see below)¹⁵.
• Test case 16 is analogous to test case 15 but tests computational accuracy¹⁶.
• In test case 17 the expected result of a 99.99% discount is highly speculative. The specifications of the calculate_price() method state that prices are added, so it makes sense to test the maxima individually (see test cases 18-20). The values for the other parameters were taken from test case 1 in table 5-7. Further useful test cases result if we set the other parameter values to 0.00 to test whether the maxima are correctly processed without further addition (no overflow?).
• Similarly to test cases 17-20, test cases for the value MAX_DOUBLE need to be executed too.
• Additional test cases need to be set up for the boundary values that haven’t yet been tested (extras = 5, 6, MAX_INT-1, MAX_INT and discount = 100.00).

The boundary values outside the valid equivalence partitions are not included here.

It is never too early to think about testing. Start with the specifications!

This example shows clearly the effects that imprecise specifications can have on testing¹⁷. If a tester gets information about the parameter range directly from the customer before setting up any test cases, the overall testing effort will be reduced.

The following example illustrates the principle.

The customer provides the following information:

• The base price lies between 10000 and 150000
• The special edition premium is between 800 and 3500
• There are up to 25 optional extras with prices between 50 und 750
• The maximum dealer discount is 25%

Forming equivalence partitions for these numbers results in the following valid parameter values for boundary testing:

baseprice: 10000.00, 10000.01, 149999.99, 150000.00

specialprice: 800.00, 800.01, 3499.99, 3500.00

extraprice: 50.00, 50.01, 18749.99, 18750.00¹⁸

extras: 0, 1, 2, 3, 4, 5, 6, 24, 25

discount: 0.00, 0.01, 24.99, 25.00

All of these values can be freely combined to form test cases. A single test case is required for each value that lies outside of an equivalence partition:

baseprice: 9999.99, 150000.01

specialprice: 799.99, 3500.01

extraprice: 49.99, 18750.01

extras: -1, 26

discount: -0.01, 25.01

Precise specifications reduce the number of test cases

It is obvious that precise specifications provide clear expected results and require fewer test cases.

Adding a test case for the “machine boundary values” (MAX_DOUBLE, MIN_DOUBLE and so on) is recommended. This ensures that any potential failures caused by hardware limitations are successfully identified.

Here too, you have to decide whether a boundary needs to be tested with two or three input values. The following tips assume that code reviews have already detected any faulty queries and that two input values are therefore sufficient.

Our Tip on setting up test cases using boundary value analysis

• For every input value range, you need to include the boundary values and the neighboring values outside the range. For example, for the range [-1.0; +1.0], the test data are -1.0 and +1.0, -1.001 and +1.001¹⁹.
• If a file contains a number of records specified between 1 and 100, boundary value analysis results in the following options: the file contains no records, the file contains one record, the file contains100 records, the file contains 101 records.
• If test cases are based on an output range, you could proceed as follows: If the expected output is a whole number between 500 and 1000, you need to test the results 500 and 1000, as well as 499 and 1001. Determining the corresponding input values might involve significant effort and you may find that it is impossible to provoke defective output. Nevertheless, considering how to do this will help you to identify additional faults.
• If the number of output values is critical, you need to proceed the same way as for input values—i.e., for 1-4 output values, you need to set up test cases that check for 1, 4, 0, and 5 output values.
• For ordered data sets, the first and last elements in the set are the most important when it comes to testing.
• If the input or output values are complex data structures, you can include a zero matrix or an empty list as a test boundary value.
• For numerical calculations, your test cases need to include closely neighboring values and ones that lie far apart.
• Boundary value analysis only makes sense for invalid equivalence partitions if you expect the boundary values to produce differing behaviors in the test object.
• Always perform additional test with very large data structures, lists, tables or similar objects to check the test object’s behavior when buffer, file, and memory limits are exceeded. When testing lists and tables, the first and last elements (and empty lists) are the most useful cases to watch out for, as these are often subject to coding errors (so-called off-by-one issues).

Defining Exit Criteria

As with equivalence partitions, a measurable degree of boundary value (BV) coverage can be defined as an exit criterion:

BV coverage = (number of tested BV / total number of BV)×100%

Note that this calculation includes the boundary values and their direct neighbors on either side of the boundary, but that only unique values are used. Because they are covered by only one test case, coincident values from neighboring equivalence partitions count as one value.

Benefits and Limitations

In combination with equivalence partitions

Boundary value analysis is most useful when used in combination with equivalence partitions, as errors at the boundaries of equivalence partitions are more common than ones based on values from the middle of a class. Both techniques offer a good degree of freedom in the selection of concrete test data.

Boundary value analysis may appear simple, but determining the relevant boundaries is a complex task. Selecting appropriate boundary test data requires a high degree of creativity, especially when analysis isn’t based on purely numerical data ranges.