Structural testing alone often is not enough

If code is the source of all truth, why can’t we just do structural testing? This is a very interesting question. Test suites derived only with structural testing can be reasonably effective, but they may not be strong enough. Let’s look at an example (see the “counting clumps” problem, inspired by a CodingBat assignment.

The program should count the number of clumps in an array. A clump is a sequence of the same element with a length of at least 2.

• nums—The array for which to count the clumps. The array must be non-null and length > 0; the program returns 0 if any pre-condition is violated.

The program returns the number of clumps in the array.

The following listing shows an implementation.

Listing 3.9 Implementing the code clumps requirement

public static int countClumps(int[] nums) {
  if (nums == null || nums.length == 0) {    ❶
    return 0;
  }
  int count = 0;
  int prev = nums[0];
  boolean inClump = false;
  for (int i = 1; i < nums.length; i++) {
    if (nums[i] == prev && !inClump) {       ❷
      inClump = true;
      count += 1;
    }
    if (nums[i] != prev) {                   ❸
      prev = nums[i];
      inClump = false;
    }
  }
  return count;
}

❶ If null or empty (pre-condition), return 0 right away.

❷ If the current number is the same as the previous number, we have identified a clump.

❸ If the current number differs from the previous one, we are not in a clump.

Suppose we decide not to look at the requirements. We want to achieve, say, 100% branch coverage. Three tests are enough to do that (T1–T3). Maybe we also want to do some extra boundary testing and decide to exercise the loop, iterating a single time (T4):

• T1: an empty array
• T2: a null array
• T3: an array with a single clump of three elements in the middle (for example, [1,2,2,2,1])
• T4: an array with a single element

To check that for yourself, write down these three tests as (JUnit) automated test cases and run your favorite code coverage tool as in the following.

Listing 3.10 100% branch coverage for the clump-counting problem

@ParameterizedTest
@MethodSource("generator")
void testClumps(int[] nums, int expectedNoOfClumps) {
  assertThat(Clumps.countClumps(nums))
      .isEqualTo(expectedNoOfClumps);
}
 
static Stream<Arguments> generator() {     ❶
  return Stream.of(
    of(new int[]{}, 0), // empty
    of(null, 0), // null
    of(new int[]{1,2,2,2,1}, 1), // one clump
    of(new int[]{1}, 0) // one element
  );
}

❶ The four test cases we defined

This test suite is reasonable and exercises the main behavior of the program, but note how weak it is. It achieves 100% branch coverage, but it misses many interesting test cases. Even without performing systematic specification testing, in a program that counts clumps, it is natural to try the program with multiple clumps instead of just one. We could try it with the last clump happening at the last item of the array or with an array that has a clump starting in the first position. Such specific cases cannot be captured by pure structural testing guided mainly by coverage. This is yet another reason not to rely blindly on coverage. Structural testing shows its value when combined with knowledge of the specification.