Code coverage, the right way

Consider the following requirement for a small program that counts the number of words in a string that end with either “r” or “s” (inspired by a CodingBat problem.

Given a sentence, the program should count the number of words that end with either “s” or “r”. A word ends when a non-letter appears. The program returns the number of words.

A developer implements this requirement as shown in the following listing.

Listing 3.1 Implementing the CountWords program

public class CountWords {
  public int count(String str) {
    int words = 0;
    char last = ' ';
 
    for (int i = 0; i < str.length(); i++) {   ❶
 
      if (!isLetter(str.charAt(i)) &&          ❷
       (last == 's' || last == 'r')) {
          words++;
      }
 
      last = str.charAt(i);                    ❸
    }
 
    if (last == 'r' || last == 's') {          ❹
      words++;
    }
 
    return words;
  }
}

❶ Loops through each character in the string

❷ If the current character is a non-letter and the previous character was “s” or “r”, we have a word!

❸ Stores the current character as the “last” one

❹ Counts one more word if the string ends in “r” or “s”

Now, consider a developer who does not know much about specification-based testing techniques and writes the following two JUnit tests for the implementation.

Listing 3.2 Initial (incomplete) tests for CountWords

@Test
void twoWordsEndingWithS() {                            ❶
  int words = new CountLetters().count("dogs cats");
  assertThat(words).isEqualTo(2);
}
 
@Test
void noWordsAtAll() {                                   ❷
  int words = new CountLetters().count("dog cat");
  assertThat(words).isEqualTo(0);
}

❶ Two words ending in “s” (dogs and cats): we expect the program to return 2.

❷ No words ending in “s” or “r” in the string: the program returns 0.

This test suite is far from complete—for example, it does not exercise words ending in “r”. Structural testing shows its value in such situations: we can identify parts of the test code that our test suite does not exercise, determine why this is the case, and create new test cases.

Identifying which parts of the code our tests exercise is straightforward today, thanks to the many production-ready code coverage tools on the market for all programming languages and environments. For example, figure 3.1 shows the report generated by JaCoCo a very popular code coverage tool for Java, after running the two tests in listing 3.2.

Figure 3.1 Code coverage achieved by the two tests in the CountWords implementation. The two if lines are only partially covered.

The background color of each line indicates its coverage.

• A green background indicates that a line is completely covered by the test suite. In the figure, all lines with the exception of the two ifs are green.
• A yellow background means the line is partially covered by the test suite. For example, in the figure, the two if statement lines are only partially covered.
• A red background means the line is not covered. In the figure, there are no red lines, which means all lines are exercised by at least one test.
• Lines with no background color (such as }) are lines the coverage tool does not see. Behind the scenes, coverage tools are instrumenting the compiled bytecode of the program. Things like closing brackets and method declaration lines are not really counted.

JaCoCo also uses a diamond to identify a line that may branch the program, including the for and if statements in figure 3.1, as well as while, for, do-while, ternary ifs, lambda expressions, and so on. Hovering your mouse over the diamond shows the details.

As previously mentioned, the first if statement has a yellow background, indicating that although the line is covered, not all of its branches are. When I look at the details of the report, the tool says that one out of six combinations (three conditions in the if statement times two options, true and false) is not covered. See figure 3.2.

Figure 3.2 JaCoCo shows how many branches we are missing.

The current test suite does not fully exercise the last == 'r' condition. This is useful information; thanks to structural testing, the tester can now figure out why this test case did not emerge before.

Reasons to miss a test case

Here are some pragmatic reasons a developer may miss a test case:

• The developer made a mistake. The specification was clear about the requirement.
• The specification did not mention the case, and it is unclear whether the behavior is expected. The developer must decide whether to bring it to the requirements engineer. Is it a mistake in the implementation?
• The specification did not mention the case, but the code has a reason to exist. For example, implementation details such as performance and persistence often force developers to write code that is not reflected in the (functional) requirement. The developer should add a new test to the test suite, which will exercise the implementation-specific behavior that may cause bugs.

Moving on with the example, we write a test case that exercises the “words that end in ‘r’” partition as follows.

Listing 3.3 Testing for words that end in “r”

@Test
void wordsThatEndInR() {                          ❶
  int words = new CountWords().count("car bar");
  assertThat(words).isEqualTo(2);
}

❶ Words that end in “r” should be counted.

With the newly added test case in the test suite, we rerun the coverage tool. Figure 3.3 shows the new JaCoCo report. Every line is now fully covered: we have covered all the lines and conditions of the code under test. If parts of the code were still not covered, we would repeat the process: identify uncovered parts, understand why they are not covered, and write a test that exercises that piece of code.

Figure 3.3 Code coverage of the three tests for the CountWords implementation. The test suite now achieves full coverage of branches and conditions.