Common issues in property-based tests

I see three common issues in the property-based tests my students write when they learn this technique. The first is requiring jqwik to generate data that is very expensive or even impossible. If you ask jqwik to, say, generate an array of 100 elements in which the numbers have to be unique and multiples of 2, 3, 5, and 15, such an array can be difficult to find, given jqwik’s random approach. Or if you want an array with 10 unique elements, but you give jqwik a range of 2 to 8, the array is impossible to generate. In general, if jqwik is taking too long to generate the data for you, maybe you can find a better way to generate the data or write the test.

Second, we saw in previous chapters that boundaries are a perfect place for bugs. So, we want to exercise those boundaries when writing property-based tests. Ensure that you are expressing the boundaries of the property correctly. When we wrote the tests for the passing-grade problem (section 5.1), we wrote arbitraries like Arbitraries .floats().lessThan(1f) and Arbitraries.floats().greaterThan(10f). Jqwik will do its best to generate boundary values: for example, the closest possible number to 1f or the smallest possible float. The default configuration for jqwik is to mix edge cases with random data points. Again, all of this will work well only if you express the properties and boundaries correctly.

The third caveat is ensuring that the input data you pass to the method under test is fairly distributed among all the possible options. Jqwik does its best to generate well-distributed inputs. For example, if you ask for an integer between 0 and 10, all the numbers in the interval will have the same probability of being generated. But I have seen tests that manipulate the generated data and then harm this property. For example, imagine testing a method that receives three integers, a, b, and c, and returns a boolean indicating whether these three sides can form a triangle. The implementation of this method is simple, as shown in the following listing.

Listing 5.23 Implementation of the isTriangle method

public class Triangle {
  public static boolean isTriangle(int a, int b, int c) {
    boolean hasABadSide = a >= (b + c) || c >= (b + a) || b >= (a + c);
    return !hasABadSide;
  }
}

To write a property-based test for this method, we need to express two properties: valid triangles and invalid triangles. If the developer generates three random integer values as shown next, there is a very low chance of them forming a valid triangle.

Listing 5.24 A bad property-based test for isTriangle

@Property
void triangleBadTest(                  ❶
  @ForAll @IntRange(max = 100) int a,
  @ForAll @IntRange(max = 100) int b,
  @ForAll @IntRange(max = 100) int c) {
 
   // ... test here ...
 
}

❶ Generates three different integers. The odds are that these a, b, and c will be an invalid triangle. We therefore do not exercise the valid triangle property as much as we wanted to.

The test exercises the invalid triangle property more than the valid triangle property. A good property-based test for this problem would ensure that jqwik generates the same number of valid and invalid triangles. The easiest way to do that would be to split it into two tests: one for valid triangles and one for invalid triangles. (The solution is available in the code repository.)