Consider the following requirement, inspired by a similar problem in Kaner et al.’s book (2013):
A student passes an exam if they get a grade >= 5.0. Grades below that are a fail. Grades fall in the range [1.0, 10.0].
A simple implementation for this program is shown in the following listing.
Listing 5.1 Implementation of the PassingGrade program
public class PassingGrade {
public boolean passed(float grade) {
if (grade < 1.0 || grade > 10.0) ❶
throw new IllegalArgumentException();
return grade >= 5.0;
}
}❶ Note the pre-condition check here.
If we were to apply specification-based testing to this program, we would probably devise partitions such as “passing grade,” “failing grade,” and “grades outside the range.” We would then devise a single test case per partition. With property-based testing, we want to formulate properties that the program should have. I see the following properties for this requirement:
fail—For all numbers ranging from 1.0 (inclusive) to 5.0 (exclusive), the program should returnfalse.pass—For all numbers ranging from 5.0 (inclusive) to 10.0 (inclusive), the program should returntrue.invalid—For all invalid grades (which we define as any number below 1.0 or greater than 10.0), the program must throw an exception.
Can you see the difference between what we do in specification-based testing and what we aim to do in property-based testing? Let’s write a suite test by test, starting with the fail property. For that, we will use jqwik, a popular property-based testing framework for Java.
NOTE Property-based testing frameworks are available in many different languages, although their APIs vary significantly (unlike unit testing frameworks like JUnit, which all look similar). If you are applying this knowledge to another language, your task is to study the framework that is available in your programming language. The way to think and reason is the same across different languages.
Before I show the concrete implementation, let me break down property-based testing step by step, using jqwik’s lingo:
- For each property we want to express, we create a method and annotate it with
@Property. These methods look like JUnit tests, but instead of containing a single example, they contain an overall property. - Properties use randomly generated data. Jqwik includes several generators for various types (including
Strings,Integers,Lists,Dates, and so on.). Jqwik allows you to define different sets of constraints and restrictions to these parameters: for example, to generate only positiveIntegers or onlyStrings with a length between 5 and 10 characters. Thepropertymethod receives all the required data for that test as parameters. - The
propertymethod calls the method under test and asserts that the method’s behavior is correct. - When the test runs, jqwik generates a large amount of random data (following the characteristics you defined) and calls the test for it, looking for an input that would break the property. If jqwik finds an input that makes your test fail, the tool reports this input back to the developer. The developer then has an example of an input that breaks their program.
The following listing shows the code for the fail property.
Listing 5.2 A property-based test for the fail property
public class PassingGradesPBTest {
private final PassingGrade pg = new PassingGrade();
@Property
void fail( ❶
@ForAll ❷
@FloatRange(min = 1f, max = 5f, maxIncluded = false) ❸
float grade) { ❹
assertThat(pg.passed(grade)).isFalse();
}
}❶ Defines the characteristics of the values we want to generate via annotations
❷ Any parameter to be generated by jqwik must be annotated with ForAll.
❸ We want random floats in a [1.0, 5.0] interval (max value excluded), which we define in the FloatRange annotation.
❹ The grade parameter will be generated according to the rules specified in the annotations.
We annotate the test method with @Property instead of @Test. The test method receives a grade parameter that jqwik will set, following the rules we give it. We then annotate the grade parameter with two properties. First, we say that this property should hold for all (@ForAll) grades. This is jqwik’s terminology. If we left only the @ForAll annotation, jqwik would try any possible float as input. However, for this fail property, we want numbers varying from 1.0 to 5.0, which we specify using the @FloatRange annotation. The test then asserts that the program returns false for all the provided grades.
When we run the test, jqwik randomly provides values for the grade parameter, following the ranges we specified. With its default configuration, jqwik randomly generates 1,000 different inputs for this method. If this is your first time with property-based testing, I suggest that you write some print statements in the body of the test method to see the values generated by the framework. Note how random they are and how much they vary.
Correspondingly, we can test the pass property using a similar strategy, as shown next.
Listing 5.3 A property-based test for the pass property
@Property
void pass(
@ForAll
@FloatRange(min = 5f, max = 10f, maxIncluded = true) ❶
float grade) {
assertThat(pg.passed(grade)).isTrue();
}❶ We want random floats in the range of [5.0, 10.0], max value included.
Finally, to make jqwik generate numbers that are outside of the valid range of grades, we need to use a smarter generator (as FloatRange does not allow us to express things like “grade < 1.0 or grade > 10.0”). See the invalidGrades() provider method in the following listing: methods annotated with @Provide are used to express more complex inputs that need to be generated.
Listing 5.4 A property-based test for the invalidGrades property
@Property
void invalid(
@ForAll("invalidGrades") ❶
float grade) {
assertThatThrownBy(() -> {
pg.passed(grade);
}).isInstanceOf(IllegalArgumentException.class); ❷
}
@Provide ❸
private Arbitrary<Float> invalidGrades() {
return Arbitraries.oneOf( ❹
Arbitraries.floats().lessThan(1f), ❺
Arbitraries.floats().greaterThan(10f) ❻
);
}❶ The @ForAll annotation receives the name of a Provider method that will generate the data.
❷ Asserts that an exception is thrown for any value outside the boundary
❸ A provider method needs to be annotated with @Provide.
❹ Makes the method randomly return …
❺ … a float that is less than 1.0 …
The @Property test method is straightforward: for all grades generated, we assert that an exception is thrown. The challenge is generating random grades. We express this in the invalidGrades provider method, which should return either a grade smaller than 1 or a grade greater than 10. Also, note that the method returns an Arbitrary. An Arbitrary is how jqwik handles arbitrary values that need to be generated. If you need, say, arbitrary floats, your provider method should return an Arbitrary<Float>.
To give the two options to jqwik, we use the Arbitraries.oneOf() method. The Arbitraries class contains dozens of methods that help build arbitrary data. The oneOf() method receives a list of arbitrary values it may return. Behind the scenes, this method ensures that the distribution of data points generated is fairly distributed: for example, it generates as many “smaller than 1” inputs as “greater than 10” inputs. Then, we use another helper, the Arbitraries.floats() method, to generate random floats. Finally, we use the lessThan() and greaterThan() methods to generate numbers less than 1 and greater than 10, respectively.
NOTE I suggest exploring the methods that the Arbitraries class provides! Jqwik is a very extensive framework and contains lots of methods to help you build any property you need. I will not discuss every feature of the framework, as that would be an entire book by itself. Instead, I recommend that you dive into jqwik’s excellent.
When we run the tests, all of them pass, since our implementation is correct. Now, let’s introduce a bug in the code to see the jqwik output. For example, let’s change return grade >= 5.0 to return grade > 5.0, a simple off-by-one mistake. When we run our test suite again, the pass property test fails as expected! Jqwik also produces nice output to help us debug the problem.
Listing 5.5 An example of a jqwik test failure
|-------------------jqwik-------------------
tries = 11 | \# of calls to property
checks = 11 | \# of not rejected calls
generation = RANDOMIZED | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
when-fixed-seed = ALLOW | fixing the random seed is allowed
edge-cases\#mode = MIXIN | edge cases are mixed in
edge-cases\#total = 2 | \# of all combined edge cases
edge-cases\#tried = 1 | \# of edge cases tried in current run
seed = 7015333710778187630 | random seed to reproduce generated values
Sample
------
arg0: 5.0The output shows that jqwik found a counterexample in attempt number 11. Only 11 trials were enough to find the bug! Jqwik then shows a set of configuration parameters that may be useful when reproducing and debugging more complex cases. In particular, note the seed information: you can reuse that seed later to force jqwik to come up with the same sequence of inputs. Below the configuration, we see the sample that caused the bug: the value 5.0, as expected.
NOTE You may be wondering about boundary testing. Jqwik is smart enough to also generate boundary values! If we ask jqwik to generate, say, floats smaller than 1.0, jqwik will generate 1.0 as a test. If we ask jqwik to generate any integer, jqwik will try the maximum and minimum possible integers as well as 0 and negative numbers.
Leave a Reply