After reading the great blog article from Johannes Link about property testing in Kotlin, I was inclined trying it out. Having written property tests with kotest before, I was curious what I was missing and how much I would like a specialized property testing framework.
What is property testing all about?
As a software crafter I am a big fan of testing everything, preferably using TDD. While most tests that I write use concrete examples (so called example tests), I think it is useful also to write tests that describe properties of some part of the software that holds true for any input (so called property tests). Especially for algorithmic parts of my code.
kotest is a general purpose testing framework for Kotlin. Besides property tests it also covers example and data-driven tests. It is multi platform supporting all flavors of Kotlin (JVM, JS, Native). It has also a very convenient and idiomatic assertions library (which I use btw also for my experiments in jqwik).
jqwik is a specialized framework for writing property tests on the JVM platform, including Kotlin. It is written in Java, but has also a Kotlin module providing many convenience functions. It should come with no surprise that jqwik is more sophisticated and has more features than kotest for property testing (for instance assumptions, state-based testing, recursive generators, statistics, type-based generation, exhaustive generation, edge cases, better shrinkers).
Show case
Imagine you have a list that describes the change of your bank account balance over time.
val list = listOf(
Balance(LocalDate.parse("2022-05-01"), 100.toBigDecimal()),
Balance(LocalDate.parse("2022-05-04"), 200.toBigDecimal()),
Balance(LocalDate.parse("2022-05-08"), 50.toBigDecimal()),
)
Code language: Kotlin (kotlin)That’s great, but what if you want to fill the gaps in between? I need something like that in my current project to display the changes over time in a chart.
val expected = listOf(
Balance(LocalDate.parse("2022-05-01"), 100.toBigDecimal()),
Balance(LocalDate.parse("2022-05-02"), 100.toBigDecimal()),
Balance(LocalDate.parse("2022-05-03"), 100.toBigDecimal()),
Balance(LocalDate.parse("2022-05-04"), 200.toBigDecimal()),
Balance(LocalDate.parse("2022-05-05"), 200.toBigDecimal()),
Balance(LocalDate.parse("2022-05-06"), 200.toBigDecimal()),
Balance(LocalDate.parse("2022-05-07"), 200.toBigDecimal()),
Balance(LocalDate.parse("2022-05-08"), 50.toBigDecimal()),
)
Code language: JavaScript (javascript)I wrote a simple function expandToDaily that transforms the list above to the expected output. I won’t bother you with the implementation details, you can look it up in my GitHub learning repo. The interesting part are the properties of this function:
- The output should be strictly increasing by date
- The output should include all items present in the input
- The output should cover the whole range between the minimum and maximum of the input’s range
There might be additional properties, I am by no means an expert in property testing. Here are some helpful patterns for finding properties. I usually cover the rest using example tests.
Property Tests with kotest
class DailyBalancesPropKotestTest : DescribeSpec({
describe("properties") {
val arbBalance = Arb.bind(Arb.localDate(), Arb.bigDecimal()) { d, a ->
Balance(d, a)
}
fun arbUniqueList(minLength: Int = 0) = Arb
.list(arbBalance, minLength..100)
.map { it.distinctBy { it.date } }
it("contains all original elements") {
checkAll(arbUniqueList()) { list ->
val result = list.expandToDaily()
result.shouldContainAll(list)
}
}
it("result is strictly increasing by the balance date") {
checkAll(arbUniqueList()) { list ->
val result = list.expandToDaily()
result.shouldBeStrictlyIncreasingWith(compareBy { it.date })
}
}
it("returns the whole range") {
checkAll(arbUniqueList(minLength = 1)) { list ->
val result = list.expandToDaily()
result.map { it.date } shouldBe wholeDateRange(
result.minOf { it.date },
result.maxOf { it.date }
)
}
}
}
})
Code language: Kotlin (kotlin)I used the typical BDD describe/it-style popularized by JS test frameworks like jest for structuring my tests. kotest also provides several other styles. The downside is that the tests won’t be separately executable from the IDE without the IntelliJ plugin.
Lines 3 to 8 are the interesting part and are used for creating generators for arbitrary values used in the property tests. Line 3 constructs a generator for balances. Line 8 uses this generator to construct an arbitrary list of balances. kotest’s checkAll function acts as the entrypoint into property testing. It runs the test 10001 times with different values, including many edge cases (such as an empty list). In case of an error the seed is displayed that can be used to re-run the test deterministically (using the same input). You can use kotest’s PropTestConfig() class to set the seed and pass it to the test in question.
checkAll(PropTestConfig(seed = -12234354253), yourArb) { input ->
...
}Code language: Kotlin (kotlin)There is an easier way to construct the balance generator on the JVM using reflection:
val arbBalance = Arb.bind<Balance>()Code language: HTML, XML (xml)The downside besides only working on the JVM is that you lose control of the generators for its properties.
An interesting and easy way to construct custom generators is the arbitrary { ... } syntax2.
val arbBalance = arbitrary {
val date = Arb.localDate().bind()
val amount = Arb.bigDecimal().bind()
Balance(date, amount)
}Code language: Kotlin (kotlin)This might look like an overkill here, but is really useful for more complex generators (such as a date range) without being forced to use flatMap chains.
// pseudo-imperative
val arbNonEmptyRange = arbitrary {
val start = Arb.localDate().bind()
val minDate = start.plusDays(1)
val maxDate = maxOf(LocalDate.of(2030, 12, 31), minDate.plusDays(1))
val end = Arb.localDate(minDate = minDate, maxDate = maxDate).bind()
start..end
}
// monadic flat mapping
val arbNonEmptyRangeFlat = Arb
.localDate()
.flatMap { start ->
val minDate = start.plusDays(1)
val maxDate = maxOf(LocalDate.of(2030, 12, 31), minDate.plusDays(1))
val end = Arb.localDate(minDate = minDate, maxDate = maxDate)
end.map { start..it }
}Code language: Kotlin (kotlin)I like that all generators in kotest are companion functions of the Arb class, so they are easily discoverable via the IDE.
As a side note, not all standard generators contain shrinkers, which are very helpful with providing a simpler sample in case of an error. And the provided shrinkers are less powerful than those in jqwik.
Property Tests with jqwik
class DailyBalancesPropJqwikTest {
@Property
fun testArbitrary(@ForAll("uniqueBalances") list: List<Balance>) {
Statistics.label("sizes").collect(list.size)
}
@Property
fun `contains all original elements`(
@ForAll("uniqueBalances") list: List<Balance>) {
val result = list.expandToDaily()
result.shouldContainAll(list)
}
@Property
fun `result is strictly increasing by the balance date`(
@ForAll("uniqueBalances") list: List<Balance>) {
val result = list.expandToDaily()
result.shouldBeStrictlyIncreasingWith(compareBy { it.date })
}
@Property
fun `returns the whole range`(
@ForAll("uniqueBalances") @NotEmpty list: List<Balance>) {
val result = list.expandToDaily()
result.map { it.date } shouldBe wholeDateRange(
result.minOf { it.date },
result.maxOf { it.date }
)
}
@Provide
fun uniqueBalances(): Arbitrary<List<Balance>> {
val balanceArbitrary = combine(
Dates.dates()
.between(LocalDate.of(1970, 1, 1), LocalDate.of(2030, 12, 31)),
Arbitraries.bigDecimals()
) { d, a -> Balance(d, a) }
return balanceArbitrary
.list()
.uniqueElements { it.date }
.ofMaxSize(100)
}
}
Code language: Kotlin (kotlin)You write your tests basically using the usual JUnit style except you annotate your tests with @Property instead of @Test. In fact, the tests are executable in the IDE without any additional plugins! Custom generators are referenced as a string in the @ForAll annotation which corresponds to a method annotated with @Provide that provides the generator. Unfortunately, the IDE won’t warn me when the reference does not match a method name, but thankfully, jqwik has really great error messages.
net.jqwik.api.CannotFindArbitraryException:
Cannot find an Arbitrary [uniqueBalances1] for Parameter of type [@net.jqwik.api.ForAll(value="uniqueBalances1", supplier=net.jqwik.api.ArbitrarySupplier$NONE.class) @net.jqwik.api.constraints.NotEmpty() @net.jqwik.api.constraints.Size(value=0, max=0, min=1) @net.jqwik.api.constraints.StringLength(value=0, max=0, min=1) List<Balance>]
I specified the date range explicitly to match the default range in kotest to make a fair performance comparison.
jqwik has many helpful constraints like the @NotEmpty annotation or the .uniqueElements() builder method. This makes building custom generators in my opinion easier than in kotest where I have to use map/filter/flatMap more often (in my example above I used the map function to force the list to contain only unique elements).
The combine(…) syntax is identical to kotest’s Arb.bind(…)3. As in kotest, using reflection to build a generator is also possible (with the same downsides). You can even specify which constructors or factories you want to use!
val balanceArbitrary = anyForType<Balance>()
Code language: Kotlin (kotlin)An interesting choice is the inclusion of the .list() builder method that makes the construction of arbitrary lists containing arbitrary elements more straight forward.
jqwik is using builders with fluent interface methods to construct its generators. kotest is using methods with optional parameters which is in my eyes the idiomatic way doing it in Kotlin. But I can understand that a framework not limited to Kotlin like jqwik has to do it the usual Java way.
What I dislike is that the arbitraries are scattered across different types such as Dates or Arbitraries. The jqwik Kotlin module adds also some convenience methods such as any() extension methods on standard types such as Int or String4. I’d prefer the kotest way of a single entrypoint for all generators though.
The test is line 3 is very interesting. It serves the purpose of inspecting my custom generator, for instance if its content or statistical distribution of values is satifying. Sadly, there is nothing comparable in kotest. There are some basic classification features not documented yet. I guess I have to wait till a later version ?.
It is very convenient to re-run failed property tests as jqwik automatically keeps track of failed tests and the seed last used. This behaviour is even configurable.
INFORMATION: After Failure Handling: PREVIOUS_SEED, Previous Generation: <GenerationInfo(-4596258530446308373, 1, [size=1])>
Due to its heavy usage of Java reflection, jqwik is unfortunately currently bound to the JVM and offers no multiplatform support.
Conclusion
Both kotest and jqwik provide everything necessary to start with property testing. I noticed no performance differences when executing my property tests. Both kotest & jqwik are more than sufficient to implement my show case.
The main advantage of kotest is its multiplatform support. Being part of a bigger package you are probably already using with Kotlin 5 makes it also very accessible in case you want to start with property testing. The biggest disadvantage is its smaller & less sophisticated feature set.
On the other hand, jqwik is the way to go if you want the best experience with property testing. While it is more idiomatically leaned towards Java, it works really great for Kotlin as well. At least on the JVM as it does not support multiplatform6.
For now, kotest is good enough for my use cases, but I will keep taking a look at jqwik!
No matter which “poison” you choose, I wholeheartedly recommend Johannes’ blog series about property testing, which concepts are not limited to a specific implementation!
- which is configurable
- which is by the way a monadic comprehension, while the initial bind syntax is an applicative functor in disguise, for you FP folks out there
- though zip would be a better name in both frameworks, as it is more in line with Kotlin’s standard library
- but not on LocalDate unfortunately ?
- if not, I’d strongly recommend you take a look at it ?
- which is its biggest disadvantage