Shutl

Incorporate property testing to improve confidence in your code

Note: All the examples and libraries use clojure, but the content is portable on any language of your choice.

Use Case

Imagine you need a function which computes the average of values. Using TDD you might write your code likewise:

(deftest average-unit-test
 (is (= 1 (average [1])))
 (is (= 10 (average [5 15 10]))))
(defn average [values]
 (/ (reduce + values) (count values)))

And sure enough, when you run your tests you can see

Ran 1 tests containing 2 assertions.
0 failures, 0 errors.

There are few comments on that simple piece of code. Why did we choose these 2 examples for testing? Where do the “magic numbers” come from. Is that enough testing? – spoiler alert, the answer is NO –

The most obvious bug in this code is “Division By Zero” — when the values is an empty collection. So after spotting it we can update the code to something like:

(deftest average-unit-test
 (is (= 1 (average [1])))
 (is (= 10 (average [5 15 10])))
 (is (nil? (average []))))
(defn average [values]
   (if-not (empty? values)
     (/ (reduce + values) (count values))
     nil))

Now we have fixed this obvious issue with this really simple function, we have to ask ourselves “are there any more edge cases in this function?”, “how many more tests should I add to make sure I’ve got everything covered?”. If you strictly follow TDD, we also have to find a failing test first.

Property testing to the rescue

A property testing library is trying the run the algorithm under test with random values, and check if its properties are satisfied. It’s a bit of a brute force attack.

Property

It’s an invariant of your algorithm. For example for a switching the case of every character of a string, it could be, applying twice the function on a string result on the original string. The property takes in parameter one or multiple generators and check the invariant on each generated value.

Generator

In order to check your algorithm with a lot of different combinations, you need to be able to generate random data. Every property testing library comes with a way to generator complex data structure, constraining the range of the values.

Reducing

The reduce phase is a really important part of the method. When the library detects a combination of values which does not satisfy the invariant, as the data is randomly generated it can contains a lot of “noise”.

So the reducing phase is there is try to find the minimum combination of values which triggers the bug. For example, reducing a list means trying to remove some values and check if the invariant is still failing.

Introducing test.check

test.check is an open source library inspired by QuickCheck. It allows to describe…….

so let’s add the dependency on the library:

[org.clojure/test.check "0.9.0"]

and require it from the test

(:require [clojure.test.check.clojure-test :refer [defspec]]
          [clojure.test.check :as tc]
          [clojure.test.check.generators :as gen]
          [clojure.test.check.properties :as prop])

To come back to your average function, we have to answer two questions:

  • What are the invariants
  • What’s the input

Generator

The input is quite trivial, it’s any vector of numbers, which can be translated in test.check by

Basically we are trying to generate a vector with a random number of large random integer.

We can test it using the Clojure REPL via the test.check sample function

Screen Shot 2016-07-12 at 12.25.53

Invariant

In term of invariants, we can for example state that the result of the function should not depend on the order of the values in the vector. In terms of code

(defn- order-independant? [values]
  (let [regular-average (average values) 
        shuffled-average (average (shuffle values))]
    (equals-within regular-average shuffled-average 0.1)))

This function checks the average value of a vector equals the average of the same vector but shuffled.

All together

(def vector-generator (gen/vector gen/large-integer))

(defn- order-independant? [values]
  (let [regular-average (average values) 
        shuffled-average (average (shuffle values))]
   (equals-within regular-average shuffled-average 0.1)))

(defspec the-average-of-a-list-does-not-depend-on-the-order-of-the-elements number-of-iterations
  (prop/for-all [values vector-generator]
    (order-independant? values)))

The defspec macro allow to generate a test.unit test, which is going to be executing with the unit tests.

When we run the tests, we come across the following issue:

[clojure.main main "main.java" 37]]}, :seed 1467906144689, :failing-size 76, :num-tests 77, :fail [[-8791245474 3169063838 -6055278567293294141 -8543319887 1612827468009 545 27784351334004543 -2885 -107647 -456743012890065 719911797491 7 -15141358 -408950911 -21398427889 -538944277814 24 770462 -7524750475275866215 15236063177479454 3107788752529]], :shrunk {:total-nodes-visited 1069, :depth 46, :result #error {
 :cause "integer overflow"
 :via
 [{:type java.lang.ArithmeticException
   :message "integer overflow"
   :at [clojure.lang.Numbers throwIntOverflow "Numbers.java" 1501]}]

So basically what’s happening is test.check generated a vector which caused an exception. When using large number, computing the sum of all the values triggers an overflow.

It might seem like an obvious problem, but that’s an edge case rarely covered straight away by an unit test. Solving this particularly problem is out of the scope of this post, but now you can decide if you won’t support high numbers, or if you need a better algorithm to limit this issue.

 

Property testing in your toolbox

So, hopefully, by now you are convinced that property testing is awesome and you should use it. You now have to think how many “traditional” tests you want to write, and when can you write properties.

One noticeable thing in the previous example, is that we never test the proper calculation in the spec. We only check the average does not depend on the order of the elements. That’s, I think, were property testing falls short, if you wanted to write a spec for the actual results with random values, you can’t use “magic numbers”, you almost need to write a second implementation of the calculation.

Your mileage might vary, but I think the two techniques are complimentary.

Further information

There is a lot of material you can find on this topic, in no particular order, a selection of videos, articles I found useful for me:

 

Happy testing

Discussion

2 responses to ‘Incorporate property testing to improve confidence in your code

  1. Interesting stuff. There’s a popular implementation in Python called Hypothesis which seems very reliable.

Leave a Reply

Your email address will not be published. Required fields are marked *