Lecture 21 QuickCheck

Joseph Haugh

University of New Mexico

The Problem with Unit Tests

Suppose we write a sorting function and want to test it.

The unit testing approach: write specific examples.

sort [3,1,2]   == [1,2,3]   -- test 1
sort []        == []        -- test 2
sort [1]       == [1]       -- test 3
sort [5,4,3,2] == [1,2,3,4] -- wrong expected value

Each test requires us to:

Choose an input we think is interesting
Compute the correct expected output by hand
Hope we have not made an error in either

The Problem with Unit Tests

The deeper issue: unit tests are examples, not specifications.

A test like sort [3,1,2] == [1,2,3] tells us the output for one particular input.

But what we really want to say is:

For all lists, sorting produces an ordered list that contains the same elements.

That is a property, a universally quantified law.

Properties as Specifications

A property is a Haskell function that returns Bool (or Property).

prop_SortOrdered :: [Int] -> Bool
prop_SortOrdered xs = ordered (sort xs)
  where
    ordered []       = True
    ordered [_]      = True
    ordered (x:y:zs) = x <= y && ordered (y:zs)

This says: for every list of Ints, sort produces an ordered result.

QuickCheck will generate hundreds of random lists and check that this holds for all of them.

QuickCheck

QuickCheck was introduced by Claessen and Hughes in 2000. The central idea: let the computer choose the test inputs.

import Test.QuickCheck

main :: IO ()
main = quickCheck prop_SortOrdered

OK, passed 100 tests.

QuickCheck generates 100 random lists of varying lengths and contents, runs our property on each one, and reports whether any failed.

The Reverse Laws

Let us start with a simpler example from the original paper: reverse.

What laws must reverse satisfy? From mathematics:

reverse [x]        = [x]
reverse (xs ++ ys) = reverse ys ++ reverse xs
reverse (reverse xs) = xs

The first two laws characterize reverse uniquely; the third follows from them.

Translating Laws to Properties

prop_RevSingle :: Int -> Bool
prop_RevSingle x =
    reverse [x] == [x]

prop_RevApp :: [Int] -> [Int] -> Bool
prop_RevApp xs ys =
    reverse (xs ++ ys) == reverse ys ++ reverse xs

prop_RevRev :: [Int] -> Bool
prop_RevRev xs =
    reverse (reverse xs) == xs

ghci> quickCheck prop_RevApp
OK, passed 100 tests.
ghci> quickCheck prop_RevRev
OK, passed 100 tests.

Catching a Bug Immediately

Suppose we accidentally write prop_RevApp backwards:

prop_RevApp_Wrong :: [Int] -> [Int] -> Bool
prop_RevApp_Wrong xs ys =
    reverse (xs ++ ys) == reverse xs ++ reverse ys

ghci> quickCheck prop_RevApp_Wrong
Falsifiable, after 1 tests:
[0]
[1,-1]

QuickCheck found a counterexample immediately: with xs = [0] and ys = [1,-1], the left side gives [1,-1,0] but the right gives [0,1,-1].

Conditional Properties

Some properties only hold under a precondition.

Consider insert, which inserts an element into an already-ordered list:

insert :: Ord a => a -> [a] -> [a]
insert x []     = [x]
insert x (y:ys)
    | x <= y    = x : y : ys
    | otherwise = y : insert x ys

The property: inserting into an ordered list yields an ordered list.

But this only makes sense when the input list is already ordered.

The ==> Operator

QuickCheck provides (==>) to express conditional properties:

ordered :: Ord a => [a] -> Bool
ordered []       = True
ordered [_]      = True
ordered (x:y:zs) = x <= y && ordered (y:zs)

prop_Insert :: Int -> [Int] -> Property
prop_Insert x xs =
    ordered xs ==> ordered (insert x xs)

If the precondition ordered xs is False, QuickCheck discards that test case and tries another.

The ==> Operator

ordered :: Ord a => [a] -> Bool
ordered []       = True
ordered [_]      = True
ordered (x:y:zs) = x <= y && ordered (y:zs)

prop_Insert :: Int -> [Int] -> Property
prop_Insert x xs =
    ordered xs ==> ordered (insert x xs)

ghci> quickCheck prop_Insert
OK, passed 100 tests.

Looks good! But there is a hidden problem.

A Hidden Problem: Data Distribution

How many of those 100 tests actually exercised a list with more than one element?

We can ask QuickCheck to report what it is testing with collect:

prop_Insert_Report :: Int -> [Int] -> Property
prop_Insert_Report x xs =
    ordered xs ==>
    collect (length xs) $
    ordered (insert x xs)

OK, passed 100 tests.
49% 0.
32% 1.
12% 2.
4%  3.
2%  4.
1%  5.

A Hidden Problem: Data Distribution

Only 19 out of 100 tests had a list with more than one element!

Why? The precondition ordered xs silently skews the distribution:

Every length-0 list is ordered (trivially)
Every length-1 list is ordered (trivially)
Only half of length-2 lists are ordered
Fewer and fewer longer lists pass the filter

We are mostly testing trivial cases: inserting into empty or singleton lists.

The Fix: Custom Generators

Instead of generating random lists and discarding the unordered ones, we should generate ordered lists directly:

orderedList :: (Arbitrary a, Ord a) => Gen [a]
orderedList = fmap sort arbitrary

We use forAll to supply a custom generator:

prop_Insert_Good :: Int -> Property
prop_Insert_Good x =
    forAll orderedList $ \xs ->
    ordered (insert x xs)

ghci> quickCheck prop_Insert_Good
OK, passed 100 tests.

Now every test case is a genuinely interesting ordered list.

The Arbitrary Typeclass

How does QuickCheck know how to generate random Int values, or random [Int] values?

class Arbitrary a where
    arbitrary :: Gen a

Gen a is a generator, a computation that produces random values of type a.

QuickCheck uses type inference to pick the right instance automatically. When it sees prop :: [Int] -> Bool, it knows it needs arbitrary :: Gen [Int].

Standard Arbitrary Instances

instance Arbitrary Bool where
    arbitrary = elements [True, False]

instance Arbitrary Int where
    arbitrary = sized (\n -> choose (-n, n))

elements picks uniformly from a list. sized gives access to a size parameter that QuickCheck grows across tests. choose generates a random value in a range.

Standard Arbitrary Instances

instance Arbitrary Bool where
    arbitrary = elements [True, False]

instance Arbitrary Int where
    arbitrary = sized (\n -> choose (-n, n))

instance (Arbitrary a, Arbitrary b) => Arbitrary (a, b) where
    arbitrary = do
        x <- arbitrary
        y <- arbitrary
        return (x, y)

instance Arbitrary a => Arbitrary [a] where
    arbitrary = sized $ \n -> do
        k <- choose (0, n)
        sequence (replicate k arbitrary)

Instances for pairs and lists are derived from instances for their element types.

Writing a Custom Arbitrary Instance

data Expr
    = Num Int
    | Add Expr Expr
    | Mul Expr Expr

The size parameter starts at 0 to force a base case, then grows with each recursive level, halved at each branch.

instance Arbitrary Expr where
    arbitrary = sized arbExpr
      where
        arbExpr 0 =
            fmap Num arbitrary
        arbExpr n = oneof
            [ fmap Num arbitrary
            , do l <- arbExpr (n `div` 2)
                 r <- arbExpr (n `div` 2)
                 return (Add l r)
            , do l <- arbExpr (n `div` 2)
                 r <- arbExpr (n `div` 2)
                 return (Mul l r)
            ]

Why sized?

Without sized, a recursive generator can fail to terminate.

Consider a naive attempt:

-- DANGER: may loop forever!
instance Arbitrary Expr where
    arbitrary = oneof
        [ fmap Num arbitrary
        , liftA2 Add arbitrary arbitrary
        , liftA2 Mul arbitrary arbitrary
        ]

oneof picks each branch with equal probability (1/3). But the two recursive branches each call arbitrary twice, so the expected number of nodes is infinite.

sized lets us shrink the bound at each recursive call, guaranteeing termination and controlling tree depth.

The Generator Combinators

Combinator	Meaning
`elements xs`	Pick uniformly from `xs`
`choose (lo, hi)`	Random value in range
`oneof gens`	Pick uniformly from generators
`frequency [(w,g)]`	Pick generator `g` with weight `w`
`sized f`	Pass size bound to `f`
`forAll gen prop`	Use custom generator `gen`

Frequency

frequency is useful when you want one branch to be picked more or less often than others:

arbitrary = frequency
    [ (1, return [])                         -- 1/5 chance of []
    , (4, liftA2 (:) arbitrary arbitrary) ]  -- 4/5 chance of non-empty

Shrinking: Finding Minimal Failures

When QuickCheck finds a failing case, the counterexample may be large and hard to understand. Shrinking automatically finds a smaller failing case.

The Arbitrary typeclass has an optional shrink method:

class Arbitrary a where
    arbitrary :: Gen a
    shrink :: a -> [a]
    shrink _ = []     -- default: no shrinking

shrink x should return a list of values that are “smaller” than x in some meaningful sense.

Shrinking: Lists and Integers

For lists, shrinking means: remove elements, or shrink individual elements.

instance Arbitrary a => Arbitrary [a] where
    shrink []     = []
    shrink (x:xs) = 
        xs : map (:xs) (shrink x)
           ++ map (x:) (shrink xs)

For integers, shrinking means: move toward zero.

instance Arbitrary Int where
    shrink n | n > 0  = [n-1, n `div` 2]
             | n < 0  = [n+1, n `div` 2]
             | otherwise = []

Shrinking in Action

Suppose we test whether all lists are palindromes:

prop_Palindrome :: [Int] -> Bool
prop_Palindrome xs = reverse xs == xs

Without shrinking, QuickCheck might report:

Falsifiable, after 3 tests:
[-3, 0, 2, 5, 1, -2]

With shrinking, it reports:

Falsifiable, after 3 tests and 8 shrinks:
[0, 1]

The minimal counterexample: the shortest non-palindrome with the smallest possible values. Much easier to understand and reason about.

Shrinking: The Process

When a property fails on input x:

Compute shrink x, the list of “smaller” candidates
Test the property on each candidate
If any candidate also fails, make it the new counterexample and repeat
If none fail, report the current counterexample

This is a greedy search: QuickCheck keeps shrinking as long as it can find a smaller failing case. The result is a locally minimal counterexample: every element of it is necessary to trigger the bug.

Properties as Executable Specifications

QuickCheck blurs the line between tests and documentation.

A property like:

prop_SortCorrect :: [Int] -> Bool
prop_SortCorrect xs =
    let ys = sort xs
    in  ordered ys && sameElements xs ys

is simultaneously:

A machine-checkable test run on hundreds of inputs
A precise mathematical statement of what sort must do
Documentation that lives in the same module as the code

Three Kinds of Bugs

The QuickCheck paper reports a striking finding: when testing real programs, bugs found are divided roughly evenly into three categories.

Bugs in the generator: the test data does not cover the cases you think it does (as we saw with ordered xs ==>)
Bugs in the specification: the property itself is wrong; you had a mistaken belief about what the function should do
Bugs in the implementation: the actual code is wrong

All three are valuable. Finding a specification bug reveals a misunderstanding, even if the code happens to be correct.

Case Study: A Theorem Prover

Claessen and Hughes tested two propositional theorem provers (Davis-Putnam and Stalmarck) against each other with:

prop_Agree :: Formula -> Property
prop_Agree f =
    davisPutnam f == stalmarck f

QuickCheck found three bugs, all due to implicit assumptions:

Both algorithms assumed no clause contained the same literal twice
Stalmarck assumed no empty clauses in the input
Neither assumption was written down anywhere

Hand-written tests would likely never have generated these edge cases.

Unit Tests vs Properties: A Comparison

Unit Tests

sort [3,1,2] == [1,2,3]
sort []       == []
sort [5]      == [5]
sort [2,2]    == [2,2]

Specific inputs
Manual expected values
Easy to forget edge cases
Scales poorly

Properties

prop_Sorted xs =
  ordered (sort xs)

prop_Stable xs =
  sameElements xs (sort xs)

All inputs (randomly sampled)
No expected value needed
Edge cases found automatically
Shrinking pinpoints failures

When Unit Tests Are Still Useful

Properties and unit tests are complementary, not opposed.

Use properties to express universal laws (correctness conditions)
Use unit tests (or golden tests) when the expected output is complex and hard to characterize, e.g., a specific parse tree or rendered output
Use both: properties catch classes of bugs, unit tests catch regressions

QuickCheck in Practice

import Test.QuickCheck

-- Basic property
prop_RevRev :: [Int] -> Bool
prop_RevRev xs = reverse (reverse xs) == xs

-- Conditional property
prop_Insert :: Int -> [Int] -> Property
prop_Insert x xs =
    ordered xs ==> ordered (insert x xs)

-- Custom generator
prop_InsertGood :: Int -> Property
prop_InsertGood x =
    forAll orderedList $ \xs ->
    ordered (insert x xs)

main :: IO ()
main = do
    quickCheck prop_RevRev
    quickCheck prop_Insert
    quickCheck prop_InsertGood

Summary

Unit tests are examples; properties are universal laws
QuickCheck generates hundreds of random inputs and checks that a property holds for all of them
(==>) expresses conditional properties, but beware of skewed distributions; use collect to inspect test data
forAll with a custom generator solves the distribution problem
The Arbitrary typeclass uses type inference to select generators; sized controls the depth of recursive structures
Shrinking automatically finds the minimal failing counterexample, making bugs much easier to understand
Properties find bugs in generators, specifications, and implementations (all three kinds are valuable)