An example¶

Suppose we've written a run length encoding system and we want to test it out.

We have the following code which I took straight from the Rosetta Code wiki (OK, I removed some commented out code and fixed the formatting, but there are no functional modifications):

def encode(input_string):


    count = 1


    prev = ''


    lst = []


    for character in input_string:


        if character != prev:


            if prev:


                entry = (prev, count)


                lst.append(entry)


            count = 1


            prev = character


        else:


            count += 1


    else:


        entry = (character, count)


        lst.append(entry)


    return lst
def decode(lst):


    q = ''


    for character, count in lst:


        q += character * count


    return q

We want to write a test for this that will check some invariant of these functions.

The invariant one tends to try when you've got this sort of encoding / decoding is that if you encode something and then decode it then you get the same value back.

Lets see how you'd do that with Hypothesis:

from hypothesis import given
from hypothesis.strategies import text
@given(text())
def test_decode_inverts_encode(s):


    assert decode(encode(s)) == s

(For this example we'll just let pytest discover and run the test. We'll cover other ways you could have run it later).

The text function returns what Hypothesis calls a search strategy. An object with methods that describe how to generate and simplify certain kinds of values. The @given decorator then takes our test function and turns it into a parametrized one which, when called, will run the test function over a wide range of matching data from that strategy.

Anyway, this test immediately finds a bug in the code:

Falsifying example: test_decode_inverts_encode(s='')
UnboundLocalError: local variable 'character' referenced before assignment

Hypothesis correctly points out that this code is simply wrong if called on an empty string.

If we fix that by just adding the following code to the beginning of the function then Hypothesis tells us the code is correct (by doing nothing as you'd expect a passing test to).

if not input_string:


    return []

If we wanted to make sure this example was always checked we could add it in explicitly:

from hypothesis import given, example
from hypothesis.strategies import text
@given(text())
@example('')
def test_decode_inverts_encode(s):


    assert decode(encode(s)) == s

You don't have to do this, but it can be useful both for clarity purposes and for reliably hitting hard to find examples. Also in local development Hypothesis will just remember and reuse the examples anyway, but there's not currently a very good workflow for sharing those in your CI.

It's also worth noting that both example and given support keyword arguments as well as positional. The following would have worked just as well:

@given(s=text())
@example(s='')
def test_decode_inverts_encode(s):


    assert decode(encode(s)) == s

Suppose we had a more interesting bug and forgot to reset the count each time. Say we missed a line in our encode method:

def encode(input_string):


  count = 1


  prev = ''


  lst = []


  for character in input_string:


      if character != prev:


          if prev:


              entry = (prev, count)


              lst.append(entry)


          # count = 1  # Missing reset operation


          prev = character


      else:


          count += 1


  else:


      entry = (character, count)


      lst.append(entry)


  return lst

Hypothesis quickly informs us of the following example:

Falsifying example: test_decode_inverts_encode(s='001')

Note that the example provided is really quite simple. Hypothesis doesn't just find any counter-example to your tests, it knows how to simplify the examples it finds to produce small easy to understand ones. In this case, two identical values are enough to set the count to a number different from one, followed by another distinct value which should have reset the count but in this case didn't.

The examples Hypothesis provides are valid Python code you can run. Any arguments that you explicitly provide when calling the function are not generated by Hypothesis, and if you explicitly provide all the arguments Hypothesis will just call the underlying function the once rather than running it multiple times.

Installing¶

Hypothesis is available on pypi as "hypothesis". You can install it with:

pip install hypothesis

If you want to install directly from the source code (e.g. because you want to make changes and install the changed version) you can do this with:

pip install -e .

You should probably run the tests first to make sure nothing is broken. You can do this with:

python setup.py test

Note that if they're not already installed this will try to install the test dependencies.

You may wish to do all of this in a virtualenv. For example:

virtualenv venv
source venv/bin/activate
pip install hypothesis

Will create an isolated environment for you to try hypothesis out in without affecting your system installed packages.

Running tests¶

In our example above we just let pytest discover and run our tests, but we could also have run it explicitly ourselves:

if __name__ == '__main__':


    test_decode_inverts_encode()

We could also have done this as a unittest TestCase:

import unittest
class TestEncoding(unittest.TestCase):


    @given(text())


    def test_decode_inverts_encode(self, s):


        self.assertEqual(decode(encode(s)), s)
if __name__ == '__main__':


    unittest.main()

A detail: This works because Hypothesis ignores any arguments it hasn't been told to provide (positional arguments start from the right), so the self argument to the test is simply ignored and works as normal. This also means that Hypothesis will play nicely with other ways of parameterizing tests. e.g it works fine if you use pytest fixtures for some arguments and Hypothesis for others.

Writing tests¶

A test in Hypothesis consists of two parts: A function that looks like a normal test in your test framework of choice but with some additional arguments, and a @given decorator that specifies how to provide those arguments.

Here are some other examples of how you could use that:

from hypothesis import given
import hypothesis.strategies as st
@given(st.integers(), st.integers())
def test_ints_are_commutative(x, y):


    assert x + y == y + x
@given(x=st.integers(), y=st.integers())
def test_ints_cancel(x, y):


    assert (x + y) - y == x
@given(st.lists(st.integers()))
def test_reversing_twice_gives_same_list(xs):


    # This will generate lists of arbitrary length (usually between 0 and


    # 100 elements) whose elements are integers.


    ys = list(xs)


    ys.reverse()


    ys.reverse()


    assert xs == ys
@given(st.tuples(st.booleans(), st.text()))
def test_look_tuples_work_too(t):


    # A tuple is generated as the one you provided, with the corresponding


    # types in those positions.


    assert len(t) == 2


    assert isinstance(t[0], bool)


    assert isinstance(t[1], str)

Note that as we saw in the above example you can pass arguments to @given either as positional or as keywords.

Where to start¶

You should now know enough of the basics to write some tests for your code using Hypothesis. The best way to learn is by doing, so go have a try.

If you're stuck for ideas for how to use this sort of test for your code, here are some good starting points:

If you have any trouble getting started, don't feel shy about asking for help.

Additional test output¶

Normally the output of a failing test will look something like:

Falsifying example: test_a_thing(x=1, y="foo")

With the repr of each keyword argument being printed.

Sometimes this isn't enough, either because you have values with a repr that isn't very descriptive or because you need to see the output of some intermediate steps of your test. That's where the note function comes in:

>>> from hypothesis import given, note, strategies as st
>>> @given(st.lists(st.integers()), st.randoms())
... def test_shuffle_is_noop(ls, r):
...     ls2 = list(ls)
...     r.shuffle(ls2)
...     note("Shuffle: %r" % (ls2))
...     assert ls == ls2
...
>>> try:
...     test_shuffle_is_noop()
... except AssertionError:
...     print('ls != ls2')
Falsifying example: test_shuffle_is_noop(ls=[0, 1], r=RandomWithSeed(53))
Shuffle: [1, 0]
ls != ls2

The note is printed in the final run of the test in order to include any additional information you might need in your test.

Test Statistics¶

If you are using py.test you can see a number of statistics about the executed tests by passing the command line argument --hypothesis-show-statistics. This will include some general statistics about the test:

For example if you ran the following with --hypothesis-show-statistics:

from hypothesis import given, strategies as st
@given(st.integers())
def test_integers(i):


    pass

You would see:

test_integers:


  - 100 passing examples, 0 failing examples, 0 invalid examples


  - Typical runtimes: ~ 1ms


  - Fraction of time spent in data generation: ~ 12%


  - Stopped because settings.max_examples=100

The final "Stopped because" line is particularly important to note: It tells you the setting value that determined when the test should stop trying new examples. This can be useful for understanding the behaviour of your tests. Ideally you'd always want this to be max_examples.

In some cases (such as filtered and recursive strategies) you will see events mentioned which describe some aspect of the data generation:

from hypothesis import given, strategies as st
@given(st.integers().filter(lambda x: x % 2 == 0))
def test_even_integers(i):


    pass

You would see something like:

test_even_integers:


    - 100 passing examples, 0 failing examples, 36 invalid examples


    - Typical runtimes: 0-1 ms


    - Fraction of time spent in data generation: ~ 16%


    - Stopped because settings.max_examples=100


    - Events:


      * 80.88%, Retried draw from integers().filter(lambda x: <unknown>) to satisfy filter


      * 26.47%, Aborted test because unable to satisfy integers().filter(lambda x: <unknown>)

You can also mark custom events in a test using the event function:

from hypothesis import given, event, strategies as st
@given(st.integers().filter(lambda x: x % 2 == 0))
def test_even_integers(i):


    event("i mod 3 = %d" % (i % 3,))

You will then see output like:

test_even_integers:


  - 100 passing examples, 0 failing examples, 38 invalid examples


  - Typical runtimes: 0-1 ms


  - Fraction of time spent in data generation: ~ 16%


  - Stopped because settings.max_examples=100


  - Events:


    * 80.43%, Retried draw from integers().filter(lambda x: <unknown>) to satisfy filter


    * 31.88%, i mod 3 = 0


    * 27.54%, Aborted test because unable to satisfy integers().filter(lambda x: <unknown>)


    * 21.74%, i mod 3 = 1


    * 18.84%, i mod 3 = 2

Arguments to event can be any hashable type, but two events will be considered the same if they are the same when converted to a string with str.

Making assumptions¶

Sometimes Hypothesis doesn't give you exactly the right sort of data you want - it's mostly of the right shape, but some examples won't work and you don't want to care about them. You can just ignore these by aborting the test early, but this runs the risk of accidentally testing a lot less than you think you are. Also it would be nice to spend less time on bad examples - if you're running 200 examples per test (the default) and it turns out 150 of those examples don't match your needs, that's a lot of wasted time.

For example suppose you had the following test:

@given(floats())
def test_negation_is_self_inverse(x):


    assert x == -(-x)

Running this gives us:

Falsifying example: test_negation_is_self_inverse(x=float('nan'))
AssertionError

This is annoying. We know about NaN and don't really care about it, but as soon as Hypothesis finds a NaN example it will get distracted by that and tell us about it. Also the test will fail and we want it to pass.

So lets block off this particular example:

from math import isnan
@given(floats())
def test_negation_is_self_inverse_for_non_nan(x):


    assume(not isnan(x))


    assert x == -(-x)

And this passes without a problem.

In order to avoid the easy trap where you assume a lot more than you intended, Hypothesis will fail a test when it can't find enough examples passing the assumption.

If we'd written:

@given(floats())
def test_negation_is_self_inverse_for_non_nan(x):


    assume(False)


    assert x == -(-x)

Then on running we'd have got the exception:

Unsatisfiable: Unable to satisfy assumptions of hypothesis test_negation_is_self_inverse_for_non_nan. Only 0 examples considered satisfied assumptions

How good is assume?¶

Hypothesis has an adaptive exploration strategy to try to avoid things which falsify assumptions, which should generally result in it still being able to find examples in hard to find situations.

Suppose we had the following:

@given(lists(integers()))
def test_sum_is_positive(xs):


  assert sum(xs) > 0

Unsurprisingly this fails and gives the falsifying example [].

Adding assume(xs) to this removes the trivial empty example and gives us [0].

Adding assume(all(x > 0 for x in xs)) and it passes: the sum of a list of positive integers is positive.

The reason that this should be surprising is not that it doesn't find a counter-example, but that it finds enough examples at all.

In order to make sure something interesting is happening, suppose we wanted to try this for long lists. e.g. suppose we added an assume(len(xs) > 10) to it. This should basically never find an example: a naive strategy would find fewer than one in a thousand examples, because if each element of the list is negative with probability one-half, you'd have to have ten of these go the right way by chance. In the default configuration Hypothesis gives up long before it's tried 1000 examples (by default it tries 200).

Here's what happens if we try to run this:

@given(lists(integers()))
def test_sum_is_positive(xs):


    assume(len(xs) > 10)


    assume(all(x > 0 for x in xs))


    print(xs)


    assert sum(xs) > 0
In: test_sum_is_positive()
[17, 12, 7, 13, 11, 3, 6, 9, 8, 11, 47, 27, 1, 31, 1]
[6, 2, 29, 30, 25, 34, 19, 15, 50, 16, 10, 3, 16]
[25, 17, 9, 19, 15, 2, 2, 4, 22, 10, 10, 27, 3, 1, 14, 17, 13, 8, 16, 9, 2...
[17, 65, 78, 1, 8, 29, 2, 79, 28, 18, 39]
[13, 26, 8, 3, 4, 76, 6, 14, 20, 27, 21, 32, 14, 42, 9, 24, 33, 9, 5, 15, ...
[2, 1, 2, 2, 3, 10, 12, 11, 21, 11, 1, 16]

As you can see, Hypothesis doesn't find many examples here, but it finds some - enough to keep it happy.

In general if you can shape your strategies better to your tests you should - for example integers(1, 1000) is a lot better than assume(1 <= x <= 1000), but assume will take you a long way if you can't.

Defining strategies¶

The type of object that is used to explore the examples given to your test function is called a SearchStrategy. These are created using the functions exposed in the hypothesis.strategies module.

Many of these strategies expose a variety of arguments you can use to customize generation. For example for integers you can specify min and max values of integers you want. If you want to see exactly what a strategy produces you can ask for an example:

>>> integers(min_value=0, max_value=10).example()
1

Many strategies are built out of other strategies. For example, if you want to define a tuple you need to say what goes in each element:

>>> from hypothesis.strategies import tuples
>>> tuples(integers(), integers()).example()
(-24597, 12566)

Further details are available in a separate document.

The gory details of given parameters¶

The @given decorator may be used to specify which arguments of a function should be parametrized over. You can use either positional or keyword arguments or a mixture of the two.

For example all of the following are valid uses:

@given(integers(), integers())
def a(x, y):


  pass
@given(integers())
def b(x, y):


  pass
@given(y=integers())
def c(x, y):


  pass
@given(x=integers())
def d(x, y):


  pass
@given(x=integers(), y=integers())
def e(x, **kwargs):


  pass
@given(x=integers(), y=integers())
def f(x, *args, **kwargs):


  pass
class SomeTest(TestCase):


    @given(integers())


    def test_a_thing(self, x):


        pass

The following are not:

@given(integers(), integers(), integers())
def g(x, y):


    pass
@given(integers())
def h(x, *args):


    pass
@given(integers(), x=integers())
def i(x, y):


    pass
@given()
def j(x, y):


    pass

The rules for determining what are valid uses of given are as follows:

The reason for the "rightmost named arguments" behaviour is so that using @given with instance methods works: self will be passed to the function as normal and not be parametrized over.

The function returned by given has all the same arguments as the original test, minus those that are filled in by given.

Custom function execution¶

Hypothesis provides you with a hook that lets you control how it runs examples.

This lets you do things like set up and tear down around each example, run examples in a subprocess, transform coroutine tests into normal tests, etc.

The way this works is by introducing the concept of an executor. An executor is essentially a function that takes a block of code and run it. The default executor is:

def default_executor(function):


    return function()

You define executors by defining a method execute_example on a class. Any test methods on that class with @given used on them will use self.execute_example as an executor with which to run tests. For example, the following executor runs all its code twice:

from unittest import TestCase
class TestTryReallyHard(TestCase):


    @given(integers())


    def test_something(self, i):


        perform_some_unreliable_operation(i)


    def execute_example(self, f):


        f()


        return f()

Note: The functions you use in map, etc. will run inside the executor. i.e. they will not be called until you invoke the function passed to execute_example.

An executor must be able to handle being passed a function which returns None, otherwise it won't be able to run normal test cases. So for example the following executor is invalid:

from unittest import TestCase
class TestRunTwice(TestCase):


    def execute_example(self, f):


        return f()()

and should be rewritten as:

from unittest import TestCase
import inspect
class TestRunTwice(TestCase):


    def execute_example(self, f):


        result = f()


        if inspect.isfunction(result):


            result = result()


        return result

Using Hypothesis to find values¶

You can use Hypothesis's data exploration features to find values satisfying some predicate. This is generally useful for exploring custom strategies defined with @composite, or experimenting with conditions for filtering data.

>>> from hypothesis import find
>>> from hypothesis.strategies import sets, lists, integers
>>> find(lists(integers()), lambda x: sum(x) >= 10)
[10]
>>> find(lists(integers()), lambda x: sum(x) >= 10 and len(x) >= 3)
[0, 0, 10]
>>> find(sets(integers()), lambda x: sum(x) >= 10 and len(x) >= 3)
{0, 1, 9}

The first argument to find() describes data in the usual way for an argument to given(), and supports all the same data types. The second is a predicate it must satisfy.

Of course not all conditions are satisfiable. If you ask Hypothesis for an example to a condition that is always false it will raise an error:

>>> find(integers(), lambda x: False)
Traceback (most recent call last):


    ...
hypothesis.errors.NoSuchExample: No examples of condition lambda x: <unknown>

(The lambda x: unknown is because Hypothesis can't retrieve the source code of lambdas from the interactive python console. It gives a better error message most of the time which contains the actual condition)

Inferred Strategies¶

In some cases, Hypothesis can work out what to do when you omit arguments. This is based on introspection, not magic, and therefore has well-defined limits.

builds() will check the signature of the target (using getfullargspec()). If there are required arguments with type annotations and no strategy was passed to builds(), from_type() is used to fill them in. You can also pass the special value hypothesis.infer as a keyword argument, to force this inference for arguments with a default value.

>>> def func(a: int, b: str):
...     return [a, b]
>>> builds(func).example()
[-6993, '']

@given does not perform any implicit inference for required arguments, as this would break compatibility with pytest fixtures. infer can be used as a keyword argument to explicitly fill in an argument from its type annotation.

@given(a=infer)
def test(a: int): pass
# is equivalent to
@given(a=integers())
def test(a): pass

Limitations¶

PEP 3107 type annotations are not supported on Python 2, and Hypothesis does not inspect PEP 484 type comments at runtime. While from_type() will work as usual, inference in builds() and @given will only work if you manually create the __annotations__ attribute (e.g. by using @annotations(...) and @returns(...) decorators). The python:typing module is fully supported on Python 2 if you have the backport installed.

The python:typing module is provisional and has a number of internal changes between Python 3.5.0 and 3.6.1, including at minor versions. These are all supported on a best-effort basis, but you may encounter problems with an old version of the module. Please report them to us, and consider updating to a newer version of Python as a workaround.

Available settings¶

Controlling What Runs¶

Hypothesis divides tests into four logically distinct phases:

The phases setting provides you with fine grained control over which of these run, with each phase corresponding to a value on the Phase enum:

The phases argument accepts a collection with any subset of these. e.g. settings(phases=[Phase.generate, Phase.shrink]) will generate new examples and shrink them, but will not run explicit examples or reuse previous failures, while settings(phases=[Phase.explicit]) will only run the explicit examples.

Seeing intermediate result¶

To see what's going on while Hypothesis runs your tests, you can turn up the verbosity setting. This works with both find() and @given.

>>> from hypothesis import find, settings, Verbosity
>>> from hypothesis.strategies import lists, booleans
>>> find(lists(integers()), any, settings=settings(verbosity=Verbosity.verbose))
Trying example []
Found satisfying example [-1198601713, -67, 116, -29578]
Shrunk example to [113, -29578]
Shrunk example to [-18163]
Shrunk example to [-18163]
Shrunk example to [-18163]
Shrunk example to [-18163]
Shrunk example to [-18163]
Shrunk example to [-70]
Shrunk example to [1]
Shrunk example to [1]
[1]

The four levels are quiet, normal, verbose and debug. normal is the default, while in quiet Hypothesis will not print anything out, even the final falsifying example. debug is basically verbose but a bit more so. You probably don't want it.

You can also override the default by setting the environment variable HYPOTHESIS_VERBOSITY_LEVEL to the name of the level you want. So e.g. setting HYPOTHESIS_VERBOSITY_LEVEL=verbose will run all your tests printing intermediate results and errors.

If you are using pytest, you may also need to disable output capturing for passing tests.

Building settings objects¶

Settings can be created by calling settings with any of the available settings values. Any absent ones will be set to defaults:

>>> from hypothesis import settings
>>> settings().max_examples
100
>>> settings(max_examples=10).max_examples
10

You can also copy settings from other settings:

>>> s = settings(max_examples=10)
>>> t = settings(s, max_iterations=20)
>>> s.max_examples
10
>>> t.max_iterations
20
>>> s.max_iterations
1000
>>> s.max_shrinks
500
>>> t.max_shrinks
500

Default settings¶

At any given point in your program there is a current default settings, available as settings.default. As well as being a settings object in its own right, all newly created settings objects which are not explicitly based off another settings are based off the default, so will inherit any values that are not explicitly set from it.

You can change the defaults by using profiles (see next section), but you can also override them locally by using a settings object as a context manager

>>> with settings(max_examples=150):
...     print(settings.default.max_examples)
...     print(settings().max_examples)
150
150
>>> settings().max_examples
100

Note that after the block exits the default is returned to normal.

You can use this by nesting test definitions inside the context:

from hypothesis import given, settings
with settings(max_examples=500):


    @given(integers())


    def test_this_thoroughly(x):


        pass

All settings objects created or tests defined inside the block will inherit their defaults from the settings object used as the context. You can still override them with custom defined settings of course.

Warning: If you use define test functions which don't use @given inside a context block, these will not use the enclosing settings. This is because the context manager only affects the definition, not the execution of the function.

settings Profiles¶

Depending on your environment you may want different default settings. For example: during development you may want to lower the number of examples to speed up the tests. However, in a CI environment you may want more examples so you are more likely to find bugs.

Hypothesis allows you to define different settings profiles. These profiles can be loaded at any time.

Loading a profile changes the default settings but will not change the behavior of tests that explicitly change the settings.

>>> from hypothesis import settings
>>> settings.register_profile("ci", settings(max_examples=1000))
>>> settings().max_examples
100
>>> settings.load_profile("ci")
>>> settings().max_examples
1000

Instead of loading the profile and overriding the defaults you can retrieve profiles for specific tests.

>>> with settings.get_profile("ci"):
...     print(settings().max_examples)
...
1000

Optionally, you may define the environment variable to load a profile for you. This is the suggested pattern for running your tests on CI. The code below should run in a conftest.py or any setup/initialization section of your test suite. If this variable is not defined the Hypothesis defined defaults will be loaded.

>>> import os
>>> from hypothesis import settings, Verbosity
>>> settings.register_profile("ci", settings(max_examples=1000))
>>> settings.register_profile("dev", settings(max_examples=10))
>>> settings.register_profile("debug", settings(max_examples=10, verbosity=Verbosity.verbose))
>>> settings.load_profile(os.getenv(u'HYPOTHESIS_PROFILE', 'default'))

If you are using the hypothesis pytest plugin and your profiles are registered by your conftest you can load one with the command line option --hypothesis-profile.

$ py.test tests --hypothesis-profile <profile-name>

Timeouts¶

The timeout functionality of Hypothesis is being deprecated, and will eventually be removed. For the moment, the timeout setting can still be set and the old default timeout of one minute remains.

If you want to future proof your code you can get the future behaviour by setting it to the value unlimited, which you can import from the main Hypothesis package:

from hypothesis import given, settings, unlimited
from hypothesis import strategies as st
@settings(timeout=unlimited)
@given(st.integers())
def test_something_slow(i):


    ...

This will cause your code to run until it hits the normal Hypothesis example limits, regardless of how long it takes. timeout=unlimited will remain a valid setting after the timeout functionality has been deprecated (but will then have its own deprecation cycle).

There is however now a timing related health check which is designed to catch tests that run for ages by accident. If you really want your test to run forever, the following code will enable that:

from hypothesis import given, settings, unlimited, HealthCheck
from hypothesis import strategies as st
@settings(timeout=unlimited, suppress_health_check=[


    HealthCheck.hung_test
])
@given(st.integers())
def test_something_slow(i):


    ...

Shrinking¶

When using strategies it is worth thinking about how the data shrinks. Shrinking is the process by which Hypothesis tries to produce human readable examples when it finds a failure - it takes a complex example and turns it into a simpler one.

Each strategy defines an order in which it shrinks - you won't usually need to care about this much, but it can be worth being aware of as it can affect what the best way to write your own strategies is.

The exact shrinking behaviour is not a guaranteed part of the API, but it doesn't change that often and when it does it's usually because we think the new way produces nicer examples.

Possibly the most important one to be aware of is one_of(), which has a preference for values produced by strategies earlier in its argument list. Most of the others should largely "do the right thing" without you having to think about it.

Adapting strategies¶

Often it is the case that a strategy doesn't produce exactly what you want it to and you need to adapt it. Sometimes you can do this in the test, but this hurts reuse because you then have to repeat the adaption in every test.

Hypothesis gives you ways to build strategies from other strategies given functions for transforming the data.

Mapping¶

map is probably the easiest and most useful of these to use. If you have a strategy s and a function f, then an example s.map(f).example() is f(s.example()), i.e. we draw an example from s and then apply f to it.

e.g.:

>>> lists(integers()).map(sorted).example()
[-25527, -24245, -23118, -93, -70, -7, 0, 39, 40, 65, 88, 112, 6189, 9480, 19469, 27256, 32526, 1566924430]

Note that many things that you might use mapping for can also be done with builds().

Filtering¶

filter lets you reject some examples. s.filter(f).example() is some example of s such that f(example) is truthy.

>>> integers().filter(lambda x: x > 11).example()
26126
>>> integers().filter(lambda x: x > 11).example()
23324

It's important to note that filter isn't magic and if your condition is too hard to satisfy then this can fail:

>>> integers().filter(lambda x: False).example()
Traceback (most recent call last):


    ...
hypothesis.errors.NoExamples: Could not find any valid examples in 20 tries

In general you should try to use filter only to avoid corner cases that you don't want rather than attempting to cut out a large chunk of the search space.

A technique that often works well here is to use map to first transform the data and then use filter to remove things that didn't work out. So for example if you wanted pairs of integers (x,y) such that x < y you could do the following:

>>> tuples(integers(), integers()).map(sorted).filter(lambda x: x[0] < x[1]).example()
[-8543729478746591815, 3760495307320535691]

Chaining strategies together¶

Finally there is flatmap. flatmap draws an example, then turns that example into a strategy, then draws an example from that strategy.

It may not be obvious why you want this at first, but it turns out to be quite useful because it lets you generate different types of data with relationships to each other.

For example suppose we wanted to generate a list of lists of the same length:

>>> rectangle_lists = integers(min_value=0, max_value=10).flatmap(
... lambda n: lists(lists(integers(), min_size=n, max_size=n)))
>>> find(rectangle_lists, lambda x: True)
[]
>>> find(rectangle_lists, lambda x: len(x) >= 10)
[[], [], [], [], [], [], [], [], [], []]
>>> find(rectangle_lists, lambda t: len(t) >= 3 and len(t[0]) >= 3)
[[0, 0, 0], [0, 0, 0], [0, 0, 0]]
>>> find(rectangle_lists, lambda t: sum(len(s) for s in t) >= 10)
[[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]]

In this example we first choose a length for our tuples, then we build a strategy which generates lists containing lists precisely of that length. The finds show what simple examples for this look like.

Most of the time you probably don't want flatmap, but unlike filter and map which are just conveniences for things you could just do in your tests, flatmap allows genuinely new data generation that you wouldn't otherwise be able to easily do.

(If you know Haskell: Yes, this is more or less a monadic bind. If you don't know Haskell, ignore everything in these parentheses. You do not need to understand anything about monads to use this, or anything else in Hypothesis).

Recursive data¶

Sometimes the data you want to generate has a recursive definition. e.g. if you wanted to generate JSON data, valid JSON is:

The problem is that you cannot call a strategy recursively and expect it to not just blow up and eat all your memory. The other problem here is that not all unicode strings display consistently on different machines, so we'll restrict them in our doctest.

The way Hypothesis handles this is with the recursive() function which you pass in a base case and a function that, given a strategy for your data type, returns a new strategy for it. So for example:

>>> from string import printable; from pprint import pprint
>>> json = recursive(none() | booleans() | floats() | text(printable),
... lambda children: lists(children) | dictionaries(text(printable), children))
>>> pprint(json.example())
['dy',


 [None, True, 6.297399055778002e+16, False],


 {'a{h\\:694K~{mY>a1yA:#CmDYb': None},


 '\\kP!4',


 {'#1J1': '',


  'cx.': None,


  "jv'A?qyp_sB\n$62g": [],


  'qgnP': [False, -inf, 'la)']},


 [],


 {}]
>>> pprint(json.example())
{'': None,


 '(Rt)': 1.192092896e-07,


 ',': [],


 '6': 2.2250738585072014e-308,


 'HA=/': [],


 'YU]gy8': inf,


 'l': None,


 'nK': False}
>>> pprint(json.example())
[]

That is, we start with our leaf data and then we augment it by allowing lists and dictionaries of anything we can generate as JSON data.

The size control of this works by limiting the maximum number of values that can be drawn from the base strategy. So for example if we wanted to only generate really small JSON we could do this as:

>>> small_lists = recursive(booleans(), lists, max_leaves=5)
>>> small_lists.example()
[False]
>>> small_lists.example()
True
>>> small_lists.example()
[]

Composite strategies¶

The @composite decorator lets you combine other strategies in more or less arbitrary ways. It's probably the main thing you'll want to use for complicated custom strategies.

The composite decorator works by giving you a function as the first argument that you can use to draw examples from other strategies. For example, the following gives you a list and an index into it:

>>> @composite
... def list_and_index(draw, elements=integers()):
...     xs = draw(lists(elements, min_size=1))
...     i = draw(integers(min_value=0, max_value=len(xs) - 1))
...     return (xs, i)

draw(s) is a function that should be thought of as returning s.example(), except that the result is reproducible and will minimize correctly. The decorated function has the initial argument removed from the list, but will accept all the others in the expected order. Defaults are preserved.

>>> list_and_index()
list_and_index()
>>> list_and_index().example()
([-21904], 0)
>>> list_and_index(booleans())
list_and_index(elements=booleans())
>>> list_and_index(booleans()).example()
([True], 0)

Note that the repr will work exactly like it does for all the built-in strategies: it will be a function that you can call to get the strategy in question, with values provided only if they do not match the defaults.

You can use assume inside composite functions:

@composite
def distinct_strings_with_common_characters(draw):


    x = draw(text(), min_size=1)


    y = draw(text(alphabet=x))


    assume(x != y)


    return (x, y)

This works as assume normally would, filtering out any examples for which the passed in argument is falsey.

Drawing interactively in tests¶

There is also the data() strategy, which gives you a means of using strategies interactively. Rather than having to specify everything up front in @given you can draw from strategies in the body of your test:

@given(data())
def test_draw_sequentially(data):


    x = data.draw(integers())


    y = data.draw(integers(min_value=x))


    assert x < y

If the test fails, each draw will be printed with the falsifying example. e.g. the above is wrong (it has a boundary condition error), so will print:

Falsifying example: test_draw_sequentially(data=data(...))
Draw 1: 0
Draw 2: 0

As you can see, data drawn this way is simplified as usual.

Test functions using the data() strategy do not support explicit @example(...)s. In this case, the best option is usually to construct your data with @composite or the explicit example, and unpack this within the body of the test.

Optionally, you can provide a label to identify values generated by each call to data.draw(). These labels can be used to identify values in the output of a falsifying example.

For instance:

@given(data())
def test_draw_sequentially(data):


    x = data.draw(integers(), label='First number')


    y = data.draw(integers(min_value=x), label='Second number')


    assert x < y

will produce the output:

Falsifying example: test_draw_sequentially(data=data(...))
Draw 1 (First number): 0
Draw 2 (Second number): 0

hypothesis[pytz]¶

This module provides pytz timezones.

You can use this strategy to make hypothesis.strategies.datetimes() and hypothesis.strategies.times() produce timezone-aware values.

hypothesis[datetime]¶

This module provides deprecated time and date related strategies.

It depends on the pytz package, which is stable enough that almost any version should be compatible - most updates are for the timezone database.

hypothesis[fakefactory]¶

NOTE:

Faker (previously fake-factory) is a Python package that generates fake data for you. It's great for bootstraping your database, creating good-looking XML documents, stress-testing a database, or anonymizing production data. However, it's not designed for automated testing - data from Hypothesis looks less realistic, but produces minimal bug-triggering examples and uses coverage information to check more cases.

hypothesis.extra.fakefactory lets you use Faker generators to parametrize Hypothesis tests. This was only ever meant to ease your transition to Hypothesis, but we've improved Hypothesis enough since then that we no longer recommend using Faker for automated tests under any circumstances.

hypothesis.extra.fakefactory defines a function fake_factory which returns a strategy for producing text data from any Faker provider.

So for example the following will parametrize a test by an email address:

>>> fake_factory('email').example()
'tnader@prosacco.info'
>>> fake_factory('name').example()
'Zbyněk Černý CSc.'

You can explicitly specify the locale (otherwise it uses any of the available locales), either as a single locale or as several:

>>> fake_factory('name', locale='en_GB').example()
'Antione Gerlach'
>>> fake_factory('name', locales=['en_GB', 'cs_CZ']).example()
'Miloš Šťastný'
>>> fake_factory('name', locales=['en_GB', 'cs_CZ']).example()
'Harm Sanford'

You can use custom Faker providers via the providers argument:

>>> from faker.providers import BaseProvider
>>> class KittenProvider(BaseProvider):
...     def meows(self):
...         return 'meow %d' % (self.random_number(digits=10),)
>>> fake_factory('meows', providers=[KittenProvider]).example()
'meow 9139348419'

Tips and tricks¶

Custom field types¶

If you have a custom Django field type you can register it with Hypothesis's model deriving functionality by registering a default strategy for it:

>>> from toystore.models import CustomishField, Customish
>>> models(Customish).example()
hypothesis.errors.InvalidArgument: Missing arguments for mandatory field


    customish for model Customish
>>> from hypothesis.extra.django.models import add_default_field_mapping
>>> from hypothesis.strategies import just
>>> add_default_field_mapping(CustomishField, just("hi"))
>>> x = models(Customish).example()
>>> x.customish
'hi'

Note that this mapping is on exact type. Subtypes will not inherit it.

Generating child models¶

For the moment there's no explicit support in hypothesis-django for generating dependent models. i.e. a Company model will generate no Shops. However if you want to generate some dependent models as well, you can emulate this by using the flatmap function as follows:

from hypothesis.strategies import lists, just
def generate_with_shops(company):


  return lists(models(Shop, company=just(company))).map(lambda _: company)
company_with_shops_strategy = models(Company).flatmap(generate_with_shops)

Lets unpack what this is doing:

The way flatmap works is that we draw a value from the original strategy, then apply a function to it which gives us a new strategy. We then draw a value from that strategy. So in this case we're first drawing a company, and then we're drawing a list of shops belonging to that company: The just strategy is a strategy such that drawing it always produces the individual value, so models(Shop, company=just(company)) is a strategy that generates a Shop belonging to the original company.

So the following code would give us a list of shops all belonging to the same company:

models(Company).flatmap(lambda c: lists(models(Shop, company=just(c))))

The only difference from this and the above is that we want the company, not the shops. This is where the inner map comes in. We build the list of shops and then throw it away, instead returning the company we started for. This works because the models that Hypothesis generates are saved in the database, so we're essentially running the inner strategy purely for the side effect of creating those children in the database.

Using default field values¶

Hypothesis ignores field defaults and always tries to generate values, even if it doesn't know how to. You can tell it to use the default value for a field instead of generating one by passing fieldname=default_value to models():

>>> from toystore.models import DefaultCustomish
>>> models(DefaultCustomish).example()
hypothesis.errors.InvalidArgument: Missing arguments for mandatory field


    customish for model DefaultCustomish
>>> from hypothesis.extra.django.models import default_value
>>> x = models(DefaultCustomish, customish=default_value).example()
>>> x.customish
'b'

numpy¶

Hypothesis offers a number of strategies for NumPy testing, available in the hypothesis[numpy] extra. It lives in the hypothesis.extra.numpy package.

The centerpiece is the arrays() strategy, which generates arrays with any dtype, shape, and contents you can specify or give a strategy for. To make this as useful as possible, strategies are provided to generate array shapes and generate all kinds of fixed-size or compound dtypes.

>>> import numpy as np
>>> arrays(np.int8, (2, 3)).example()
array([[-8,  6,  3],


       [-6,  4,  6]], dtype=int8)

>>> import numpy as np
>>> from hypothesis.strategies import floats
>>> arrays(np.float, 3, elements=floats(0, 1)).example()
array([ 0.88974794,  0.77387938,  0.1977879 ])

pandas¶

Hypothesis provides strategies for several of the core pandas data types: pandas.Index, pandas.Series and pandas.DataFrame.

The general approach taken by the pandas module is that there are multiple strategies for generating indexes, and all of the other strategies take the number of entries they contain from their index strategy (with sensible defaults). So e.g. a Series is specified by specifying its numpy.dtype (and/or a strategy for generating elements for it).

>>> series(dtype=int).example()
0   -2001747478
1    1153062837

>>> from hypothesis.extra.pandas import column, data_frames
>>> data_frames([
... column('A', dtype=int), column('B', dtype=float)]).example()


            A              B
0  2021915903  1.793898e+232
1  1146643993            inf
2 -2096165693   1.000000e+07

>>> from hypothesis.extra.pandas import column, data_frames
>>> import hypothesis.strategies as st
>>> data_frames(
...     rows=st.tuples(st.floats(allow_nan=False),
...                    st.floats(allow_nan=False)).map(sorted)
... ).example()


               0             1
0  -3.402823e+38  9.007199e+15
1 -1.562796e-298  5.000000e-01

>>> from hypothesis.extra.pandas import columns, data_frames
>>> import hypothesis.strategies as st
>>> data_frames(
...     columns=columns(["lo", "hi"], dtype=float),
...     rows=st.tuples(st.floats(allow_nan=False),
...                    st.floats(allow_nan=False)).map(sorted)
... ).example()


         lo            hi
0   9.314723e-49  4.353037e+45
1  -9.999900e-01  1.000000e+07
2 -2.152861e+134 -1.069317e-73

Supported Versions¶

There is quite a lot of variation between pandas versions. We only commit to supporting the latest version of pandas, but older minor versions are supported on a "best effort" basis. Hypothesis is currently tested against and confirmed working with Pandas 0.19, 0.20, 0.21, and 0.22.

Releases that are not the latest patch release of their minor version are not tested or officially supported, but will probably also work unless you hit a pandas bug.

Limitations¶

The database is best thought of as a cache that you never need to invalidate: Information may be lost when you upgrade a Hypothesis version or change your test, so you shouldn't rely on it for correctness - if there's an example you want to ensure occurs each time then there's a feature for including them in your source code - but it helps the development workflow considerably by making sure that the examples you've just found are reproduced.

File locations¶

The default storage format is as a fairly opaque directory structure. Each test corresponds to a directory, and each example to a file within that directory. The standard location for it is .hypothesis/examples in your current working directory. You can override this, either by setting either the database_file property on a settings object (you probably want to specify it on settings.default) or by setting the HYPOTHESIS_DATABASE_FILE environment variable.

There is also a legacy sqlite3 based format. This is mostly still supported for compatibility reasons, and support will be dropped in some future version of Hypothesis. If you use a database file name ending in .db, .sqlite or .sqlite3 that format will be used instead.

Upgrading Hypothesis and changing your tests¶

The design of the Hypothesis database is such that you can put arbitrary data in the database and not get wrong behaviour. When you upgrade Hypothesis, old data might be invalidated, but this should happen transparently. It should never be the case that e.g. changing the strategy that generates an argument sometimes gives you data from the old strategy.

Sharing your example database¶

NOTE:

The examples database can be shared simply by checking the directory into version control, for example with the following .gitignore:

# Ignore files cached by Hypothesis...
.hypothesis/
# except for the examples directory
!.hypothesis/examples/

Like everything under .hypothesis/, the examples directory will be transparently created on demand. Unlike the other subdirectories, examples/ is designed to handle merges, deletes, etc if you just add the directory into git, mercurial, or any similar version control system.

Rule based state machines¶

Rule based state machines are the ones you're most likely to want to use. They're significantly more user friendly and should be good enough for most things you'd want to do.

A rule based state machine is a collection of functions (possibly with side effects) which may depend on both values that Hypothesis can generate and also on values that have resulted from previous function calls.

You define a rule based state machine as follows:

import unittest
from collections import namedtuple
from hypothesis import strategies as st
from hypothesis.stateful import RuleBasedStateMachine, Bundle, rule
Leaf = namedtuple('Leaf', ('label',))
Split = namedtuple('Split', ('left', 'right'))
class BalancedTrees(RuleBasedStateMachine):


    trees = Bundle('BinaryTree')


    @rule(target=trees, x=st.integers())


    def leaf(self, x):


        return Leaf(x)


    @rule(target=trees, left=trees, right=trees)


    def split(self, left, right):


        return Split(left, right)


    @rule(tree=trees)


    def check_balanced(self, tree):


        if isinstance(tree, Leaf):


            return


        else:


            assert abs(self.size(tree.left) - self.size(tree.right)) <= 1


            self.check_balanced(tree.left)


            self.check_balanced(tree.right)


    def size(self, tree):


        if isinstance(tree, Leaf):


            return 1


        else:


            return 1 + self.size(tree.left) + self.size(tree.right)

In this we declare a Bundle, which is a named collection of previously generated values. We define two rules which put data onto this bundle - one which just generates leaves with integer labels, the other of which takes two previously generated values and returns a new one.

We can then integrate this into our test suite by getting a unittest TestCase from it:

TestTrees = BalancedTrees.TestCase
if __name__ == '__main__':


    unittest.main()

(these will also be picked up by py.test if you prefer to use that). Running this we get:

Step #1: v1 = leaf(x=0)
Step #2: v2 = split(left=v1, right=v1)
Step #3: v3 = split(left=v2, right=v1)
Step #4: check_balanced(tree=v3)
F
======================================================================
FAIL: runTest (hypothesis.stateful.BalancedTrees.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
(...)
assert abs(self.size(tree.left) - self.size(tree.right)) <= 1
AssertionError

Note how it's printed out a very short program that will demonstrate the problem.

...the problem of course being that we've not actually written any code to balance this tree at all, so of course it's not balanced.

So lets balance some trees.

from collections import namedtuple
from hypothesis import strategies as st
from hypothesis.stateful import RuleBasedStateMachine, Bundle, rule
Leaf = namedtuple('Leaf', ('label',))
Split = namedtuple('Split', ('left', 'right'))
class BalancedTrees(RuleBasedStateMachine):


    trees = Bundle('BinaryTree')


    balanced_trees = Bundle('balanced BinaryTree')


    @rule(target=trees, x=st.integers())


    def leaf(self, x):


        return Leaf(x)


    @rule(target=trees, left=trees, right=trees)


    def split(self, left, right):


        return Split(left, right)


    @rule(tree=balanced_trees)


    def check_balanced(self, tree):


        if isinstance(tree, Leaf):


            return


        else:


            assert abs(self.size(tree.left) - self.size(tree.right)) <= 1, \


                repr(tree)


            self.check_balanced(tree.left)


            self.check_balanced(tree.right)


    @rule(target=balanced_trees, tree=trees)


    def balance_tree(self, tree):


        return self.split_leaves(self.flatten(tree))


    def size(self, tree):


        if isinstance(tree, Leaf):


            return 1


        else:


            return self.size(tree.left) + self.size(tree.right)


    def flatten(self, tree):


        if isinstance(tree, Leaf):


            return (tree.label,)


        else:


            return self.flatten(tree.left) + self.flatten(tree.right)


    def split_leaves(self, leaves):


        assert leaves


        if len(leaves) == 1:


            return Leaf(leaves[0])


        else:


            mid = len(leaves) // 2


            return Split(


                self.split_leaves(leaves[:mid]),


                self.split_leaves(leaves[mid:]),


            )

We've now written a really noddy tree balancing implementation. This takes trees and puts them into a new bundle of data, and we only assert that things in the balanced_trees bundle are actually balanced.

If you run this it will sit there silently for a while (you can turn on verbose output to get slightly more information about what's happening. debug will give you all the intermediate programs being run) and then run, telling you your test has passed! Our balancing algorithm worked.

Now lets break it to make sure the test is still valid:

Changing the split to mid = max(len(leaves) // 3, 1) this should no longer balance, which gives us the following counter-example:

v1 = leaf(x=0)
v2 = split(left=v1, right=v1)
v3 = balance_tree(tree=v1)
v4 = split(left=v2, right=v2)
v5 = balance_tree(tree=v4)
check_balanced(tree=v5)

Note that the example could be shrunk further by deleting v3. Due to some technical limitations, Hypothesis was unable to find that particular shrink. In general it's rare for examples produced to be long, but they won't always be minimal.

You can control the detailed behaviour with a settings object on the TestCase (this is a normal hypothesis settings object using the defaults at the time the TestCase class was first referenced). For example if you wanted to run fewer examples with larger programs you could change the settings to:

TestTrees.settings = settings(max_examples=100, stateful_step_count=100)

Which doubles the number of steps each program runs and halves the number of runs relative to the example. settings.timeout will also be respected as usual.

Preconditions¶

While it's possible to use assume() in RuleBasedStateMachine rules, if you use it in only a few rules you can quickly run into a situation where few or none of your rules pass their assumptions. Thus, Hypothesis provides a precondition() decorator to avoid this problem. The precondition() decorator is used on rule-decorated functions, and must be given a function that returns True or False based on the RuleBasedStateMachine instance.

class MyTestMachine(RuleBasedStateMachine):


    state = 1


    @precondition(lambda self: self.state != 0)


    @rule(numerator=integers())


    def divide_with(self, numerator):


        self.state = numerator / self.state

from hypothesis.stateful import RuleBasedStateMachine, rule, precondition
class NumberModifier(RuleBasedStateMachine):


    num = 0


    @rule()


    def add_one(self):


        self.num += 1


    @precondition(lambda self: self.num != 0)


    @rule()


    def divide_with_one(self):


        self.num = 1 / self.num

By using precondition() here instead of assume(), Hypothesis can filter the inapplicable rules before running them. This makes it much more likely that a useful sequence of steps will be generated.

Note that currently preconditions can't access bundles; if you need to use preconditions, you should store relevant data on the instance instead.

Invariant¶

Often there are invariants that you want to ensure are met after every step in a process. It would be possible to add these as rules that are run, but they would be run zero or multiple times between other rules. Hypothesis provides a decorator that marks a function to be run after every step.

class MyTestMachine(RuleBasedStateMachine):


    state = 1


    @invariant()


    def is_nonzero(self):


        assert self.state != 0

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant
class NumberModifier(RuleBasedStateMachine):


    num = 0


    @rule()


    def add_two(self):


        self.num += 2


        if self.num > 50:


            self.num += 1


    @invariant()


    def divide_with_one(self):


        assert self.num % 2 == 0
NumberTest = NumberModifier.TestCase

Invariants can also have precondition()s applied to them, in which case they will only be run if the precondition function returns true.

Note that currently invariants can't access bundles; if you need to use invariants, you should store relevant data on the instance instead.

Generic state machines¶

The class GenericStateMachine is the underlying machinery of stateful testing in Hypothesis. In execution it looks much like the RuleBasedStateMachine but it allows the set of steps available to depend in essentially arbitrary ways on what has happened so far. For example, if you wanted to use Hypothesis to test a game, it could choose each step in the machine based on the game to date and the set of actions the game program is telling it it has available.

It essentially executes the following loop:

machine = MyStateMachine()
try:


    machine.check_invariants()


    for _ in range(n_steps):


        step = machine.steps().example()


        machine.execute_step(step)


        machine.check_invariants()
finally:


    machine.teardown()

Where steps and execute_step are methods you must implement, and teardown and check_invarants are methods you can implement if required. steps returns a strategy, which is allowed to depend arbitrarily on the current state of the test execution. Ideally a good steps implementation should be robust against minor changes in the state. Steps that change a lot between slightly different executions will tend to produce worse quality examples because they're hard to simplify.

The steps method may depend on external state, but it's not advisable and may produce flaky tests.

If any of execute_step, check_invariants or teardown produces an exception, Hypothesis will try to find a minimal sequence of values steps such that the following throws an exception:

machine = MyStateMachine()
try:


    machine.check_invariants()


    for step in steps:


        machine.execute_step(step)


        machine.check_invariants()
finally:


    machine.teardown()

and such that at every point, the step executed is one that could plausible have come from a call to steps in the current state.

Here's an example of using stateful testing to test a broken implementation of a set in terms of a list (note that you could easily do something close to this example with the rule based testing instead, and probably should. This is mostly for illustration purposes):

import unittest
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, integers
class BrokenSet(GenericStateMachine):


    def __init__(self):


        self.data = []


    def steps(self):


        add_strategy = tuples(just("add"), integers())


        if not self.data:


            return add_strategy


        else:


            return (


                add_strategy |


                tuples(just("delete"), sampled_from(self.data)))


    def execute_step(self, step):


        action, value = step


        if action == 'delete':


            try:


                self.data.remove(value)


            except ValueError:


                pass


            assert value not in self.data


        else:


            assert action == 'add'


            self.data.append(value)


            assert value in self.data
TestSet = BrokenSet.TestCase
if __name__ == '__main__':


    unittest.main()

Note that the strategy changes each time based on the data that's currently in the state machine.

Running this gives us the following:

Step #1: ('add', 0)
Step #2: ('add', 0)
Step #3: ('delete', 0)
F
======================================================================
FAIL: runTest (hypothesis.stateful.BrokenSet.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
(...)


    assert value not in self.data
AssertionError

So it adds two elements, then deletes one, and throws an assertion when it finds out that this only deleted one of the copies of the element.

More fine grained control¶

If you want to bypass the TestCase infrastructure you can invoke these manually. The stateful module exposes the function run_state_machine_as_test, which takes an arbitrary function returning a GenericStateMachine and an optional settings parameter and does the same as the class based runTest provided.

In particular this may be useful if you wish to pass parameters to a custom __init__ in your subclass.

Python versions¶

Hypothesis is supported and tested on CPython 2.7 and CPython 3.4+.

Hypothesis also supports PyPy2, and will support PyPy3 when there is a stable release supporting Python 3.4+. Hypothesis does not currently work on Jython, though could feasibly be made to do so. IronPython might work but hasn't been tested. 32-bit and narrow builds should work, though this is currently only tested on Windows.

In general Hypothesis does not officially support anything except the latest patch release of any version of Python it supports. Earlier releases should work and bugs in them will get fixed if reported, but they're not tested in CI and no guarantees are made.

Operating systems¶

In theory Hypothesis should work anywhere that Python does. In practice it is only known to work and regularly tested on OS X, Windows and Linux, and you may experience issues running it elsewhere.

If you're using something else and it doesn't work, do get in touch and I'll try to help, but unless you can come up with a way for me to run a CI server on that operating system it probably won't stay fixed due to the inevitable march of time.

Testing frameworks¶

In general Hypothesis goes to quite a lot of effort to generate things that look like normal Python test functions that behave as closely to the originals as possible, so it should work sensibly out of the box with every test framework.

If your testing relies on doing something other than calling a function and seeing if it raises an exception then it probably won't work out of the box. In particular things like tests which return generators and expect you to do something with them (e.g. nose's yield based tests) will not work. Use a decorator or similar to wrap the test to take this form.

In terms of what's actually known to work:

Coverage works out of the box with Hypothesis (and Hypothesis has 100% branch coverage in its own tests). However you should probably not use Coverage, Hypothesis and PyPy together. Because Hypothesis does quite a lot of CPU heavy work compared to normal tests, it really exacerbates the performance problems the two normally have working together.

Optional Packages¶

The supported versions of optional packages, for strategies in hypothesis.extra, are listed in the documentation for that extra. Our general goal is to support all versions that are supported upstream.

Regularly verifying this¶

Everything mentioned above as explicitly supported is checked on every commit with Travis and Appveyor and goes green before a release happens, so when I say they're supported I really mean it.

Hypothesis versions¶

Backwards compatibility is better than backporting fixes, so we use semantic versioning and only support the most recent version of Hypothesis. See support for more information.

How not to sort by a partial order¶

The following is an example that's been extracted and simplified from a real bug that occurred in an earlier version of Hypothesis. The real bug was a lot harder to find.

Suppose we've got the following type:

class Node(object):


    def __init__(self, label, value):


        self.label = label


        self.value = tuple(value)


    def __repr__(self):


        return "Node(%r, %r)" % (self.label, self.value)


    def sorts_before(self, other):


        if len(self.value) >= len(other.value):


            return False


        return other.value[:len(self.value)] == self.value

Each node is a label and a sequence of some data, and we have the relationship sorts_before meaning the data of the left is an initial segment of the right. So e.g. a node with value [1, 2] will sort before a node with value [1, 2, 3], but neither of [1, 2] nor [1, 3] will sort before the other.

We have a list of nodes, and we want to topologically sort them with respect to this ordering. That is, we want to arrange the list so that if x.sorts_before(y) then x appears earlier in the list than y. We naively think that the easiest way to do this is to extend the partial order defined here to a total order by breaking ties arbitrarily and then using a normal sorting algorithm. So we define the following code:

from functools import total_ordering
@total_ordering
class TopoKey(object):


    def __init__(self, node):


        self.value = node


    def __lt__(self, other):


        if self.value.sorts_before(other.value):


            return True


        if other.value.sorts_before(self.value):


            return False


        return self.value.label < other.value.label
def sort_nodes(xs):


    xs.sort(key=TopoKey)

This takes the order defined by sorts_before and extends it by breaking ties by comparing the node labels.

But now we want to test that it works.

First we write a function to verify that our desired outcome holds:

def is_prefix_sorted(xs):


    for i in range(len(xs)):


        for j in range(i+1, len(xs)):


            if xs[j].sorts_before(xs[i]):


                return False


    return True

This will return false if it ever finds a pair in the wrong order and return true otherwise.

Given this function, what we want to do with Hypothesis is assert that for all sequences of nodes, the result of calling sort_nodes on it is sorted.

First we need to define a strategy for Node:

from hypothesis import settings, strategy
import hypothesis.strategies as s
NodeStrategy = s.builds(


  Node,


  s.integers(),


  s.lists(s.booleans(), average_size=5, max_size=10))

We want to generate short lists of values so that there's a decent chance of one being a prefix of the other (this is also why the choice of bool as the elements). We then define a strategy which builds a node out of an integer and one of those short lists of booleans.

We can now write a test:

from hypothesis import given
@given(s.lists(NodeStrategy))
def test_sorting_nodes_is_prefix_sorted(xs):


    sort_nodes(xs)


    assert is_prefix_sorted(xs)

this immediately fails with the following example:

[Node(0, (False, True)), Node(0, (True,)), Node(0, (False,))]

The reason for this is that because False is not a prefix of (True, True) nor vice versa, sorting things the first two nodes are equal because they have equal labels. This makes the whole order non-transitive and produces basically nonsense results.

But this is pretty unsatisfying. It only works because they have the same label. Perhaps we actually wanted our labels to be unique. Lets change the test to do that.

def deduplicate_nodes_by_label(nodes):


    table = {}


    for node in nodes:


        table[node.label] = node


    return list(table.values())
NodeSet = s.lists(Node).map(deduplicate_nodes_by_label)

We define a function to deduplicate nodes by labels, and then map that over a strategy for lists of nodes to give us a strategy for lists of nodes with unique labels. We can now rewrite the test to use that:

@given(NodeSet)
def test_sorting_nodes_is_prefix_sorted(xs):


    sort_nodes(xs)


    assert is_prefix_sorted(xs)

Hypothesis quickly gives us an example of this still being wrong:

[Node(0, (False,)), Node(-1, (True,)), Node(-2, (False, False))])

Now this is a more interesting example. None of the nodes will sort equal. What is happening here is that the first node is strictly less than the last node because (False,) is a prefix of (False, False). This is in turn strictly less than the middle node because neither is a prefix of the other and -2 < -1. The middle node is then less than the first node because -1 < 0.

So, convinced that our implementation is broken, we write a better one:

def sort_nodes(xs):


    for i in hrange(1, len(xs)):


        j = i - 1


        while j >= 0:


            if xs[j].sorts_before(xs[j+1]):


                break


            xs[j], xs[j+1] = xs[j+1], xs[j]


            j -= 1

This is just insertion sort slightly modified - we swap a node backwards until swapping it further would violate the order constraints. The reason this works is because our order is a partial order already (this wouldn't produce a valid result for a general topological sorting - you need the transitivity).

We now run our test again and it passes, telling us that this time we've successfully managed to sort some nodes without getting it completely wrong. Go us.

Time zone arithmetic¶

This is an example of some tests for pytz which check that various timezone conversions behave as you would expect them to. These tests should all pass, and are mostly a demonstration of some useful sorts of thing to test with Hypothesis, and how the hypothesis-datetime extra package works.

>>> from datetime import timedelta
>>> from hypothesis.extra.pytz import timezones
>>> # The datetimes strategy is naive by default, so tell it to use timezones
>>> aware_datetimes = datetimes(timezones=timezones())
>>> @given(aware_datetimes, timezones(), timezones())
... def test_convert_via_intermediary(dt, tz1, tz2):
...     """Test that converting between timezones is not affected
...     by a detour via another timezone.
...     """
...     assert dt.astimezone(tz1).astimezone(tz2) == dt.astimezone(tz2)
>>> @given(aware_datetimes, timezones())
... def test_convert_to_and_fro(dt, tz2):
...     """If we convert to a new timezone and back to the old one
...     this should leave the result unchanged.
...     """
...     tz1 = dt.tzinfo
...     assert dt == dt.astimezone(tz2).astimezone(tz1)
>>> @given(aware_datetimes, timezones())
... def test_adding_an_hour_commutes(dt, tz):
...     """When converting between timezones it shouldn't matter
...     if we add an hour here or add an hour there.
...     """
...     an_hour = timedelta(hours=1)
...     assert (dt + an_hour).astimezone(tz) == dt.astimezone(tz) + an_hour
>>> @given(aware_datetimes, timezones())
... def test_adding_a_day_commutes(dt, tz):
...     """When converting between timezones it shouldn't matter
...     if we add a day here or add a day there.
...     """
...     a_day = timedelta(days=1)
...     assert (dt + a_day).astimezone(tz) == dt.astimezone(tz) + a_day
>>> # And we can check that our tests pass
>>> test_convert_via_intermediary()
>>> test_convert_to_and_fro()
>>> test_adding_an_hour_commutes()
>>> test_adding_a_day_commutes()

Condorcet's Paradox¶

A classic paradox in voting theory, called Condorcet's paradox, is that majority preferences are not transitive. That is, there is a population and a set of three candidates A, B and C such that the majority of the population prefer A to B, B to C and C to A.

Wouldn't it be neat if we could use Hypothesis to provide an example of this?

Well as you can probably guess from the presence of this section, we can! This is slightly surprising because it's not really obvious how we would generate an election given the types that Hypothesis knows about.

The trick here turns out to be twofold:

Without further ado, here is the code:

from hypothesis import given, assume
from hypothesis.strategies import integers, lists
from collections import Counter
def candidates(votes):


    return {candidate for vote in votes for candidate in vote}
def build_election(votes):


    """


    Given a list of lists we extract an election out of this. We do this


    in two phases:


    1. First of all we work out the full set of candidates present in all


       votes and throw away any votes that do not have that whole set.


    2. We then take each vote and make it unique, keeping only the first


       instance of any candidate.


    This gives us a list of total orderings of some set. It will usually


    be a lot smaller than the starting list, but that's OK.


    """


    all_candidates = candidates(votes)


    votes = list(filter(lambda v: set(v) == all_candidates, votes))


    if not votes:


        return []


    rebuilt_votes = []


    for vote in votes:


        rv = []


        for v in vote:


            if v not in rv:


                rv.append(v)


        assert len(rv) == len(all_candidates)


        rebuilt_votes.append(rv)


    return rebuilt_votes
@given(lists(lists(integers(min_value=1, max_value=5))))
def test_elections_are_transitive(election):


    election = build_election(election)


    # Small elections are unlikely to be interesting


    assume(len(election) >= 3)


    all_candidates = candidates(election)


    # Elections with fewer than three candidates certainly can't exhibit


    # intransitivity


    assume(len(all_candidates) >= 3)


    # Now we check if the election is transitive


    # First calculate the pairwise counts of how many prefer each candidate


    # to the other


    counts = Counter()


    for vote in election:


        for i in range(len(vote)):


            for j in range(i+1, len(vote)):


                counts[(vote[i], vote[j])] += 1


    # Now look at which pairs of candidates one has a majority over the


    # other and store that.


    graph = {}


    all_candidates = candidates(election)


    for i in all_candidates:


        for j in all_candidates:


            if counts[(i, j)] > counts[(j, i)]:


                graph.setdefault(i, set()).add(j)


    # Now for each triple assert that it is transitive.


    for x in all_candidates:


        for y in graph.get(x, ()):


            for z in graph.get(y, ()):


                assert x not in graph.get(z, ())

The example Hypothesis gives me on my first run (your mileage may of course vary) is:

[[3, 1, 4], [4, 3, 1], [1, 4, 3]]

Which does indeed do the job: The majority (votes 0 and 1) prefer 3 to 1, the majority (votes 0 and 2) prefer 1 to 4 and the majority (votes 1 and 2) prefer 4 to 3. This is in fact basically the canonical example of the voting paradox, modulo variations on the names of candidates.

Fuzzing an HTTP API¶

Hypothesis's support for testing HTTP services is somewhat nascent. There are plans for some fully featured things around this, but right now they're probably quite far down the line.

But you can do a lot yourself without any explicit support! Here's a script I wrote to throw random data against the API for an entirely fictitious service called Waspfinder (this is only lightly obfuscated and you can easily figure out who I'm actually talking about, but I don't want you to run this code and hammer their API without their permission).

All this does is use Hypothesis to generate random JSON data matching the format their API asks for and check for 500 errors. More advanced tests which then use the result and go on to do other things are definitely also possible.

import unittest
from hypothesis import given, assume, settings, strategies as st
from collections import namedtuple
import requests
import os
import random
import time
import math
# These tests will be quite slow because we have to talk to an external
# service. Also we'll put in a sleep between calls so as to not hammer it.
# As a result we reduce the number of test cases and turn off the timeout.
settings.default.max_examples = 100
settings.default.timeout = -1
Goal = namedtuple("Goal", ("slug",))
# We just pass in our API credentials via environment variables.
waspfinder_token = os.getenv('WASPFINDER_TOKEN')
waspfinder_user = os.getenv('WASPFINDER_USER')
assert waspfinder_token is not None
assert waspfinder_user is not None
GoalData = st.fixed_dictionaries({


    'title': st.text(),


    'goal_type': st.sampled_from([


        "hustler", "biker", "gainer", "fatloser", "inboxer",


        "drinker", "custom"]),


    'goaldate': st.one_of(st.none(), st.floats()),


    'goalval': st.one_of(st.none(), st.floats()),


    'rate': st.one_of(st.none(), st.floats()),


    'initval': st.floats(),


    'panic': st.floats(),


    'secret': st.booleans(),


    'datapublic': st.booleans(),
})
needs2 = ['goaldate', 'goalval', 'rate']
class WaspfinderTest(unittest.TestCase):


    @given(GoalData)


    def test_create_goal_dry_run(self, data):


        # We want slug to be unique for each run so that multiple test runs


        # don't interfere with each other. If for some reason some slugs trigger


        # an error and others don't we'll get a Flaky error, but that's OK.


        slug = hex(random.getrandbits(32))[2:]


        # Use assume to guide us through validation we know about, otherwise


        # we'll spend a lot of time generating boring examples.


        # Title must not be empty


        assume(data["title"])


        # Exactly two of these values should be not None. The other will be


        # inferred by the API.


        assume(len([1 for k in needs2 if data[k] is not None]) == 2)


        for v in data.values():


            if isinstance(v, float):


                assume(not math.isnan(v))


        data["slug"] = slug


        # The API nicely supports a dry run option, which means we don't have


        # to worry about the user account being spammed with lots of fake goals


        # Otherwise we would have to make sure we cleaned up after ourselves


        # in this test.


        data["dryrun"] = True


        data["auth_token"] = waspfinder_token


        for d, v in data.items():


            if v is None:


                data[d] = "null"


            else:


                data[d] = str(v)


        result = requests.post(


            "https://waspfinder.example.com/api/v1/users/"


            "%s/goals.json" % (waspfinder_user,), data=data)


        # Lets not hammer the API too badly. This will of course make the


        # tests even slower than they otherwise would have been, but that's


        # life.


        time.sleep(1.0)


        # For the moment all we're testing is that this doesn't generate an


        # internal error. If we didn't use the dry run option we could have


        # then tried doing more with the result, but this is a good start.


        self.assertNotEqual(result.status_code, 500)
if __name__ == '__main__':


    unittest.main()

Code of conduct¶

Hypothesis's community is an inclusive space, and everyone in it is expected to abide by a code of conduct.

At the high level the code of conduct goes like this:

While it is impossible to enumerate everything that is unkind, disrespectful or unhelpful, here are some specific things that are definitely against the code of conduct:

What happens when this goes wrong?¶

For minor infractions, I'll just call people on it and ask them to apologise and not do it again. You should feel free to do this too if you're comfortable doing so.

Major infractions and repeat offenders will be banned from the community.

Also, people who have a track record of bad behaviour outside of the Hypothesis community may be banned even if they obey all these rules if their presence is making people uncomfortable.

At the current volume level it's not hard for me to pay attention to the whole community, but if you think I've missed something please feel free to alert me. You can either message me as DRMacIver on freenode or send a me an email at david@drmaciver.com.

Stripe¶

At Stripe we use Hypothesis to test every piece of our machine learning model training pipeline (powered by scikit). Before we migrated, our tests were filled with hand-crafted pandas Dataframes that weren't representative at all of our actual very complex data. Because we needed to craft examples for each test, we took the easy way out and lived with extremely low test coverage.

Hypothesis changed all that. Once we had our strategies for generating Dataframes of features it became trivial to slightly customize each strategy for new tests. Our coverage is now close to 90%.

Full-stop, property-based testing is profoundly more powerful - and has caught or prevented far more bugs - than our old style of example-based testing.

Kristian Glass - Director of Technology at LaterPay GmbH¶

Hypothesis has been brilliant for expanding the coverage of our test cases, and also for making them much easier to read and understand, so we're sure we're testing the things we want in the way we want.

Seth Morton¶

When I first heard about Hypothesis, I knew I had to include it in my two open-source Python libraries, natsort and fastnumbers . Quite frankly, I was a little appalled at the number of bugs and "holes" I found in the code. I can now say with confidence that my libraries are more robust to "the wild." In addition, Hypothesis gave me the confidence to expand these libraries to fully support Unicode input, which I never would have had the stomach for without such thorough testing capabilities. Thanks!

Sixty North¶

At Sixty North we use Hypothesis for testing Segpy an open source Python library for shifting data between Python data structures and SEG Y files which contain geophysical data from the seismic reflection surveys used in oil and gas exploration.

This is our first experience of property-based testing – as opposed to example-based testing. Not only are our tests more powerful, they are also much better explanations of what we expect of the production code. In fact, the tests are much closer to being specifications. Hypothesis has located real defects in our code which went undetected by traditional test cases, simply because Hypothesis is more relentlessly devious about test case generation than us mere humans! We found Hypothesis particularly beneficial for Segpy because SEG Y is an antiquated format that uses legacy text encodings (EBCDIC) and even a legacy floating point format we implemented from scratch in Python.

Hypothesis is sure to find a place in most of our future Python codebases and many existing ones too.

mulkieran¶

Just found out about this excellent QuickCheck for Python implementation and ran up a few tests for my bytesize package last night. Refuted a few hypotheses in the process.

Looking forward to using it with a bunch of other projects as well.

Adam Johnson¶

I have written a small library to serialize dicts to MariaDB's dynamic columns binary format, mariadb-dyncol. When I first developed it, I thought I had tested it really well - there were hundreds of test cases, some of them even taken from MariaDB's test suite itself. I was ready to release.

Lucky for me, I tried Hypothesis with David at the PyCon UK sprints. Wow! It found bug after bug after bug. Even after a first release, I thought of a way to make the tests do more validation, which revealed a further round of bugs! Most impressively, Hypothesis found a complicated off-by-one error in a condition with 4095 versus 4096 bytes of data - something that I would never have found.

Long live Hypothesis! (Or at least, property-based testing).

Josh Bronson¶

Adopting Hypothesis improved bidict's test coverage and significantly increased our ability to make changes to the code with confidence that correct behavior would be preserved. Thank you, David, for the great testing tool.

Cory Benfield¶

Hypothesis is the single most powerful tool in my toolbox for working with algorithmic code, or any software that produces predictable output from a wide range of sources. When using it with Priority, Hypothesis consistently found errors in my assumptions and extremely subtle bugs that would have taken months of real-world use to locate. In some cases, Hypothesis found subtle deviations from the correct output of the algorithm that may never have been noticed at all.

When it comes to validating the correctness of your tools, nothing comes close to the thoroughness and power of Hypothesis.

Jon Moore¶

One extremely satisfied user here. Hypothesis is a really solid implementation of property-based testing, adapted well to Python, and with good features such as failure-case shrinkers. I first used it on a project where we needed to verify that a vendor's Python and non-Python implementations of an algorithm matched, and it found about a dozen cases that previous example-based testing and code inspections had not. Since then I've been evangelizing for it at our firm.

Russel Winder¶

I am using Hypothesis as an integral part of my Python workshops. Testing is an integral part of Python programming and whilst unittest and, better, py.test can handle example-based testing, property-based testing is increasingly far more important than example-base testing, and Hypothesis fits the bill.

Wellfire Interactive¶

We've been using Hypothesis in a variety of client projects, from testing Django-related functionality to domain-specific calculations. It both speeds up and simplifies the testing process since there's so much less tedious and error-prone work to do in identifying edge cases. Test coverage is nice but test depth is even nicer, and it's much easier to get meaningful test depth using Hypothesis.

Cody Kochmann¶

Hypothesis is being used as the engine for random object generation with my open source function fuzzer battle_tested which maps all behaviors of a function allowing you to minimize the chance of unexpected crashes when running code in production.

With how efficient Hypothesis is at generating the edge cases that cause unexpected behavior occur, battle_tested is able to map out the entire behavior of most functions in less than a few seconds.

Hypothesis truly is a masterpiece. I can't thank you enough for building it.

Merchise Autrement¶

Just minutes after our first use of hypothesis we uncovered a subtle bug in one of our most used library. Since then, we have increasingly used hypothesis to improve the quality of our testing in libraries and applications as well.

Your name goes here¶

I know there are many more, because I keep finding out about new people I'd never even heard of using Hypothesis. If you're looking to way to give back to a tool you love, adding your name here only takes a moment and would really help a lot. As per instructions at the top, just send me a pull request and I'll add you to the list.

3.44.24 - 2018-01-27¶

This release fixes dependency information when installing Hypothesis from a binary "wheel" distribution.

3.44.23 - 2018-01-24¶

This release improves shrinking in a class of pathological examples that you are probably never hitting in practice. If you are hitting them in practice this should be a significant speed up in shrinking. If you are not, you are very unlikely to notice any difference. You might see a slight slow down and/or slightly better falsifying examples.

3.44.22 - 2018-01-23¶

This release fixes a dependency problem. It was possible to install Hypothesis with an old version of attrs, which would throw a TypeError as soon as you tried to import hypothesis. Specifically, you need attrs 16.0.0 or newer.

Hypothesis will now require the correct version of attrs when installing.

3.44.21 - 2018-01-22¶

This change adds some additional structural information that Hypothesis will use to guide its search.

You mostly shouldn't see much difference from this. The two most likely effects you would notice are:

3.44.20 - 2018-01-21¶

This is a small refactoring release that changes how Hypothesis tracks some information about the boundary of examples in its internal representation.

You are unlikely to see much difference in behaviour, but memory usage and run time may both go down slightly during normal test execution, and when failing Hypothesis might print its failing example slightly sooner.

3.44.19 - 2018-01-21¶

This changes how we compute the default average_size for all collection strategies. Previously setting a max_size without setting an average_size would have the seemingly paradoxical effect of making data generation slower, because it would raise the average size from its default. Now setting max_size will either leave the default unchanged or lower it from its default.

If you are currently experiencing this problem, this may make your tests substantially faster. If you are not, this will likely have no effect on you.

3.44.18 - 2018-01-20¶

This is a small refactoring release that changes how Hypothesis detects when the structure of data generation depends on earlier values generated (e.g. when using flatmap or composite()). It should not have any observable effect on behaviour.

3.44.17 - 2018-01-15¶

This release fixes a typo in internal documentation, and has no user-visible impact.

3.44.16 - 2018-01-13¶

This release improves test case reduction for recursive data structures. Hypothesis now guarantees that whenever a strategy calls itself recursively (usually this will happen because you are using deferred()), any recursive call may replace the top level value. e.g. given a tree structure, Hypothesis will always try replacing it with a subtree.

Additionally this introduces a new heuristic that may in some circumstances significantly speed up test case reduction - Hypothesis should be better at immediately replacing elements drawn inside another strategy with their minimal possible value.

3.44.15 - 2018-01-13¶

from_type() can now resolve recursive types such as binary trees (issue #1004). Detection of non-type arguments has also improved, leading to better error messages in many cases involving forward references.

3.44.14 - 2018-01-08¶

This release fixes a bug in the shrinker that prevented the optimisations in 3.44.6 from working in some cases. It would not have worked correctly when filtered examples were nested (e.g. with a set of integers in some range).

This would not have resulted in any correctness problems, but shrinking may have been slower than it otherwise could be.

3.44.13 - 2018-01-08¶

This release changes the average bit length of values drawn from integers() to be much smaller. Additionally it changes the shrinking order so that now size is considered before sign - e.g. -1 will be preferred to +10.

The new internal format for integers required some changes to the minimizer to make work well, so you may also see some improvements to example quality in unrelated areas.

3.44.12 - 2018-01-07¶

This changes Hypothesis's internal implementation of weighted sampling. This will affect example distribution and quality, but you shouldn't see any other effects.

3.44.11 - 2018-01-06¶

This is a change to some internals around how Hypothesis handles avoiding generating duplicate examples and seeking out novel regions of the search space.

You are unlikely to see much difference as a result of it, but it fixes a bug where an internal assertion could theoretically be triggered and has some minor effects on the distribution of examples so could potentially find bugs that have previously been missed.

3.44.10 - 2018-01-06¶

This patch avoids creating debug statements when debugging is disabled. Profiling suggests this is a 5-10% performance improvement (issue #1040).

3.44.9 - 2018-01-06¶

This patch blacklists null characters ('\x00') in automatically created strategies for Django CharField and TextField, due to a database issue which was recently fixed upstream (Hypothesis issue #1045).

3.44.8 - 2018-01-06¶

This release makes the Hypothesis shrinker slightly less greedy in order to avoid local minima - when it gets stuck, it makes a small attempt to search around the final example it would previously have returned to find a new starting point to shrink from. This should improve example quality in some cases, especially ones where the test data has dependencies among parts of it that make it difficult for Hypothesis to proceed.

3.44.7 - 2018-01-04¶

This release adds support for Django 2 in the hypothesis-django extra.

This release drops support for Django 1.10, as it is no longer supported by the Django team.

3.44.6 - 2018-01-02¶

This release speeds up test case reduction in many examples by being better at detecting large shrinks it can use to discard redundant parts of its input. This will be particularly noticeable in examples that make use of filtering and for some integer ranges.

3.44.5 - 2018-01-02¶

Happy new year!

This is a no-op release that updates the year range on all of the copyright headers in our source to include 2018.

3.44.4 - 2017-12-23¶

This release fixes issue #1044, which slowed tests by up to 6% due to broken caching.

3.44.3 - 2017-12-21¶

This release improves the shrinker in cases where examples drawn earlier can affect how much data is drawn later (e.g. when you draw a length parameter in a composite and then draw that many elements). Examples found in cases like this should now be much closer to minimal.

3.44.2 - 2017-12-20¶

This is a pure refactoring release which changes how Hypothesis manages its set of examples internally. It should have no externally visible effects.

3.44.1 - 2017-12-18¶

This release fixes issue #997, in which under some circumstances the body of tests run under Hypothesis would not show up when run under coverage even though the tests were run and the code they called outside of the test file would show up normally.

3.44.0 - 2017-12-17¶

This release adds a new feature: The @reproduce_failure, designed to make it easy to use Hypothesis's binary format for examples to reproduce a problem locally without having to share your example database between machines.

This also changes when seeds are printed:

This work was funded by Smarkets.

3.43.1 - 2017-12-17¶

This release fixes a bug with Hypothesis's database management - examples that were found in the course of shrinking were saved in a way that indicated that they had distinct causes, and so they would all be retried on the start of the next test. The intended behaviour, which is now what is implemented, is that only a bounded subset of these examples would be retried.

3.43.0 - 2017-12-17¶

HypothesisDeprecationWarning now inherits from python:FutureWarning instead of python:DeprecationWarning, as recommended by PEP 565 for user-facing warnings (issue #618). If you have not changed the default warnings settings, you will now see each distinct HypothesisDeprecationWarning instead of only the first.

3.42.2 - 2017-12-12¶

This patch fixes issue #1017, where instances of a list or tuple subtype used as an argument to a strategy would be coerced to tuple.

3.42.1 - 2017-12-10¶

This release has some internal cleanup, which makes reading the code more pleasant and may shrink large examples slightly faster.

3.42.0 - 2017-12-09¶

This release deprecates faker-extra, which was designed as a transition strategy but does not support example shrinking or coverage-guided discovery.

3.41.0 - 2017-12-06¶

sampled_from() can now sample from one-dimensional numpy ndarrays. Sampling from multi-dimensional ndarrays still results in a deprecation warning. Thanks to Charlie Tanksley for this patch.

3.40.1 - 2017-12-04¶

This release makes two changes:

3.40.0 - 2017-12-02¶

This release improves how various ways of seeding Hypothesis interact with the example database:

This work was funded by Smarkets.

3.39.0 - 2017-12-01¶

This release adds a new health check that checks if the smallest "natural" possible example of your test case is very large - this will tend to cause Hypothesis to generate bad examples and be quite slow.

This work was funded by Smarkets.

3.38.9 - 2017-11-29¶

This is a documentation release to improve the documentation of shrinking behaviour for Hypothesis's strategies.

3.38.8 - 2017-11-29¶

This release improves the performance of characters() when using blacklist_characters and from_regex() when using negative character classes.

The problems this fixes were found in the course of work funded by Smarkets.

3.38.7 - 2017-11-29¶

This is a patch release for from_regex(), which had a bug in handling of the python:re.VERBOSE flag (issue #992). Flags are now handled correctly when parsing regex.

3.38.6 - 2017-11-28¶

This patch changes a few byte-string literals from double to single quotes, thanks to an update in unify. There are no user-visible changes.

3.38.5 - 2017-11-23¶

This fixes the repr of strategies using lambda that are defined inside decorators to include the lambda source.

This would mostly have been visible when using the statistics functionality - lambdas used for e.g. filtering would have shown up with a <unknown> as their body. This can still happen, but it should happen less often now.

3.38.4 - 2017-11-22¶

This release updates the reported statistics so that they show approximately what fraction of your test run time is spent in data generation (as opposed to test execution).

This work was funded by Smarkets.

3.38.3 - 2017-11-21¶

This is a documentation release, which ensures code examples are up to date by running them as doctests in CI (issue #711).

3.38.2 - 2017-11-21¶

This release changes the behaviour of the deadline setting when used with data(): Time spent inside calls to data.draw will no longer be counted towards the deadline time.

As a side effect of some refactoring required for this work, the way flaky tests are handled has changed slightly. You are unlikely to see much difference from this, but some error messages will have changed.

This work was funded by Smarkets.

3.38.1 - 2017-11-21¶

This patch has a variety of non-user-visible refactorings, removing various minor warts ranging from indirect imports to typos in comments.

3.38.0 - 2017-11-18¶

This release overhauls the health check system in a variety of small ways. It adds no new features, but is nevertheless a minor release because it changes which tests are likely to fail health checks.

The most noticeable effect is that some tests that used to fail health checks will now pass, and some that used to pass will fail. These should all be improvements in accuracy. In particular:

If your data generation is especially slow, you may also see your tests get somewhat faster, as there is no longer a separate health check phase. This will be particularly noticeable when rerunning test failures.

This work was funded by Smarkets.

3.37.0 - 2017-11-12¶

This is a deprecation release for some health check related features.

The following are now deprecated:

In addition, passing a non-iterable value as suppress_health_check will now raise an error immediately (it would never have worked correctly, but it would previously have failed later). Some validation error messages have also been updated.

This work was funded by Smarkets.

3.36.1 - 2017-11-10¶

This is a yak shaving release, mostly concerned with our own tests.

While getfullargspec() was documented as deprecated in Python 3.5, it never actually emitted a warning. Our code to silence this (nonexistent) warning has therefore been removed.

We now run our tests with DeprecationWarning as an error, and made some minor changes to our own tests as a result. This required similar upstream updates to coverage and execnet (a test-time dependency via pytest-xdist).

There is no user-visible change in Hypothesis itself, but we encourage you to consider enabling deprecations as errors in your own tests.

3.36.0 - 2017-11-06¶

This release adds a setting to the public API, and does some internal cleanup:

3.35.0 - 2017-11-06¶

This minor release supports constraining uuids() to generate a particular version of UUID (issue #721).

Thanks to Dion Misic for this feature.

3.34.1 - 2017-11-02¶

This patch updates the documentation to suggest builds(callable) instead of just(callable()).

3.34.0 - 2017-11-02¶

Hypothesis now emits deprecation warnings if you apply @given more than once to a target.

Applying @given repeatedly wraps the target multiple times. Each wrapper will search the space of of possible parameters separately. This is equivalent but will be much more inefficient than doing it with a single call to @given.

For example, instead of @given(booleans()) @given(integers()), you could write @given(booleans(), integers())

3.33.1 - 2017-11-02¶

This is a bugfix release:

3.33.0 - 2017-10-16¶

This release supports strategy inference for more field types in Django models() - you can now omit an argument for Date, Time, Duration, Slug, IP Address, and UUID fields. (issue #642)

Strategy generation for fields with grouped choices now selects choices from each group, instead of selecting from the group names.

3.32.2 - 2017-10-15¶

This patch removes the mergedb tool, introduced in Hypothesis 1.7.1 on an experimental basis. It has never actually worked, and the new Hypothesis example database is designed to make such a tool unnecessary.

3.32.1 - 2017-10-13¶

This patch has two improvements for strategies based on enumerations.

3.32.0 - 2017-10-09¶

This changes the default value of use_coverage=True to True when running on pypy (it was already True on CPython).

It was previously set to False because we expected it to be too slow, but recent benchmarking shows that actually performance of the feature on pypy is fairly acceptable - sometimes it's slower than on CPython, sometimes it's faster, but it's generally within a factor of two either way.

3.31.6 - 2017-10-08¶

This patch improves the quality of strategies inferred from Numpy dtypes:

This has already been useful in Hypothesis - we found an overflow bug in our Pandas support, and as a result indexes() and range_indexes() now check that min_size and max_size are at least zero.

3.31.5 - 2017-10-08¶

This release fixes a performance problem in tests where use_coverage is set to True.

Tests experience a slow-down proportionate to the amount of code they cover. This is still the case, but the factor is now low enough that it should be unnoticeable. Previously it was large and became much larger in 3.28.4.

3.31.4 - 2017-10-08¶

from_type() failed with a very confusing error if passed a NewType() (issue #901). These psudeo-types are now unwrapped correctly, and strategy inference works as expected.

3.31.3 - 2017-10-06¶

This release makes some small optimisations to our use of coverage that should reduce constant per-example overhead. This is probably only noticeable on examples where the test itself is quite fast. On no-op tests that don't test anything you may see up to a fourfold speed increase (which is still significantly slower than without coverage). On more realistic tests the speed up is likely to be less than that.

3.31.2 - 2017-09-30¶

This release fixes some formatting and small typos/grammar issues in the documentation, specifically the page docs/settings.rst, and the inline docs for the various settings.

3.31.1 - 2017-09-30¶

This release improves the handling of deadlines so that they act better with the shrinking process. This fixes issue #892.

This involves two changes:

In addition, this release also clarifies the documentation of the deadline setting slightly to be more explicit about where it applies.

This work was funded by Smarkets.

3.31.0 - 2017-09-29¶

This release blocks installation of Hypothesis on Python 3.3, which reached its end of life date on 2017-09-29.

This should not be of interest to anyone but downstream maintainers - if you are affected, migrate to a secure version of Python as soon as possible or at least seek commercial support.

3.30.4 - 2017-09-27¶

This release makes several changes:

The new algorithm for 1 also makes some changes to Hypothesis's low level data generation which apply even with coverage turned off. They generally reduce the total amount of data generated, which should improve test performance somewhat. Between this and 3 you should see a noticeable reduction in test runtime (how much so depends on your tests and how much example size affects their performance. On our benchmarks, where data generation dominates, we saw up to a factor of two performance improvement, but it's unlikely to be that large.

3.30.3 - 2017-09-25¶

This release fixes some formatting and small typos/grammar issues in the documentation, specifically the page docs/details.rst, and some inline docs linked from there.

3.30.2 - 2017-09-24¶

This release changes Hypothesis's caching approach for functions in hypothesis.strategies. Previously it would have cached extremely aggressively and cache entries would never be evicted. Now it adopts a least-frequently used, least recently used key invalidation policy, and is somewhat more conservative about which strategies it caches.

Workloads which create strategies based on dynamic values, e.g. by using flatmap or composite(), will use significantly less memory.

3.30.1 - 2017-09-22¶

This release fixes a bug where when running with use_coverage=True inside an existing running instance of coverage, Hypothesis would frequently put files that the coveragerc excluded in the report for the enclosing coverage.

3.30.0 - 2017-09-20¶

This release introduces two new features:

This work was funded by Smarkets.

3.29.0 - 2017-09-19¶

This release makes Hypothesis coverage aware. Hypothesis now runs all test bodies under coverage, and uses this information to guide its testing.

The use_coverage setting can be used to disable this behaviour if you want to test code that is sensitive to coverage being enabled (either because of performance or interaction with the trace function).

The main benefits of this feature are:

This also has the following side-effects:

This feature is only partially supported under pypy. It is significantly slower than on CPython and is turned off by default as a result, but it should still work correctly if you want to use it.

3.28.3 - 2017-09-18¶

This release is an internal change that affects how Hypothesis handles calculating certain properties of strategies.

The primary effect of this is that it fixes a bug where use of deferred() could sometimes trigger an internal assertion error. However the fix for this bug involved some moderately deep changes to how Hypothesis handles certain constructs so you may notice some additional knock-on effects.

In particular the way Hypothesis handles drawing data from strategies that cannot generate any values has changed to bail out sooner than it previously did. This may speed up certain tests, but it is unlikely to make much of a difference in practice for tests that were not already failing with Unsatisfiable.

3.28.2 - 2017-09-18¶

This is a patch release that fixes a bug in the hypothesis.extra.pandas documentation where it incorrectly referred to column() instead of columns().

3.28.1 - 2017-09-16¶

This is a refactoring release. It moves a number of internal uses of namedtuple() over to using attrs based classes, and removes a couple of internal namedtuple classes that were no longer in use.

It should have no user visible impact.

3.28.0 - 2017-09-15¶

This release adds support for testing pandas via the hypothesis.extra.pandas module.

It also adds a dependency on attrs.

This work was funded by Stripe.

3.27.1 - 2017-09-14¶

This release fixes some formatting and broken cross-references in the documentation, which includes editing docstrings - and thus a patch release.

3.27.0 - 2017-09-13¶

This release introduces a deadline setting to Hypothesis.

When set this turns slow tests into errors. By default it is unset but will warn if you exceed 200ms, which will become the default value in a future release.

This work was funded by Smarkets.

3.26.0 - 2017-09-12¶

Hypothesis now emits deprecation warnings if you are using the legacy SQLite example database format, or the tool for merging them. These were already documented as deprecated, so this doesn't change their deprecation status, only that we warn about it.

3.25.1 - 2017-09-12¶

This release fixes a bug with generating numpy datetime and timedelta types: When inferring the strategy from the dtype, datetime and timedelta dtypes with sub-second precision would always produce examples with one second resolution. Inferring a strategy from a time dtype will now always produce example with the same precision.

3.25.0 - 2017-09-12¶

This release changes how Hypothesis shrinks and replays examples to take into account that it can encounter new bugs while shrinking the bug it originally found. Previously it would end up replacing the originally found bug with the new bug and show you only that one. Now it is (often) able to recognise when two bugs are distinct and when it finds more than one will show both.

3.24.2 - 2017-09-11¶

This release removes the (purely internal and no longer useful) strategy_test_suite function and the corresponding strategytests module.

3.24.1 - 2017-09-06¶

This release improves the reduction of examples involving floating point numbers to produce more human readable examples.

It also has some general purpose changes to the way the minimizer works internally, which may see some improvement in quality and slow down of test case reduction in cases that have nothing to do with floating point numbers.

3.24.0 - 2017-09-05¶

Hypothesis now emits deprecation warnings if you use some_strategy.example() inside a test function or strategy definition (this was never intended to be supported, but is sufficiently widespread that it warrants a deprecation path).

3.23.3 - 2017-09-05¶

This is a bugfix release for decimals() with the places argument.

3.23.2 - 2017-09-01¶

This is a small refactoring release that removes a now-unused parameter to an internal API. It shouldn't have any user visible effect.

3.23.1 - 2017-09-01¶

Hypothesis no longer propagates the dynamic scope of settings into strategy definitions.

This release is a small change to something that was never part of the public API and you will almost certainly not notice any effect unless you're doing something surprising, but for example the following code will now give a different answer in some circumstances:

import hypothesis.strategies as st
from hypothesis import settings
CURRENT_SETTINGS = st.builds(lambda: settings.default)

(We don't actually encourage you writing code like this)

Previously this would have generated the settings that were in effect at the point of definition of CURRENT_SETTINGS. Now it will generate the settings that are used for the current test.

It is very unlikely to be significant enough to be visible, but you may also notice a small performance improvement.

3.23.0 - 2017-08-31¶

This release adds a unique argument to arrays() which behaves the same ways as the corresponding one for lists(), requiring all of the elements in the generated array to be distinct.

3.22.2 - 2017-08-29¶

This release fixes an issue where Hypothesis would raise a TypeError when using the datetime-related strategies if running with PYTHONOPTIMIZE=2. This bug was introduced in v3.20.0. (See issue #822)

3.22.1 - 2017-08-28¶

Hypothesis now transparently handles problems with an internal unicode cache, including file truncation or read-only filesystems (issue #767). Thanks to Sam Hames for the patch.

3.22.0 - 2017-08-26¶

This release provides what should be a substantial performance improvement to numpy arrays generated using provided numpy support, and adds a new fill_value argument to arrays() to control this behaviour.

This work was funded by Stripe.

3.21.3 - 2017-08-26¶

This release fixes some extremely specific circumstances that probably have never occurred in the wild where users of deferred() might have seen a RuntimeError from too much recursion, usually in cases where no valid example could have been generated anyway.

3.21.2 - 2017-08-25¶

This release fixes some minor bugs in argument validation:

3.21.1 - 2017-08-24¶

This release fixes a bug where test failures that were the result of an @example would print an extra stack trace before re-raising the exception.

3.21.0 - 2017-08-23¶

This release deprecates Hypothesis's strict mode, which turned Hypothesis's deprecation warnings into errors. Similar functionality can be achieved by using simplefilter('error', HypothesisDeprecationWarning).

3.20.0 - 2017-08-22¶

This release renames the relevant arguments on the datetimes(), dates(), times(), and timedeltas() strategies to min_value and max_value, to make them consistent with the other strategies in the module.

The old argument names are still supported but will emit a deprecation warning when used explicitly as keyword arguments. Arguments passed positionally will go to the new argument names and are not deprecated.

3.19.3 - 2017-08-22¶

This release provides a major overhaul to the internals of how Hypothesis handles shrinking.

This should mostly be visible in terms of getting better examples for tests which make heavy use of composite(), data() or flatmap where the data drawn depends a lot on previous choices, especially where size parameters are affected. Previously Hypothesis would have struggled to reliably produce good examples here. Now it should do much better. Performance should also be better for examples with a non-zero min_size.

You may see slight changes to example generation (e.g. improved example diversity) as a result of related changes to internals, but they are unlikely to be significant enough to notice.

3.19.2 - 2017-08-21¶

This release fixes two bugs in hypothesis.extra.numpy:

3.19.1 - 2017-08-21¶

This is a bugfix release for issue #739, where bounds for fractions() or floating-point decimals() were not properly converted to integers before passing them to the integers strategy. This excluded some values that should have been possible, and could trigger internal errors if the bounds lay between adjacent integers.

You can now bound fractions() with two arbitrarily close fractions.

It is now an explicit error to supply a min_value, max_value, and max_denominator to fractions() where the value bounds do not include a fraction with denominator at most max_denominator.

3.19.0 - 2017-08-20¶

This release adds the from_regex() strategy, which generates strings that contain a match of a regular expression.

Thanks to Maxim Kulkin for creating the hypothesis-regex package and then helping to upstream it! (issue #662)

3.18.5 - 2017-08-18¶

This is a bugfix release for integers(). Previously the strategy would hit an internal assertion if passed non-integer bounds for min_value and max_value that had no integers between them. The strategy now raises InvalidArgument instead.

3.18.4 - 2017-08-18¶

Release to fix a bug where mocks can be used as test runners under certain conditions. Specifically, if a mock is injected into a test via pytest fixtures or patch decorators, and that mock is the first argument in the list, hypothesis will think it represents self and turns the mock into a test runner. If this happens, the affected test always passes because the mock is executed instead of the test body. Sometimes, it will also fail health checks.

Fixes issue #491 and a section of issue #198. Thanks to Ben Peterson for this bug fix.

3.18.3 - 2017-08-17¶

This release should improve the performance of some tests which experienced a slow down as a result of the 3.13.0 release.

Tests most likely to benefit from this are ones that make extensive use of min_size parameters, but others may see some improvement as well.

3.18.2 - 2017-08-16¶

This release fixes a bug introduced in 3.18.0. If the arguments whitelist_characters and blacklist_characters to characters() both contained elements, then an InvalidArgument exception would be raised.

Thanks to Zac Hatfield-Dodds for reporting and fixing this.

3.18.1 - 2017-08-14¶

This is a bug fix release to fix issue #780, where sets() and similar would trigger health check errors if their element strategy could only produce one element (e.g. if it was just()).

3.18.0 - 2017-08-13¶

This is a feature release:

Thanks to Alex Willmer for these.

3.17.0 - 2017-08-07¶

This release documents the previously undocumented phases feature, making it part of the public API. It also updates how the example database is used. Principally:

3.16.1 - 2017-08-07¶

This release makes an implementation change to how Hypothesis handles certain internal constructs.

The main effect you should see is improvement to the behaviour and performance of collection types, especially ones with a min_size parameter. Many cases that would previously fail due to being unable to generate enough valid examples will now succeed, and other cases should run slightly faster.

3.16.0 - 2017-08-04¶

This release introduces a deprecation of the timeout feature. This results in the following changes:

3.15.0 - 2017-08-04¶

This release deprecates two strategies, choices() and streaming().

Both of these are somewhat confusing to use and are entirely redundant since the introduction of the data() strategy for interactive drawing in tests, and their use should be replaced with direct use of data() instead.

3.14.2 - 2017-08-03¶

This fixes a bug where Hypothesis would not work correctly on Python 2.7 if you had the python:typing module backport installed.

3.14.1 - 2017-08-02¶

This raises the maximum depth at which Hypothesis starts cutting off data generation to a more reasonable value which it is harder to hit by accident.

This resolves (issue #751), in which some examples which previously worked would start timing out, but it will also likely improve the data generation quality for complex data types.

3.14.0 - 2017-07-23¶

Hypothesis now understands inline type annotations (issue #293):

3.13.1 - 2017-07-20¶

This is a bug fix release for issue #514 - Hypothesis would continue running examples after a SkipTest exception was raised, including printing a falsifying example. Skip exceptions from the standard python:unittest module, and pytest, nose, or unittest2 modules now abort the test immediately without printing output.

3.13.0 - 2017-07-16¶

This release has two major aspects to it: The first is the introduction of deferred(), which allows more natural definition of recursive (including mutually recursive) strategies.

The second is a number of engine changes designed to support this sort of strategy better. These should have a knock-on effect of also improving the performance of any existing strategies that currently generate a lot of data or involve heavy nesting by reducing their typical example size.

3.12.0 - 2017-07-07¶

This release makes some major internal changes to how Hypothesis represents data internally, as a prelude to some major engine changes that should improve data quality. There are no API changes, but it's a significant enough internal change that a minor version bump seemed warranted.

User facing impact should be fairly mild, but includes:

If you notice any effects you consider to be a significant regression, please open an issue about them.

3.11.6 - 2017-06-19¶

This release involves no functionality changes, but is the first to ship wheels as well as an sdist.

3.11.5 - 2017-06-18¶

This release provides a performance improvement to shrinking. For cases where there is some non-trivial "boundary" value (e.g. the bug happens for all values greater than some other value), shrinking should now be substantially faster. Other types of bug will likely see improvements too.

This may also result in some changes to the quality of the final examples - it may sometimes be better, but is more likely to get slightly worse in some edge cases. If you see any examples where this happens in practice, please report them.

3.11.4 - 2017-06-17¶

This is a bugfix release: Hypothesis now prints explicit examples when running in verbose mode. (issue #313)

3.11.3 - 2017-06-11¶

This is a bugfix release: Hypothesis no longer emits a warning if you try to use sampled_from() with python:collections.OrderedDict. (issue #688)

3.11.2 - 2017-06-10¶

This is a documentation release. Several outdated snippets have been updated or removed, and many cross-references are now hyperlinks.

3.11.1 - 2017-05-28¶

This is a minor ergonomics release. Tracebacks shown by pytest no longer include Hypothesis internals for test functions decorated with @given.

3.11.0 - 2017-05-23¶

This is a feature release, adding datetime-related strategies to the core strategies.

timezones() allows you to sample pytz timezones from the Olsen database. Use directly in a recipe for tz-aware datetimes, or compose with none() to allow a mix of aware and naive output.

The new dates(), times(), datetimes(), and timedeltas() strategies are all constrained by objects of their type. This means that you can generate dates bounded by a single day (i.e. a single date), or datetimes constrained to the microsecond.

times() and datetimes() take an optional timezones= argument, which defaults to none() for naive times. You can use our extra strategy based on pytz, or roll your own timezones strategy with dateutil or even the standard library.

The old dates, times, and datetimes strategies in hypothesis.extra.datetimes are deprecated in favor of the new core strategies, which are more flexible and have no dependencies.

3.10.0 - 2017-05-22¶

Hypothesis now uses python:inspect.getfullargspec() internally. On Python 2, there are no visible changes.

On Python 3 @given and @composite now preserve PEP 3107 annotations on the decorated function. Keyword-only arguments are now either handled correctly (e.g. @composite), or caught in validation instead of silently discarded or raising an unrelated error later (e.g. @given).

3.9.1 - 2017-05-22¶

This is a bugfix release: the default field mapping for a DateTimeField in the Django extra now respects the USE_TZ setting when choosing a strategy.

3.9.0 - 2017-05-19¶

This is feature release, expanding the capabilities of the decimals() strategy.

3.8.5 - 2017-05-16¶

Hypothesis now imports python:sqlite3 when a SQLite database is used, rather than at module load, improving compatibility with Python implementations compiled without SQLite support (such as BSD or Jython).

3.8.4 - 2017-05-16¶

This is a compatibility bugfix release. sampled_from no longer raises a deprecation warning when sampling from an Enum, as all enums have a reliable iteration order.

3.8.3 - 2017-05-09¶

This release removes a version check for older versions of pytest when using the Hypothesis pytest plugin. The pytest plugin will now run unconditionally on all versions of pytest. This breaks compatibility with any version of pytest prior to 2.7.0 (which is more than two years old).

The primary reason for this change is that the version check was a frequent source of breakage when pytest change their versioning scheme. If you are not working on pytest itself and are not running a very old version of it, this release probably doesn't affect you.

3.8.2 - 2017-04-26¶

This is a code reorganisation release that moves some internal test helpers out of the main source tree so as to not have changes to them trigger releases in future.

3.8.1 - 2017-04-26¶

This is a documentation release. Almost all code examples are now doctests checked in CI, eliminating stale examples.

3.8.0 - 2017-04-23¶

This is a feature release, adding the iterables() strategy, equivalent to lists(...).map(iter) but with a much more useful repr. You can use this strategy to check that code doesn't accidentally depend on sequence properties such as indexing support or repeated iteration.

3.7.4 - 2017-04-22¶

This is a bug fix release for a single bug:

3.7.3 - 2017-04-21¶

This release should include no user visible changes and is purely a refactoring release. This modularises the behaviour of the core given() function, breaking it up into smaller and more accessible parts, but its actual behaviour should remain unchanged.

3.7.2 - 2017-04-21¶

This reverts an undocumented change in 3.7.1 which broke installation on debian stable: The specifier for the hypothesis[django] extra_requires had introduced a wild card, which was not supported on the default version of pip.

3.7.1 - 2017-04-21¶

This is a bug fix and internal improvements release.

3.7.0 - 2017-03-20¶

This is a feature release.

New features:

Bug fixes:

There have also been a number of performance optimizations:

3.6.1 - 2016-12-20¶

This release fixes a dependency problem and makes some small behind the scenes improvements.

3.6.0 - 2016-10-31¶

This release reverts Hypothesis to its old pretty printing of lambda functions based on attempting to extract the source code rather than decompile the bytecode. This is unfortunately slightly inferior in some cases and may result in you occasionally seeing things like lambda x: <unknown> in statistics reports and strategy reprs.

This removes the dependencies on uncompyle6, xdis and spark-parser.

The reason for this is that the new functionality was based on uncompyle6, which turns out to introduce a hidden GPLed dependency - it in turn depended on xdis, and although the library was licensed under the MIT license, it contained some GPL licensed source code and thus should have been released under the GPL.

My interpretation is that Hypothesis itself was never in violation of the GPL (because the license it is under, the Mozilla Public License v2, is fully compatible with being included in a GPL licensed work), but I have not consulted a lawyer on the subject. Regardless of the answer to this question, adding a GPLed dependency will likely cause a lot of users of Hypothesis to inadvertently be in violation of the GPL.

As a result, if you are running Hypothesis 3.5.x you really should upgrade to this release immediately.

3.5.3 - 2016-10-05¶

This is a bug fix release.

Bugs fixed:

3.5.2 - 2016-09-24¶

This is a bug fix release.

3.5.1 - 2016-09-23¶

This is a bug fix release.

3.5.0 - 2016-09-22¶

This is a feature release.

Additionally there have been some minor bug fixes:

3.4.2 - 2016-07-13¶

This is a bug fix release, fixing a number of problems with the settings system:

3.4.1 - 2016-07-07¶

This is a bug fix release for a single bug:

3.4.0 - 2016-05-27¶

This release is entirely provided by Lucas Wiman:

Strategies constructed by models() will now respect much more of Django's validations out of the box. Wherever possible full_clean() should succeed.

In particular:

3.3.0 - 2016-05-27¶

This release went wrong and is functionally equivalent to 3.2.0. Ignore it.

3.2.0 - 2016-05-19¶

This is a small single-feature release:

3.1.3 - 2016-05-01¶

Single bug fix release

3.1.2 - 2016-04-30¶

Single bug fix release:

3.1.1 - 2016-04-29¶

Minor bug fix release.

3.1.0 - 2016-03-06¶

3.0.5 - 2016-02-25¶

3.0.4 - 2016-02-24¶

3.0.3 - 2016-02-23¶

3.0.2 - 2016-02-18¶

3.0.1 - 2016-02-18¶

3.0.0 - 2016-02-17¶

Codename: This really should have been 2.1.

Externally this looks like a very small release. It has one small breaking change that probably doesn't affect anyone at all (some behaviour that never really worked correctly is now outright forbidden) but necessitated a major version bump and one visible new feature.

Internally this is a complete rewrite. Almost nothing other than the public API is the same.

New features:

New limitations:

Feature removal:

Performance improvements:

In general your tests should have got faster. If they've instead got significantly slower, I'm interested in hearing about it.

Data distribution:

The data distribution should have changed significantly. This may uncover bugs the previous version missed. It may also miss bugs the previous version could have uncovered. Hypothesis is now producing less strongly correlated data than it used to, but the correlations are extended over more of the structure.

Shrinking:

Shrinking quality should have improved. In particular Hypothesis can now perform simultaneous shrinking of separate examples within a single test (previously it was only able to do this for elements of a single collection). In some cases performance will have improved, in some cases it will have got worse but generally shouldn't have by much.

2.0.0 - 2016-01-10¶

Codename: A new beginning

This release cleans up all of the legacy that accrued in the course of Hypothesis 1.0. These are mostly things that were emitting deprecation warnings in 1.19.0, but there were a few additional changes.

In particular:

This also includes two non-deprecation changes:

1.19.0 - 2016-01-09¶

Codename: IT COMES

This release heralds the beginning of a new and terrible age of Hypothesis 2.0.

It's primary purpose is some final deprecations prior to said release. The goal is that if your code emits no warnings under this release then it will probably run unchanged under Hypothesis 2.0 (there are some caveats to this: 2.0 will drop support for some Python versions, and if you're using internal APIs then as usual that may break without warning).

It does have two new features:

API changes (old usage still works but is deprecated):

Additional deprecations:

Bug fixes:

1.18.1 - 2015-12-22¶

Two behind the scenes changes:

1.18.0 - 2015-12-21¶

Features:

Bug fixes:

1.17.1 - 2015-12-16¶

A small bug fix release, which fixes the fact that the 'note' function could not be used on tests which used the @example decorator to provide explicit examples.

1.17.0 - 2015-12-15¶

This is actually the same release as 1.16.1, but 1.16.1 has been pulled because it contains the following additional change that was not intended to be in a patch release (it's perfectly stable, but is a larger change that should have required a minor version bump):

1.16.1 - 2015-12-14¶

Note: This release has been removed.

A small bugfix release that allows bdists for Hypothesis to be built under 2.7 - the compat3.py file which had Python 3 syntax wasn't intended to be loaded under Python 2, but when building a bdist it was. In particular this would break running setup.py test.

1.16.0 - 2015-12-08¶

There are no public API changes in this release but it includes a behaviour change that I wasn't comfortable putting in a patch release.

1.15.0 - 2015-11-24¶

A release with two new features.

1.14.0 - 2015-11-01¶

New features:

Bugs:

1.13.0 - 2015-10-29¶

This is quite a small release, but deprecates some public API functions and removes some internal API functionality so gets a minor version bump.

1.12.0 - 2015-10-18¶

1.11.4 - 2015-09-27¶

1.11.3 - 2015-09-23¶

1.11.2 - 2015-09-23¶

Bug fixes:

Misc:

1.11.1 - 2015-09-16¶

Bug fixes:

1.11.0 - 2015-08-31¶

1.10.6 - 2015-08-26¶

Fix support for fixtures on Django 1.7.

1.10.4 - 2015-08-21¶

Tiny bug fix release:

1.10.3 - 2015-08-19¶

Another small bug fix release.

1.10.2 - 2015-08-19¶

This is a small bug fix release:

1.10.0 - 2015-08-04¶

This is just a bugfix and performance release, but it changes some semi-public APIs, hence the minor version bump.

1.9.0 - 2015-07-27¶

Codename: The great bundling.

This release contains two fairly major changes.

The first is the deprecation of the hypothesis-extra mechanism. From now on all the packages that were previously bundled under it other than hypothesis-pytest (which is a different beast and will remain separate). The functionality remains unchanged and you can still import them from exactly the same location, they just are no longer separate packages.

The second is that this introduces a new way of building strategies which lets you build up strategies recursively from other strategies.

It also contains the minor change that calling .example() on a strategy object will give you examples that are more representative of the actual data you'll get. There used to be some logic in there to make the examples artificially simple but this proved to be a bad idea.

1.8.5 - 2015-07-24¶

This contains no functionality changes but fixes a mistake made with building the previous package that would have broken installation on Windows.

1.8.4 - 2015-07-20¶

Bugs fixed:

1.8.3 - 2015-07-20¶

"Falsifying example" would not have been printed when the failure came from an explicit example.

1.8.2 - 2015-07-18¶

Another small bugfix release:

1.8.1 - 2015-07-17¶

This is a small release that contains a workaround for people who have bad reprs returning non ascii text on Python 2.7. This is not a bug fix for Hypothesis per se because that's not a thing that is actually supposed to work, but Hypothesis leans more heavily on repr than is typical so it's worth having a workaround for.

1.8.0 - 2015-07-16¶

New features:

Mostly invisible implementation details that may result in finding new bugs in your code:

Bug fixes:

1.7.2 - 2015-07-10¶

This is purely a bug fix release:

1.7.1 - 2015-06-29¶

Codename: There is no 1.7.0.

A slight technical hitch with a premature upload means there's was a yanked 1.7.0 release. Oops.

The major feature of this release is Python 2.6 support. Thanks to Jeff Meadows for doing most of the work there.

Other minor features

Bug fixes:

1.6.2 - 2015-06-08¶

This is just a few small bug fixes:

1.6.1 - 2015-05-21¶

This is a small patch release that fixes a bug where 1.6.0 broke the use of flatmap with the deprecated API and assumed the passed in function returned a SearchStrategy instance rather than converting it to a strategy.

1.6.0 - 2015-05-21¶

This is a smallish release designed to fix a number of bugs and smooth out some weird behaviours.

1.5.0 - 2015-05-14¶

Codename: Strategic withdrawal.

The purpose of this release is a radical simplification of the API for building strategies. Instead of the old approach of @strategy.extend and things that get converted to strategies, you just build strategies directly.

The old method of defining strategies will still work until Hypothesis 2.0, because it's a major breaking change, but will now emit deprecation warnings.

The new API is also a lot more powerful as the functions for defining strategies give you a lot of dials to turn. See the updated data section for details.

Other changes:

1.4.0 - 2015-05-04¶

Codename: What a state.

The big feature of this release is the new and slightly experimental stateful testing API. You can read more about that in the appropriate section.

Two minor features the were driven out in the course of developing this:

Breakage of semi-public SearchStrategy API:

Hypothesis Stateful testing was then turned upon Hypothesis itself, which lead to an amazing number of minor bugs being found in Hypothesis itself.

Bugs fixed (most but not all from the result of stateful testing) include:

Some minor quality improvements:

1.3.0 - 2015-05-22¶

New features:

Bug fixes:

General improvements:

1.2.1 - 2015-04-16¶

A small patch release for a bug in the new executors feature. Tests which require doing something to their result in order to fail would have instead reported as flaky.

1.2.0 - 2015-04-15¶

Codename: Finders keepers.

A bunch of new features and improvements.

1.1.1 - 2015-04-07¶

Codename: Nothing to see here

This is just a patch release put out because it fixed some internal bugs that would block the Django integration release but did not actually affect anything anyone could previously have been using. It also contained a minor quality fix for floats that I'd happened to have finished in time.

1.1.0 - 2015-04-06¶

Codename: No-one mention the M word.

1.0.0 - 2015-03-27¶

Codename: Blast-off!

There are no code changes in this release. This is precisely the 0.9.2 release with some updated documentation.

0.9.2 - 2015-03-26¶

Codename: T-1 days.

0.9.1 - 2015-03-25¶

Codename: T-2 days.

0.9.0 - 2015-03-23¶

Codename: The final countdown

This release could also be called 1.0-RC1.

It contains a teeny tiny bugfix, but the real point of this release is to declare feature freeze. There will be zero functionality changes between 0.9.0 and 1.0 unless something goes really really wrong. No new features will be added, no breaking API changes will occur, etc. This is the final shakedown before I declare Hypothesis stable and ready to use and throw a party to celebrate.

Bug bounty for any bugs found between now and 1.0: I will buy you a drink (alcoholic, caffeinated, or otherwise) and shake your hand should we ever find ourselves in the same city at the same time.

The one tiny bugfix:

0.7.2 - 2015-03-22¶

Codename: Hygienic macros or bust

0.7.1 - 2015-03-21¶

Codename: Point releases go faster

0.7.0, - 2015-03-20¶

Codename: Starting to look suspiciously real

This is probably the last minor release prior to 1.0. It consists of stability improvements, a few usability things designed to make Hypothesis easier to try out, and filing off some final rough edges from the API.

0.6.0 - 2015-03-13¶

Codename: I'm sorry, were you using that API?

This is primarily a "simplify all the weird bits of the API" release. As a result there are a lot of breaking changes. If you just use @given with core types then you're probably fine.

In particular:

Bug fixes:

Hypothesis Extra:

0.5.0 - 2015-02-10¶

Codename: Read all about it.

Core hypothesis:

Hypothesis extra:

First off, introducing Hypothesis extra packages!

These are packages that are separated out from core Hypothesis because they have one or more dependencies. Every hypothesis-extra package is pinned to a specific point release of Hypothesis and will have some version requirements on its dependency. They use entry_points so you will usually not need to explicitly import them, just have them installed on the path.

This release introduces two of them:

hypothesis-datetime:

Does what it says on the tin: Generates datetimes for Hypothesis. Just install the package and datetime support will start working.

Depends on pytz for timezone support

hypothesis-pytest:

A very rudimentary pytest plugin. All it does right now is hook the display of falsifying examples into pytest reporting.

Depends on pytest.

0.4.3 - 2015-02-05¶

Codename: TIL narrow Python builds are a thing

This just fixes the one bug.

0.4.2 - 2015-02-04¶

Codename: O(dear)

This is purely a bugfix release:

0.4.1 - 2015-02-03¶

Codename: Cruel and unusual edge cases

This release is mostly about better test case generation.

Enhancements:

Bug fixes:

Other:

0.4.0 - 2015-01-21¶

FLAGSHIP FEATURE: Hypothesis now persists examples for later use. It stores data in a local SQLite database and will reuse it for all tests of the same type.

LICENSING CHANGE: Hypothesis is now released under the Mozilla Public License 2.0. This applies to all versions from 0.4.0 onwards until further notice. The previous license remains applicable to all code prior to 0.4.0.

Enhancements:

Bugs fixed:

0.3.2 - 2015-01-16¶

0.3.1 - 2015-01-13¶

0.3.0 - 2015-01-12¶

0.2.2 - 2015-01-08¶

0.2.1 - 2015-01-07¶

0.2.0 - 2015-01-07¶

0.1.4 - 2013-12-14¶

0.1.3 - 2013-05-03¶

0.1.2 - 2013-03-24¶

0.1.1 - 2013-03-24¶

0.1.0 - 2013-03-23¶

0.0.5 - 2013-03-13¶

0.0.4 - 2013-03-13¶

0.0.3 - 2013-03-13¶

0.0.2 - 2013-03-12¶

0.0.1 - 2013-03-10¶

Release Policy¶

Hypothesis releases follow semantic versioning.

We maintain backwards-compatibility wherever possible, and use deprecation warnings to mark features that have been superseded by a newer alternative. If you want to detect this, you can upgrade warnings to errors in the usual ways.

We use continuous deployment to ensure that you can always use our newest and shiniest features - every change to the source tree is automatically built and published on PyPI as soon as it's merged onto master, after code review and passing our extensive test suite.

Project Roadmap¶

Hypothesis does not have a long-term release plan. However some visibility into our plans for future compatibility may be useful:

Release tarballs¶

These are available from the GitHub releases page. The tarballs on pypi are intended for installation from a Python tool such as pip and should not be considered complete releases. Requests to include additional files in them will not be granted. Their absence is not a bug.

Dependencies¶

Python versions¶

Hypothesis is designed to work with a range of Python versions. Currently supported are:

If you feel the need to have separate Python 3 and Python 2 packages you can, but Hypothesis works unmodified on either.

Other Python libraries¶

Hypothesis has mandatory dependencies on the following libraries:

Hypothesis has optional dependencies on the following libraries:

The way this works when installing Hypothesis normally is that these features become available if the relevant library is installed.

Testing Hypothesis¶

If you want to test Hypothesis as part of your packaging you will probably not want to use the mechanisms Hypothesis itself uses for running its tests, because it has a lot of logic for installing and testing against different versions of Python.

The tests must be run with py.test. A version more recent than 2.8.0 is strongly encouraged, but it may work with earlier versions (however py.test specific logic is disabled before 2.8.0).

Tests are organised into a number of top level subdirectories of the tests/ directory.

An example invocation for running the coverage subset of these tests:

pip install -e .
pip install pytest # you will probably want to use your own packaging here
python -m pytest tests/cover

Examples¶

Providing explicit examples¶

You can explicitly ask Hypothesis to try a particular example, using

Hypothesis will run all examples you've asked for first. If any of them fail it will not go on to look for more examples.

It doesn't matter whether you put the example decorator before or after given. Any permutation of the decorators in the above will do the same thing.

Note that examples can be positional or keyword based. If they're positional then they will be filled in from the right when calling, so either of the following styles will work as expected:

@given(text())
@example("Hello world")
@example(x="Some very long string")
def test_some_code(x):


    assert True
from unittest import TestCase
class TestThings(TestCase):


    @given(text())


    @example("Hello world")


    @example(x="Some very long string")


    def test_some_code(self, x):


        assert True

As with @given, it is not permitted for a single example to be a mix of positional and keyword arguments. Either are fine, and you can use one in one example and the other in another example if for some reason you really want to, but a single example must be consistent.

Reproducing a test run with @seed¶

When a test fails unexpectedly, usually due to a health check failure, Hypothesis will print out a seed that led to that failure, if the test is not already running with a fixed seed. You can then recreate that failure using either the @seed decorator or (if you are running pytest) with --hypothesis-seed.

Reproducing an example with with @reproduce_failure¶

Hypothesis has an opaque binary representation that it uses for all examples it generates. This representation is not intended to be stable across versions or with respect to changes in the test, but can be used to to reproduce failures with the @reproduce_example decorator.

The intent is that you should never write this decorator by hand, but it is instead provided by Hypothesis. When a test fails with a falsifying example, Hypothesis may print out a suggestion to use @reproduce_failure on the test to recreate the problem as follows:

>>> from hypothesis import settings, given, PrintSettings
>>> import hypothesis.strategies as st
>>> @given(st.floats())
... @settings(print_blob=PrintSettings.ALWAYS)
... def test(f):
...     assert f == f
...
>>> try:
...     test()
... except AssertionError:
...     pass
Falsifying example: test(f=nan)
You can reproduce this example by temporarily adding @reproduce_failure(..., b'AAAA//AAAAAAAAEA') as a decorator on your test case

Adding the suggested decorator to the test should reproduce the failure (as long as everything else is the same - changing the versions of Python or anything else involved, might of course affect the behaviour of the test! Note that changing the version of Hypothesis will result in a different error - each @reproduce_failure invocation is specific to a Hypothesis version).

When to do this is controlled by the print_blob setting, which may be one of the following values: