Pyjo.Test - Writing test scripts¶

from Pyjo.Test import *

import os

if not os.getenv('TEST_ONLINE'):
    plan(skip_all='set TEST_ONLINE to enable this test')

ok(got == expected, 'test name')

is_ok(got, expected, 'test name')
isnt_ok(got, unexpected, 'test name')
like_ok(got, r'expected', 'test name')
unlike_ok(got, r'unexpected', 'test name')
cmp_ok(got, '==', expected, 'test name')
is_deeply_ok(got_complex_structure, expected_complex_structure, 'test name')
isa_ok(obj, cls)
throws_ok(lambda: int('x'), ValueError, 'test name')

with subtest('Parse'):
    is_ok(url.host, 'example.com', 'right host')

@subtest('Format')
def _():
    is_ok(str(url), 'http://example.com', 'right format')

diag("here's what went wrong")
note('what happens next')

done_testing()

Pyjo.Test is a module for writing test scripts which print their results in the Test Anything Protocol (TAP). It is inspired by Perl’s Test::More, which the tests of Mojolicious are written with, but it is not a port of it.

Each test script is a separate Python program, usually placed in the t directory.

Running tests¶

The test scripts run with Pyjo.Prove:

$ python -m Pyjo.Prove -I . -r t

or with Perl’s prove, as they print standard TAP:

$ PYTHONPATH=. prove --ext=py --exec=python -r t

pytest runs them too, each script as a test of pytest, through Pyjo.Test.pytest_plugin.

A single script runs on its own as well:

$ PYTHONPATH=. python t/Pyjo/URL.py

Plans¶

A test script declares how many tests it runs, either up front with plan() or at the end with done_testing(). A script that runs tests without a plan fails.

plan() is a context manager too, which issues the plan at the end of its block, as done_testing() does, unless the block raises an exception:

with plan():
    ok(True, 'first')
    ok(True, 'second')

Subtests¶

A subtest groups tests under a name. Its tests are numbered on their own and printed indented, and the subtest as a whole counts as one test of its parent, which passes when all of its tests pass. subtest() works as a context manager, as a decorator which runs the function at once, and as a function which runs a callback:

with subtest('Simple'):
    ok(True, 'first')

@subtest('Decorated')
def _():
    ok(True, 'second')

subtest('Callback', lambda: ok(True, 'third'))

A subtest has an implicit done_testing(). plan(skip_all=...) skips the rest of the subtest, and a subtest that runs no tests fails.

Exit status¶

The exit status of a test script is 0 when every test passed, the number of failed tests (up to 254), 254 when tests ran without a plan, and 255 when the number of tests does not match the plan, when the script died with an exception, or after bail_out().

Exceptions¶

After raise_failures(), the functions do not print TAP, so that they work in the tests of another test runner, as pytest or unittest, where Pyjo.Test.pytest_plugin calls it. A failed test raises AssertionError with the name of the test and the diagnostic message, plan(skip_all=...) raises unittest.SkipTest, which both runners report as a skipped test, or in a subtest leaves the subtest, and bail_out() raises BailOut. A subtest is a block, the plans are not checked, skip() does nothing, and diag() and note() print their messages, which the runner captures.

import unittest

import Pyjo.Test
import Pyjo.URL


def setUpModule():
    Pyjo.Test.raise_failures()


def tearDownModule():
    Pyjo.Test.raise_failures(False)


class TestURL(unittest.TestCase):
    def test_host(self):
        url = Pyjo.URL.new('HtTp://Example.Com')
        Pyjo.Test.is_ok(url.host, 'Example.Com', 'right host')

Functions¶

class Pyjo.Test.Subtest(name: str)¶: A subtest, returned by subtest() without a callback. It is a context manager and a decorator; after it ends, passed tells whether all of its tests passed.

class Pyjo.Test.BailOut¶: Exception raised by bail_out() after raise_failures(), with the reason.

class Pyjo.Test.Error¶: Exception raised for a misuse of the test functions, for example planning twice.

class Pyjo.Test.SkipAll¶: Exception raised by plan(skip_all=...) inside a subtest, to leave it. It is caught by subtest().

Pyjo.Test.bail_out(reason: str = '') → NoReturn¶

if not connected:
    bail_out('No database connection')

Stop all testing at once, here and in the test harness, which shows the reason. The exit status is 255.

Pyjo.Test.cmp_ok(got: Any, operator: str, expected: Any, test_name: str | None = None) → bool¶

# ok(got == expected)
cmp_ok(got, '==', expected, 'this == that')

Compare two values with a binary operator: ==, !=, <, <=, >, >=, is, is not, in or not in. The test passes if the comparison is true.

Pyjo.Test.diag(*messages: object) → bool¶

if not in_ok(users, 'foo', "There's a foo user"):
    diag("Since there's no foo, check that /etc/bar is set up right")

Print a diagnostic message to the standard error, which does not interfere with the test output. Returns False, so a failure is preserved.

Pyjo.Test.done_testing(how_many: int | None = None) → bool¶

ok(True)
done_testing()

ok(True)
done_testing(1)

Issue the plan when the tests are done, instead of declaring it up front with plan(). The number of tests is optional; when it is given, it has to match the number of tests run.

Pyjo.Test.fail_ok(test_name: str | None = None) → bool¶

fail_ok('This should not happen')

A test that always fails, the same as ok(False).

Pyjo.Test.in_ok(got: Any, elem: object, test_name: str | None = None) → bool¶

in_ok(['foo', 'bar'], 'foo', 'foo is in the list')

Check if the element is in the container. The same as ok(elem in got), with a better diagnostic message.

Pyjo.Test.is_deeply_ok(got: object, expected: object, test_name: str | None = None) → bool¶

is_deeply_ok([[1, 2], [3, [4, 5]]], [[1, 2], [3]], 'deep structure')

Compare two data structures, walking through lists, tuples, sets and dicts. If they differ, the diagnostic message shows where they start differing.

Pyjo.Test.is_ok(got: object, expected: object, test_name: str | None = None) → bool¶

is_ok(ultimate_answer(), 42, 'meaning of life')

Compare two values with ==.

Pyjo.Test.isa_ok(got: object, cls: type | tuple[type, ...], test_name: str | None = None) → bool¶

obj = SomeClass()
isa_ok(obj, SomeClass)
isa_ok(123, (int, float))

Check if the object is an instance of the class, or of one of the classes.

Pyjo.Test.isnt_ok(got: object, expected: object, test_name: str | None = None) → bool¶

isnt_ok(foo, '', 'got some foo')

Compare two values with !=.

Pyjo.Test.like_ok(got: Any, expected: _Pattern, test_name: str | None = None) → bool¶

like_ok(got, r'expected', 'test name')
like_ok(got, (r'expected', 'i'), 'test name')
like_ok(got, re.compile(r'expected'), 'test name')

Check if the value matches the regular expression. The expression can be compiled already, or given with flags as a tuple: a, i, m, s and x.

Pyjo.Test.none_ok(got: object, test_name: str | None = None) → bool¶

none_ok(req.headers.transfer_encoding, 'no "Transfer-Encoding" value')

Check if the value is None.

Pyjo.Test.not_in_ok(got: Any, elem: object, test_name: str | None = None) → bool¶

not_in_ok(os.environ, 'DISPLAY', 'DISPLAY is not set')

The opposite of in_ok(): check if the element is not in the container.

Pyjo.Test.note(*messages: object) → None¶

note(f'Tempfile is {tempfile}')

Print a message to the standard output as a TAP comment, which the test harness hides unless asked to show it.

Pyjo.Test.ok(check: object, test_name: str | None = None) → bool¶

ok(3**4 == 81, 'simple exponential')
ok(isinstance(film, Film), 'set_db()')
ok(None not in items, 'all items defined')

The basic test: it passes if the value is true and fails otherwise. Returns whether the test passed.

Pyjo.Test.pass_ok(test_name: str | None = None) → bool¶

pass_ok('step 1')

A test that always passes, the same as ok(True).

Pyjo.Test.plan(tests: int | None = None, skip_all: str | None = None) → Plan¶

plan(tests=3)

if not os.getenv('TEST_ONLINE'):
    plan(skip_all='set TEST_ONLINE to enable this test')

with plan():
    ok(True)

with plan(tests=1):
    ok(True)

Declare how many tests the script is going to run, which protects against a premature exit, or skip all of them with a reason. In a subtest, skip_all skips the rest of the subtest.

It returns a Plan, a context manager which calls done_testing() at the end of its block, so that the number of tests can be left out.

Pyjo.Test.plan_skip_all(why: str) → Plan¶

plan_skip_all('set TEST_ONLINE to enable this test')

The same as plan(skip_all=why).

Pyjo.Test.plan_tests(how_many: int) → Plan¶

plan_tests(3)

The same as plan(tests=how_many).

Pyjo.Test.raise_failures(enable: bool = True) → None¶

raise_failures()
raise_failures(False)

Make the functions raise exceptions instead of printing TAP, as the section Exceptions describes, or print TAP again with False.

Pyjo.Test.skip(why: str | None = None, how_many: int = 1) → bool¶

if os.getenv('TEST_ONLINE'):
    ok(Pyjo.UserAgent.new().get('https://mojolicious.org').result.is_success, 'online request')
else:
    skip('set TEST_ONLINE to enable this test', 1)

Skip the given number of tests, which still count for the plan.

Pyjo.Test.subtest(name: str, fn: None = None) → Subtest¶

Pyjo.Test.subtest(name: str, fn: Callable[[], object]) → bool

with subtest('Parse'):
    ok(True)

@subtest('Parse')
def _():
    ok(True)

passed = subtest('Parse', lambda: ok(True))

Run a group of tests as a single test of the parent. Without a callback it returns a Subtest, which works as a context manager and as a decorator; with a callback it runs it at once and returns whether the subtest passed.

Pyjo.Test.throws_ok(cb: Callable[[], object], expected: type[BaseException] | str | Pattern[str], test_name: str | None = None) → bool¶

throws_ok(lambda: [1, 2, 3][3], IndexError, 'no more elements')
throws_ok(lambda: int('x'), "invalid literal for int() with base 10: 'x'", 'right message')
throws_ok(lambda: int('x'), re.compile(r'invalid literal'), 'right message')

Check if the callback raises an exception: of the given class, with the given message, or with a message matching the regular expression.

Pyjo.Test.unlike_ok(got: Any, expected: _Pattern, test_name: str | None = None) → bool¶

unlike_ok(got, r'unexpected', 'test name')

The opposite of like_ok(): check if the value does not match the regular expression.