unittest.rst 90.3 KB
Newer Older
1 2 3 4 5 6 7 8 9 10
:mod:`unittest` --- Unit testing framework
==========================================

.. module:: unittest
   :synopsis: Unit testing framework for Python.
.. moduleauthor:: Steve Purcell <stephen_purcell@yahoo.com>
.. sectionauthor:: Steve Purcell <stephen_purcell@yahoo.com>
.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
.. sectionauthor:: Raymond Hettinger <python@rcn.com>

11 12
(If you are already familiar with the basic concepts of testing, you might want
to skip to :ref:`the list of assert methods <assert-methods>`.)
13

14 15 16 17 18
The :mod:`unittest` unit testing framework was originally inspired by JUnit
and has a similar flavor as major unit testing frameworks in other
languages.  It supports test automation, sharing of setup and shutdown code
for tests, aggregation of tests into collections, and independence of the
tests from the reporting framework.
19

20 21
To achieve this, :mod:`unittest` supports some important concepts in an
object-oriented way:
22 23 24 25 26 27 28 29

test fixture
   A :dfn:`test fixture` represents the preparation needed to perform one or more
   tests, and any associate cleanup actions.  This may involve, for example,
   creating temporary or proxy databases, directories, or starting a server
   process.

test case
30
   A :dfn:`test case` is the individual unit of testing.  It checks for a specific
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
   response to a particular set of inputs.  :mod:`unittest` provides a base class,
   :class:`TestCase`, which may be used to create new test cases.

test suite
   A :dfn:`test suite` is a collection of test cases, test suites, or both.  It is
   used to aggregate tests that should be executed together.

test runner
   A :dfn:`test runner` is a component which orchestrates the execution of tests
   and provides the outcome to the user.  The runner may use a graphical interface,
   a textual interface, or return a special value to indicate the results of
   executing the tests.


.. seealso::

   Module :mod:`doctest`
      Another test-support module with a very different flavor.

50
   `Simple Smalltalk Testing: With Patterns <https://web.archive.org/web/20150315073817/http://www.xprogramming.com/testfram.htm>`_
51 52
      Kent Beck's original paper on testing frameworks using the pattern shared
      by :mod:`unittest`.
53

54
   `Nose <https://nose.readthedocs.org/en/latest/>`_ and `py.test <http://pytest.org>`_
55 56
      Third-party unittest frameworks with a lighter-weight syntax for writing
      tests.  For example, ``assert func(10) == 42``.
57

58
   `The Python Testing Tools Taxonomy <https://wiki.python.org/moin/PythonTestingToolsTaxonomy>`_
Benjamin Peterson's avatar
Benjamin Peterson committed
59 60
      An extensive list of Python testing tools including functional testing
      frameworks and mock object libraries.
61

Benjamin Peterson's avatar
Benjamin Peterson committed
62 63 64
   `Testing in Python Mailing List <http://lists.idyll.org/listinfo/testing-in-python>`_
      A special-interest-group for discussion of testing, and testing tools,
      in Python.
65

66 67
   The script :file:`Tools/unittestgui/unittestgui.py` in the Python source distribution is
   a GUI tool for test discovery and execution.  This is intended largely for ease of use
68 69
   for those new to unit testing.  For production environments it is
   recommended that tests be driven by a continuous integration system such as
70
   `Buildbot <https://buildbot.net/>`_, `Jenkins <https://jenkins.io/>`_
71
   or  `Hudson <http://hudson-ci.org/>`_.
72 73


74 75 76 77 78 79 80 81 82
.. _unittest-minimal-example:

Basic example
-------------

The :mod:`unittest` module provides a rich set of tools for constructing and
running tests.  This section demonstrates that a small subset of the tools
suffice to meet the needs of most users.

83
Here is a short script to test three string methods::
84

85
  import unittest
86

87
  class TestStringMethods(unittest.TestCase):
88

89 90
    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')
91

92 93 94
    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())
95

96 97 98 99 100 101
    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)
102

103 104
  if __name__ == '__main__':
      unittest.main()
105 106


107
A testcase is created by subclassing :class:`unittest.TestCase`.  The three
108 109 110 111
individual tests are defined with methods whose names start with the letters
``test``.  This naming convention informs the test runner about which methods
represent tests.

112
The crux of each test is a call to :meth:`~TestCase.assertEqual` to check for an
113 114 115 116 117
expected result; :meth:`~TestCase.assertTrue` or :meth:`~TestCase.assertFalse`
to verify a condition; or :meth:`~TestCase.assertRaises` to verify that a
specific exception gets raised.  These methods are used instead of the
:keyword:`assert` statement so the test runner can accumulate all test results
and produce a report.
118

119 120
The :meth:`~TestCase.setUp` and :meth:`~TestCase.tearDown` methods allow you
to define instructions that will be executed before and after each test method.
121
They are covered in more detail in the section :ref:`organizing-tests`.
122 123

The final block shows a simple way to run the tests. :func:`unittest.main`
124
provides a command-line interface to the test script.  When run from the command
125 126 127 128 129 130 131 132
line, the above script produces an output that looks like this::

   ...
   ----------------------------------------------------------------------
   Ran 3 tests in 0.000s

   OK

133 134
Passing the ``-v`` option to your test script will instruct :func:`unittest.main`
to enable a higher level of verbosity, and produce the following output::
135

136 137 138
   test_isupper (__main__.TestStringMethods) ... ok
   test_split (__main__.TestStringMethods) ... ok
   test_upper (__main__.TestStringMethods) ... ok
139 140

   ----------------------------------------------------------------------
141
   Ran 3 tests in 0.001s
142 143 144 145 146 147 148

   OK

The above examples show the most commonly used :mod:`unittest` features which
are sufficient to meet many everyday testing needs.  The remainder of the
documentation explores the full feature set from first principles.

Benjamin Peterson's avatar
Benjamin Peterson committed
149 150 151

.. _unittest-command-line-interface:

152
Command-Line Interface
Benjamin Peterson's avatar
Benjamin Peterson committed
153 154 155 156 157 158 159 160 161 162 163 164
----------------------

The unittest module can be used from the command line to run tests from
modules, classes or even individual test methods::

   python -m unittest test_module1 test_module2
   python -m unittest test_module.TestClass
   python -m unittest test_module.TestClass.test_method

You can pass in a list with any combination of module names, and fully
qualified class or method names.

165 166 167 168 169 170 171 172 173 174
Test modules can be specified by file path as well::

   python -m unittest tests/test_something.py

This allows you to use the shell filename completion to specify the test module.
The file specified must still be importable as a module. The path is converted
to a module name by removing the '.py' and converting path separators into '.'.
If you want to execute a test file that isn't importable as a module you should
execute the file directly instead.

Benjamin Peterson's avatar
Benjamin Peterson committed
175 176 177 178
You can run tests with more detail (higher verbosity) by passing in the -v flag::

   python -m unittest -v test_module

179 180 181 182
When executed without arguments :ref:`unittest-test-discovery` is started::

   python -m unittest

183
For a list of all the command-line options::
Benjamin Peterson's avatar
Benjamin Peterson committed
184 185 186

   python -m unittest -h

187
.. versionchanged:: 3.2
Benjamin Peterson's avatar
Benjamin Peterson committed
188 189 190 191
   In earlier versions it was only possible to run individual test methods and
   not modules or classes.


192 193
Command-line options
~~~~~~~~~~~~~~~~~~~~
Benjamin Peterson's avatar
Benjamin Peterson committed
194

195
:program:`unittest` supports these command-line options:
Benjamin Peterson's avatar
Benjamin Peterson committed
196

197
.. program:: unittest
Benjamin Peterson's avatar
Benjamin Peterson committed
198

199
.. cmdoption:: -b, --buffer
Benjamin Peterson's avatar
Benjamin Peterson committed
200

201 202 203
   The standard output and standard error streams are buffered during the test
   run. Output during a passing test is discarded. Output is echoed normally
   on test fail or error and is added to the failure messages.
Benjamin Peterson's avatar
Benjamin Peterson committed
204

205
.. cmdoption:: -c, --catch
Benjamin Peterson's avatar
Benjamin Peterson committed
206

207 208
   :kbd:`Control-C` during the test run waits for the current test to end and then
   reports all the results so far. A second :kbd:`Control-C` raises the normal
209
   :exc:`KeyboardInterrupt` exception.
Benjamin Peterson's avatar
Benjamin Peterson committed
210

211
   See `Signal Handling`_ for the functions that provide this functionality.
Benjamin Peterson's avatar
Benjamin Peterson committed
212

213 214 215
.. cmdoption:: -f, --failfast

   Stop the test run on the first error or failure.
Benjamin Peterson's avatar
Benjamin Peterson committed
216

217 218 219 220
.. cmdoption:: --locals

   Show local variables in tracebacks.

221
.. versionadded:: 3.2
222
   The command-line options ``-b``, ``-c`` and ``-f`` were added.
Benjamin Peterson's avatar
Benjamin Peterson committed
223

224 225 226
.. versionadded:: 3.5
   The command-line option ``--locals``.

Benjamin Peterson's avatar
Benjamin Peterson committed
227 228 229 230 231 232 233 234 235
The command line can also be used for test discovery, for running all of the
tests in a project or just a subset.


.. _unittest-test-discovery:

Test Discovery
--------------

236
.. versionadded:: 3.2
Benjamin Peterson's avatar
Benjamin Peterson committed
237

238 239
Unittest supports simple test discovery. In order to be compatible with test
discovery, all of the test files must be :ref:`modules <tut-modules>` or
240 241 242 243
:ref:`packages <tut-packages>` (including :term:`namespace packages
<namespace package>`) importable from the top-level directory of
the project (this means that their filenames must be valid :ref:`identifiers
<identifiers>`).
Benjamin Peterson's avatar
Benjamin Peterson committed
244 245

Test discovery is implemented in :meth:`TestLoader.discover`, but can also be
246
used from the command line. The basic command-line usage is::
Benjamin Peterson's avatar
Benjamin Peterson committed
247 248 249 250

   cd project_directory
   python -m unittest discover

251 252 253 254
.. note::

   As a shortcut, ``python -m unittest`` is the equivalent of
   ``python -m unittest discover``. If you want to pass arguments to test
255
   discovery the ``discover`` sub-command must be used explicitly.
256

Benjamin Peterson's avatar
Benjamin Peterson committed
257 258
The ``discover`` sub-command has the following options:

259 260 261 262 263 264
.. program:: unittest discover

.. cmdoption:: -v, --verbose

   Verbose output

265
.. cmdoption:: -s, --start-directory directory
266

267
   Directory to start discovery (``.`` default)
268

269
.. cmdoption:: -p, --pattern pattern
270

271
   Pattern to match test files (``test*.py`` default)
272

273
.. cmdoption:: -t, --top-level-directory directory
274 275

   Top level directory of project (defaults to start directory)
Benjamin Peterson's avatar
Benjamin Peterson committed
276

Benjamin Peterson's avatar
Benjamin Peterson committed
277 278 279
The :option:`-s`, :option:`-p`, and :option:`-t` options can be passed in
as positional arguments in that order. The following two command lines
are equivalent::
Benjamin Peterson's avatar
Benjamin Peterson committed
280

281 282
   python -m unittest discover -s project_directory -p "*_test.py"
   python -m unittest discover project_directory "*_test.py"
Benjamin Peterson's avatar
Benjamin Peterson committed
283

284 285 286 287 288 289 290
As well as being a path it is possible to pass a package name, for example
``myproject.subpackage.test``, as the start directory. The package name you
supply will then be imported and its location on the filesystem will be used
as the start directory.

.. caution::

291 292 293
    Test discovery loads tests by importing them. Once test discovery has found
    all the test files from the start directory you specify it turns the paths
    into package names to import. For example :file:`foo/bar/baz.py` will be
294 295 296 297 298 299 300 301 302 303 304
    imported as ``foo.bar.baz``.

    If you have a package installed globally and attempt test discovery on
    a different copy of the package then the import *could* happen from the
    wrong place. If this happens test discovery will warn you and exit.

    If you supply the start directory as a package name rather than a
    path to a directory then discover assumes that whichever location it
    imports from is the location you intended, so you will not get the
    warning.

Benjamin Peterson's avatar
Benjamin Peterson committed
305 306 307
Test modules and packages can customize test loading and discovery by through
the `load_tests protocol`_.

308 309 310
.. versionchanged:: 3.4
   Test discovery supports :term:`namespace packages <namespace package>`.

Benjamin Peterson's avatar
Benjamin Peterson committed
311

312 313 314 315 316 317 318
.. _organizing-tests:

Organizing test code
--------------------

The basic building blocks of unit testing are :dfn:`test cases` --- single
scenarios that must be set up and checked for correctness.  In :mod:`unittest`,
319 320 321
test cases are represented by :class:`unittest.TestCase` instances.
To make your own test cases you must write subclasses of
:class:`TestCase` or use :class:`FunctionTestCase`.
322 323 324 325 326

The testing code of a :class:`TestCase` instance should be entirely self
contained, such that it can be run either in isolation or in arbitrary
combination with any number of other test cases.

327 328 329
The simplest :class:`TestCase` subclass will simply implement a test method
(i.e. a method whose name starts with ``test``) in order to perform specific
testing code::
330 331 332 333

   import unittest

   class DefaultWidgetSizeTestCase(unittest.TestCase):
334
       def test_default_widget_size(self):
335
           widget = Widget('The widget')
336
           self.assertEqual(widget.size(), (50, 50))
337

338
Note that in order to test something, we use one of the :meth:`assert\*`
339 340
methods provided by the :class:`TestCase` base class.  If the test fails, an
exception will be raised, and :mod:`unittest` will identify the test case as a
341
:dfn:`failure`.  Any other exceptions will be treated as :dfn:`errors`.
342

343 344 345 346
Tests can be numerous, and their set-up can be repetitive.  Luckily, we
can factor out set-up code by implementing a method called
:meth:`~TestCase.setUp`, which the testing framework will automatically
call for every single test we run::
347 348 349 350 351 352 353

   import unittest

   class SimpleWidgetTestCase(unittest.TestCase):
       def setUp(self):
           self.widget = Widget('The widget')

354
       def test_default_widget_size(self):
355 356
           self.assertEqual(self.widget.size(), (50,50),
                            'incorrect default size')
357

358
       def test_widget_resize(self):
359
           self.widget.resize(100,150)
360 361
           self.assertEqual(self.widget.size(), (100,150),
                            'wrong size after resize')
362

363 364 365 366 367
.. note::
   The order in which the various tests will be run is determined
   by sorting the test method names with respect to the built-in
   ordering for strings.

368
If the :meth:`~TestCase.setUp` method raises an exception while the test is
369 370
running, the framework will consider the test to have suffered an error, and
the test method will not be executed.
371

372
Similarly, we can provide a :meth:`~TestCase.tearDown` method that tidies up
373
after the test method has been run::
374 375 376 377 378 379 380 381 382 383

   import unittest

   class SimpleWidgetTestCase(unittest.TestCase):
       def setUp(self):
           self.widget = Widget('The widget')

       def tearDown(self):
           self.widget.dispose()

384 385
If :meth:`~TestCase.setUp` succeeded, :meth:`~TestCase.tearDown` will be
run whether the test method succeeded or not.
386 387 388 389 390

Such a working environment for the testing code is called a :dfn:`fixture`.

Test case instances are grouped together according to the features they test.
:mod:`unittest` provides a mechanism for this: the :dfn:`test suite`,
391 392 393
represented by :mod:`unittest`'s :class:`TestSuite` class.  In most cases,
calling :func:`unittest.main` will do the right thing and collect all the
module's test cases for you, and then execute them.
394

395 396
However, should you want to customize the building of your test suite,
you can do it yourself::
397 398 399

   def suite():
       suite = unittest.TestSuite()
400 401
       suite.addTest(WidgetTestCase('test_default_size'))
       suite.addTest(WidgetTestCase('test_resize'))
402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445
       return suite

You can place the definitions of test cases and test suites in the same modules
as the code they are to test (such as :file:`widget.py`), but there are several
advantages to placing the test code in a separate module, such as
:file:`test_widget.py`:

* The test module can be run standalone from the command line.

* The test code can more easily be separated from shipped code.

* There is less temptation to change test code to fit the code it tests without
  a good reason.

* Test code should be modified much less frequently than the code it tests.

* Tested code can be refactored more easily.

* Tests for modules written in C must be in separate modules anyway, so why not
  be consistent?

* If the testing strategy changes, there is no need to change the source code.


.. _legacy-unit-tests:

Re-using old test code
----------------------

Some users will find that they have existing test code that they would like to
run from :mod:`unittest`, without converting every old test function to a
:class:`TestCase` subclass.

For this reason, :mod:`unittest` provides a :class:`FunctionTestCase` class.
This subclass of :class:`TestCase` can be used to wrap an existing test
function.  Set-up and tear-down functions can also be provided.

Given the following test function::

   def testSomething():
       something = makeSomething()
       assert something.name is not None
       # ...

446 447
one can create an equivalent test case instance as follows, with optional
set-up and tear-down methods::
448 449 450 451 452 453 454

   testcase = unittest.FunctionTestCase(testSomething,
                                        setUp=makeSomethingDB,
                                        tearDown=deleteSomethingDB)

.. note::

455 456 457 458
   Even though :class:`FunctionTestCase` can be used to quickly convert an
   existing test base over to a :mod:`unittest`\ -based system, this approach is
   not recommended.  Taking the time to set up proper :class:`TestCase`
   subclasses will make future test refactorings infinitely easier.
459

460 461 462 463 464
In some cases, the existing tests may have been written using the :mod:`doctest`
module.  If so, :mod:`doctest` provides a :class:`DocTestSuite` class that can
automatically build :class:`unittest.TestSuite` instances from the existing
:mod:`doctest`\ -based tests.

465

466 467 468 469 470
.. _unittest-skipping:

Skipping tests and expected failures
------------------------------------

471 472
.. versionadded:: 3.1

473
Unittest supports skipping individual test methods and even whole classes of
474
tests.  In addition, it supports marking a test as an "expected failure," a test
475 476 477 478 479 480
that is broken and will fail, but shouldn't be counted as a failure on a
:class:`TestResult`.

Skipping a test is simply a matter of using the :func:`skip` :term:`decorator`
or one of its conditional variants.

Ezio Melotti's avatar
Ezio Melotti committed
481
Basic skipping looks like this::
482 483 484 485 486 487 488

   class MyTestCase(unittest.TestCase):

       @unittest.skip("demonstrating skipping")
       def test_nothing(self):
           self.fail("shouldn't happen")

489 490
       @unittest.skipIf(mylib.__version__ < (1, 3),
                        "not supported in this library version")
491 492 493 494 495 496 497 498 499
       def test_format(self):
           # Tests that work for only a certain version of the library.
           pass

       @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
       def test_windows_support(self):
           # windows specific testing code
           pass

Ezio Melotti's avatar
Ezio Melotti committed
500
This is the output of running the example above in verbose mode::
501

502
   test_format (__main__.MyTestCase) ... skipped 'not supported in this library version'
503
   test_nothing (__main__.MyTestCase) ... skipped 'demonstrating skipping'
504
   test_windows_support (__main__.MyTestCase) ... skipped 'requires Windows'
505 506

   ----------------------------------------------------------------------
507 508 509
   Ran 3 tests in 0.005s

   OK (skipped=3)
510

Ezio Melotti's avatar
Ezio Melotti committed
511
Classes can be skipped just like methods::
512

513
   @unittest.skip("showing class skipping")
514 515 516 517
   class MySkippedTestCase(unittest.TestCase):
       def test_not_run(self):
           pass

518 519 520
:meth:`TestCase.setUp` can also skip the test.  This is useful when a resource
that needs to be set up is not available.

521 522 523 524 525 526 527 528 529
Expected failures use the :func:`expectedFailure` decorator. ::

   class ExpectedFailureTestCase(unittest.TestCase):
       @unittest.expectedFailure
       def test_fail(self):
           self.assertEqual(1, 0, "broken")

It's easy to roll your own skipping decorators by making a decorator that calls
:func:`skip` on the test when it wants it to be skipped.  This decorator skips
Ezio Melotti's avatar
Ezio Melotti committed
530
the test unless the passed object has a certain attribute::
531 532 533 534

   def skipUnlessHasattr(obj, attr):
       if hasattr(obj, attr):
           return lambda func: func
535
       return unittest.skip("{!r} doesn't have {!r}".format(obj, attr))
536 537 538

The following decorators implement test skipping and expected failures:

539
.. decorator:: skip(reason)
540 541 542 543

   Unconditionally skip the decorated test.  *reason* should describe why the
   test is being skipped.

544
.. decorator:: skipIf(condition, reason)
545 546 547

   Skip the decorated test if *condition* is true.

548
.. decorator:: skipUnless(condition, reason)
549

Georg Brandl's avatar
Georg Brandl committed
550
   Skip the decorated test unless *condition* is true.
551

552
.. decorator:: expectedFailure
553 554 555 556

   Mark the test as an expected failure.  If the test fails when run, the test
   is not counted as a failure.

557 558 559 560 561 562 563
.. exception:: SkipTest(reason)

   This exception is raised to skip a test.

   Usually you can use :meth:`TestCase.skipTest` or one of the skipping
   decorators instead of raising this directly.

564 565 566
Skipped tests will not have :meth:`~TestCase.setUp` or :meth:`~TestCase.tearDown` run around them.
Skipped classes will not have :meth:`~TestCase.setUpClass` or :meth:`~TestCase.tearDownClass` run.
Skipped modules will not have :func:`setUpModule` or :func:`tearDownModule` run.
Benjamin Peterson's avatar
Benjamin Peterson committed
567

568

569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630
.. _subtests:

Distinguishing test iterations using subtests
---------------------------------------------

.. versionadded:: 3.4

When some of your tests differ only by a some very small differences, for
instance some parameters, unittest allows you to distinguish them inside
the body of a test method using the :meth:`~TestCase.subTest` context manager.

For example, the following test::

   class NumbersTest(unittest.TestCase):

       def test_even(self):
           """
           Test that numbers between 0 and 5 are all even.
           """
           for i in range(0, 6):
               with self.subTest(i=i):
                   self.assertEqual(i % 2, 0)

will produce the following output::

   ======================================================================
   FAIL: test_even (__main__.NumbersTest) (i=1)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "subtests.py", line 32, in test_even
       self.assertEqual(i % 2, 0)
   AssertionError: 1 != 0

   ======================================================================
   FAIL: test_even (__main__.NumbersTest) (i=3)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "subtests.py", line 32, in test_even
       self.assertEqual(i % 2, 0)
   AssertionError: 1 != 0

   ======================================================================
   FAIL: test_even (__main__.NumbersTest) (i=5)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "subtests.py", line 32, in test_even
       self.assertEqual(i % 2, 0)
   AssertionError: 1 != 0

Without using a subtest, execution would stop after the first failure,
and the error would be less easy to diagnose because the value of ``i``
wouldn't be displayed::

   ======================================================================
   FAIL: test_even (__main__.NumbersTest)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "subtests.py", line 32, in test_even
       self.assertEqual(i % 2, 0)
   AssertionError: 1 != 0


631 632 633 634 635
.. _unittest-contents:

Classes and functions
---------------------

636 637 638 639 640 641 642
This section describes in depth the API of :mod:`unittest`.


.. _testcase-objects:

Test cases
~~~~~~~~~~
643

644
.. class:: TestCase(methodName='runTest')
645

646
   Instances of the :class:`TestCase` class represent the logical test units
647 648 649
   in the :mod:`unittest` universe.  This class is intended to be used as a base
   class, with specific tests being implemented by concrete subclasses.  This class
   implements the interface needed by the test runner to allow it to drive the
650
   tests, and methods that the test code can use to check for and report various
651 652
   kinds of failure.

653
   Each instance of :class:`TestCase` will run a single base method: the method
654 655
   named *methodName*.
   In most uses of :class:`TestCase`, you will neither change
656
   the *methodName* nor reimplement the default ``runTest()`` method.
657

658
   .. versionchanged:: 3.2
659 660 661
      :class:`TestCase` can be instantiated successfully without providing a
      *methodName*. This makes it easier to experiment with :class:`TestCase`
      from the interactive interpreter.
662

663 664 665 666
   :class:`TestCase` instances provide three groups of methods: one group used
   to run the test, another used by the test implementation to check conditions
   and report failures, and some inquiry methods allowing information about the
   test itself to be gathered.
667

668
   Methods in the first group (running the test) are:
669

670
   .. method:: setUp()
671

672
      Method called to prepare the test fixture.  This is called immediately
Terry Jan Reedy's avatar
Terry Jan Reedy committed
673 674
      before calling the test method; other than :exc:`AssertionError` or :exc:`SkipTest`,
      any exception raised by this method will be considered an error rather than
675
      a test failure. The default implementation does nothing.
676 677


678
   .. method:: tearDown()
679

680 681 682
      Method called immediately after the test method has been called and the
      result recorded.  This is called even if the test method raised an
      exception, so the implementation in subclasses may need to be particularly
683 684 685 686 687 688
      careful about checking internal state.  Any exception, other than
      :exc:`AssertionError` or :exc:`SkipTest`, raised by this method will be
      considered an additional error rather than a test failure (thus increasing
      the total number of reported errors). This method will only be called if
      the :meth:`setUp` succeeds, regardless of the outcome of the test method.
      The default implementation does nothing.
689 690


Benjamin Peterson's avatar
Benjamin Peterson committed
691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720
   .. method:: setUpClass()

      A class method called before tests in an individual class run.
      ``setUpClass`` is called with the class as the only argument
      and must be decorated as a :func:`classmethod`::

        @classmethod
        def setUpClass(cls):
            ...

      See `Class and Module Fixtures`_ for more details.

      .. versionadded:: 3.2


   .. method:: tearDownClass()

      A class method called after tests in an individual class have run.
      ``tearDownClass`` is called with the class as the only argument
      and must be decorated as a :meth:`classmethod`::

        @classmethod
        def tearDownClass(cls):
            ...

      See `Class and Module Fixtures`_ for more details.

      .. versionadded:: 3.2


721
   .. method:: run(result=None)
722

723 724 725 726 727
      Run the test, collecting the result into the :class:`TestResult` object
      passed as *result*.  If *result* is omitted or ``None``, a temporary
      result object is created (by calling the :meth:`defaultTestResult`
      method) and used. The result object is returned to :meth:`run`'s
      caller.
728

729 730
      The same effect may be had by simply calling the :class:`TestCase`
      instance.
731

732 733 734
      .. versionchanged:: 3.3
         Previous versions of ``run`` did not return the result. Neither did
         calling an instance.
735

Benjamin Peterson's avatar
Benjamin Peterson committed
736
   .. method:: skipTest(reason)
737

738
      Calling this during a test method or :meth:`setUp` skips the current
739
      test.  See :ref:`unittest-skipping` for more information.
740

741
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
742

743

744 745 746 747 748 749 750 751 752 753 754 755 756 757 758
   .. method:: subTest(msg=None, **params)

      Return a context manager which executes the enclosed code block as a
      subtest.  *msg* and *params* are optional, arbitrary values which are
      displayed whenever a subtest fails, allowing you to identify them
      clearly.

      A test case can contain any number of subtest declarations, and
      they can be arbitrarily nested.

      See :ref:`subtests` for more information.

      .. versionadded:: 3.4


759
   .. method:: debug()
760

761 762 763
      Run the test without collecting the result.  This allows exceptions raised
      by the test to be propagated to the caller, and can be used to support
      running tests under a debugger.
764

765
   .. _assert-methods:
766

767 768 769
   The :class:`TestCase` class provides several assert methods to check for and
   report failures.  The following table lists the most commonly used methods
   (see the tables below for more assert methods):
770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809

   +-----------------------------------------+-----------------------------+---------------+
   | Method                                  | Checks that                 | New in        |
   +=========================================+=============================+===============+
   | :meth:`assertEqual(a, b)                | ``a == b``                  |               |
   | <TestCase.assertEqual>`                 |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertNotEqual(a, b)             | ``a != b``                  |               |
   | <TestCase.assertNotEqual>`              |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertTrue(x)                    | ``bool(x) is True``         |               |
   | <TestCase.assertTrue>`                  |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertFalse(x)                   | ``bool(x) is False``        |               |
   | <TestCase.assertFalse>`                 |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertIs(a, b)                   | ``a is b``                  | 3.1           |
   | <TestCase.assertIs>`                    |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertIsNot(a, b)                | ``a is not b``              | 3.1           |
   | <TestCase.assertIsNot>`                 |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertIsNone(x)                  | ``x is None``               | 3.1           |
   | <TestCase.assertIsNone>`                |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertIsNotNone(x)               | ``x is not None``           | 3.1           |
   | <TestCase.assertIsNotNone>`             |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertIn(a, b)                   | ``a in b``                  | 3.1           |
   | <TestCase.assertIn>`                    |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertNotIn(a, b)                | ``a not in b``              | 3.1           |
   | <TestCase.assertNotIn>`                 |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertIsInstance(a, b)           | ``isinstance(a, b)``        | 3.2           |
   | <TestCase.assertIsInstance>`            |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
   | :meth:`assertNotIsInstance(a, b)        | ``not isinstance(a, b)``    | 3.2           |
   | <TestCase.assertNotIsInstance>`         |                             |               |
   +-----------------------------------------+-----------------------------+---------------+
810

811 812 813 814 815
   All the assert methods accept a *msg* argument that, if specified, is used
   as the error message on failure (see also :data:`longMessage`).
   Note that the *msg* keyword argument can be passed to :meth:`assertRaises`,
   :meth:`assertRaisesRegex`, :meth:`assertWarns`, :meth:`assertWarnsRegex`
   only when they are used as a context manager.
816

817
   .. method:: assertEqual(first, second, msg=None)
818

819
      Test that *first* and *second* are equal.  If the values do not
820
      compare equal, the test will fail.
821

822
      In addition, if *first* and *second* are the exact same type and one of
823
      list, tuple, dict, set, frozenset or str or any type that a subclass
824
      registers with :meth:`addTypeEqualityFunc` the type-specific equality
825
      function will be called in order to generate a more useful default
826 827
      error message (see also the :ref:`list of type-specific methods
      <type-specific-methods>`).
828

829
      .. versionchanged:: 3.1
830
         Added the automatic calling of type-specific equality function.
831

832 833 834
      .. versionchanged:: 3.2
         :meth:`assertMultiLineEqual` added as the default type equality
         function for comparing strings.
835

836

837
   .. method:: assertNotEqual(first, second, msg=None)
838

839
      Test that *first* and *second* are not equal.  If the values do
840
      compare equal, the test will fail.
841

842
   .. method:: assertTrue(expr, msg=None)
843
               assertFalse(expr, msg=None)
Benjamin Peterson's avatar
Benjamin Peterson committed
844

845
      Test that *expr* is true (or false).
Benjamin Peterson's avatar
Benjamin Peterson committed
846

847 848 849 850 851
      Note that this is equivalent to ``bool(expr) is True`` and not to ``expr
      is True`` (use ``assertIs(expr, True)`` for the latter).  This method
      should also be avoided when more specific methods are available (e.g.
      ``assertEqual(a, b)`` instead of ``assertTrue(a == b)``), because they
      provide a better error message in case of failure.
Benjamin Peterson's avatar
Benjamin Peterson committed
852

853

854 855
   .. method:: assertIs(first, second, msg=None)
               assertIsNot(first, second, msg=None)
856

857
      Test that *first* and *second* evaluate (or don't evaluate) to the
858
      same object.
859

860
      .. versionadded:: 3.1
861 862


863
   .. method:: assertIsNone(expr, msg=None)
864
               assertIsNotNone(expr, msg=None)
865

866
      Test that *expr* is (or is not) None.
Benjamin Peterson's avatar
Benjamin Peterson committed
867

868
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
869 870


871 872 873
   .. method:: assertIn(first, second, msg=None)
               assertNotIn(first, second, msg=None)

874
      Test that *first* is (or is not) in *second*.
875

876
      .. versionadded:: 3.1
877 878


879
   .. method:: assertIsInstance(obj, cls, msg=None)
880
               assertNotIsInstance(obj, cls, msg=None)
881

882 883
      Test that *obj* is (or is not) an instance of *cls* (which can be a
      class or a tuple of classes, as supported by :func:`isinstance`).
884
      To check for the exact type, use :func:`assertIs(type(obj), cls) <assertIs>`.
885

886
      .. versionadded:: 3.2
887 888 889



890
   It is also possible to check the production of exceptions, warnings, and
Antoine Pitrou's avatar
Antoine Pitrou committed
891
   log messages using the following methods:
892

893 894 895
   +---------------------------------------------------------+--------------------------------------+------------+
   | Method                                                  | Checks that                          | New in     |
   +=========================================================+======================================+============+
896
   | :meth:`assertRaises(exc, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *exc*  |            |
897 898
   | <TestCase.assertRaises>`                                |                                      |            |
   +---------------------------------------------------------+--------------------------------------+------------+
899 900
   | :meth:`assertRaisesRegex(exc, r, fun, *args, **kwds)    | ``fun(*args, **kwds)`` raises *exc*  | 3.1        |
   | <TestCase.assertRaisesRegex>`                           | and the message matches regex *r*    |            |
901
   +---------------------------------------------------------+--------------------------------------+------------+
902
   | :meth:`assertWarns(warn, fun, *args, **kwds)            | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
903 904
   | <TestCase.assertWarns>`                                 |                                      |            |
   +---------------------------------------------------------+--------------------------------------+------------+
905 906
   | :meth:`assertWarnsRegex(warn, r, fun, *args, **kwds)    | ``fun(*args, **kwds)`` raises *warn* | 3.2        |
   | <TestCase.assertWarnsRegex>`                            | and the message matches regex *r*    |            |
907
   +---------------------------------------------------------+--------------------------------------+------------+
908 909
   | :meth:`assertLogs(logger, level)                        | The ``with`` block logs on *logger*  | 3.4        |
   | <TestCase.assertLogs>`                                  | with minimum *level*                 |            |
Antoine Pitrou's avatar
Antoine Pitrou committed
910
   +---------------------------------------------------------+--------------------------------------+------------+
911

912
   .. method:: assertRaises(exception, callable, *args, **kwds)
913
               assertRaises(exception, msg=None)
914

915 916 917 918 919 920
      Test that an exception is raised when *callable* is called with any
      positional or keyword arguments that are also passed to
      :meth:`assertRaises`.  The test passes if *exception* is raised, is an
      error if another exception is raised, or fails if no exception is raised.
      To catch any of a group of exceptions, a tuple containing the exception
      classes may be passed as *exception*.
921

922 923 924
      If only the *exception* and possibly the *msg* arguments are given,
      return a context manager so that the code under test can be written
      inline rather than as a function::
925

926
         with self.assertRaises(SomeException):
927 928
             do_something()

929 930 931
      When used as a context manager, :meth:`assertRaises` accepts the
      additional keyword argument *msg*.

932
      The context manager will store the caught exception object in its
933
      :attr:`exception` attribute.  This can be useful if the intention
934
      is to perform additional checks on the exception raised::
935

936 937
         with self.assertRaises(SomeException) as cm:
             do_something()
938

939 940
         the_exception = cm.exception
         self.assertEqual(the_exception.error_code, 3)
941

942
      .. versionchanged:: 3.1
943
         Added the ability to use :meth:`assertRaises` as a context manager.
944

945 946 947
      .. versionchanged:: 3.2
         Added the :attr:`exception` attribute.

948 949 950
      .. versionchanged:: 3.3
         Added the *msg* keyword argument when used as a context manager.

951

952
   .. method:: assertRaisesRegex(exception, regex, callable, *args, **kwds)
953
               assertRaisesRegex(exception, regex, msg=None)
954

955 956
      Like :meth:`assertRaises` but also tests that *regex* matches
      on the string representation of the raised exception.  *regex* may be
957 958 959
      a regular expression object or a string containing a regular expression
      suitable for use by :func:`re.search`.  Examples::

960
         self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$",
961
                                int, 'XYZ')
962 963 964

      or::

965
         with self.assertRaisesRegex(ValueError, 'literal'):
966 967
            int('XYZ')

968 969
      .. versionadded:: 3.1
         under the name ``assertRaisesRegexp``.
970

971
      .. versionchanged:: 3.2
972
         Renamed to :meth:`assertRaisesRegex`.
973

974 975 976
      .. versionchanged:: 3.3
         Added the *msg* keyword argument when used as a context manager.

977

978
   .. method:: assertWarns(warning, callable, *args, **kwds)
979
               assertWarns(warning, msg=None)
980 981 982 983

      Test that a warning is triggered when *callable* is called with any
      positional or keyword arguments that are also passed to
      :meth:`assertWarns`.  The test passes if *warning* is triggered and
984
      fails if it isn't.  Any exception is an error.
985 986 987
      To catch any of a group of warnings, a tuple containing the warning
      classes may be passed as *warnings*.

988
      If only the *warning* and possibly the *msg* arguments are given,
989
      return a context manager so that the code under test can be written
990
      inline rather than as a function::
991 992 993 994

         with self.assertWarns(SomeWarning):
             do_something()

995
      When used as a context manager, :meth:`assertWarns` accepts the
996 997
      additional keyword argument *msg*.

998 999 1000 1001
      The context manager will store the caught warning object in its
      :attr:`warning` attribute, and the source line which triggered the
      warnings in the :attr:`filename` and :attr:`lineno` attributes.
      This can be useful if the intention is to perform additional checks
1002
      on the warning caught::
1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014

         with self.assertWarns(SomeWarning) as cm:
             do_something()

         self.assertIn('myfile.py', cm.filename)
         self.assertEqual(320, cm.lineno)

      This method works regardless of the warning filters in place when it
      is called.

      .. versionadded:: 3.2

1015 1016 1017
      .. versionchanged:: 3.3
         Added the *msg* keyword argument when used as a context manager.

1018

1019
   .. method:: assertWarnsRegex(warning, regex, callable, *args, **kwds)
1020
               assertWarnsRegex(warning, regex, msg=None)
1021

1022 1023
      Like :meth:`assertWarns` but also tests that *regex* matches on the
      message of the triggered warning.  *regex* may be a regular expression
1024 1025 1026
      object or a string containing a regular expression suitable for use
      by :func:`re.search`.  Example::

1027 1028 1029
         self.assertWarnsRegex(DeprecationWarning,
                               r'legacy_function\(\) is deprecated',
                               legacy_function, 'XYZ')
1030 1031 1032

      or::

1033
         with self.assertWarnsRegex(RuntimeWarning, 'unsafe frobnicating'):
1034 1035 1036 1037
             frobnicate('/etc/passwd')

      .. versionadded:: 3.2

1038 1039
      .. versionchanged:: 3.3
         Added the *msg* keyword argument when used as a context manager.
1040

1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081
   .. method:: assertLogs(logger=None, level=None)

      A context manager to test that at least one message is logged on
      the *logger* or one of its children, with at least the given
      *level*.

      If given, *logger* should be a :class:`logging.Logger` object or a
      :class:`str` giving the name of a logger.  The default is the root
      logger, which will catch all messages.

      If given, *level* should be either a numeric logging level or
      its string equivalent (for example either ``"ERROR"`` or
      :attr:`logging.ERROR`).  The default is :attr:`logging.INFO`.

      The test passes if at least one message emitted inside the ``with``
      block matches the *logger* and *level* conditions, otherwise it fails.

      The object returned by the context manager is a recording helper
      which keeps tracks of the matching log messages.  It has two
      attributes:

      .. attribute:: records

         A list of :class:`logging.LogRecord` objects of the matching
         log messages.

      .. attribute:: output

         A list of :class:`str` objects with the formatted output of
         matching messages.

      Example::

         with self.assertLogs('foo', level='INFO') as cm:
            logging.getLogger('foo').info('first message')
            logging.getLogger('foo.bar').error('second message')
         self.assertEqual(cm.output, ['INFO:foo:first message',
                                      'ERROR:foo.bar:second message'])

      .. versionadded:: 3.4

1082

1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105
   There are also other methods used to perform more specific checks, such as:

   +---------------------------------------+--------------------------------+--------------+
   | Method                                | Checks that                    | New in       |
   +=======================================+================================+==============+
   | :meth:`assertAlmostEqual(a, b)        | ``round(a-b, 7) == 0``         |              |
   | <TestCase.assertAlmostEqual>`         |                                |              |
   +---------------------------------------+--------------------------------+--------------+
   | :meth:`assertNotAlmostEqual(a, b)     | ``round(a-b, 7) != 0``         |              |
   | <TestCase.assertNotAlmostEqual>`      |                                |              |
   +---------------------------------------+--------------------------------+--------------+
   | :meth:`assertGreater(a, b)            | ``a > b``                      | 3.1          |
   | <TestCase.assertGreater>`             |                                |              |
   +---------------------------------------+--------------------------------+--------------+
   | :meth:`assertGreaterEqual(a, b)       | ``a >= b``                     | 3.1          |
   | <TestCase.assertGreaterEqual>`        |                                |              |
   +---------------------------------------+--------------------------------+--------------+
   | :meth:`assertLess(a, b)               | ``a < b``                      | 3.1          |
   | <TestCase.assertLess>`                |                                |              |
   +---------------------------------------+--------------------------------+--------------+
   | :meth:`assertLessEqual(a, b)          | ``a <= b``                     | 3.1          |
   | <TestCase.assertLessEqual>`           |                                |              |
   +---------------------------------------+--------------------------------+--------------+
1106
   | :meth:`assertRegex(s, r)              | ``r.search(s)``                | 3.1          |
1107
   | <TestCase.assertRegex>`               |                                |              |
1108
   +---------------------------------------+--------------------------------+--------------+
1109
   | :meth:`assertNotRegex(s, r)           | ``not r.search(s)``            | 3.2          |
1110
   | <TestCase.assertNotRegex>`            |                                |              |
1111
   +---------------------------------------+--------------------------------+--------------+
1112
   | :meth:`assertCountEqual(a, b)         | *a* and *b* have the same      | 3.2          |
1113
   | <TestCase.assertCountEqual>`          | elements in the same number,   |              |
1114 1115
   |                                       | regardless of their order      |              |
   +---------------------------------------+--------------------------------+--------------+
1116

1117

1118 1119
   .. method:: assertAlmostEqual(first, second, places=7, msg=None, delta=None)
               assertNotAlmostEqual(first, second, places=7, msg=None, delta=None)
1120

1121
      Test that *first* and *second* are approximately (or not approximately)
1122 1123 1124 1125
      equal by computing the difference, rounding to the given number of
      decimal *places* (default 7), and comparing to zero.  Note that these
      methods round the values to the given number of *decimal places* (i.e.
      like the :func:`round` function) and not *significant digits*.
1126

1127
      If *delta* is supplied instead of *places* then the difference
1128
      between *first* and *second* must be less or equal to (or greater than) *delta*.
1129

1130
      Supplying both *delta* and *places* raises a ``TypeError``.
1131

1132
      .. versionchanged:: 3.2
1133 1134 1135
         :meth:`assertAlmostEqual` automatically considers almost equal objects
         that compare equal.  :meth:`assertNotAlmostEqual` automatically fails
         if the objects compare equal.  Added the *delta* keyword argument.
Benjamin Peterson's avatar
Benjamin Peterson committed
1136

1137

1138 1139 1140 1141
   .. method:: assertGreater(first, second, msg=None)
               assertGreaterEqual(first, second, msg=None)
               assertLess(first, second, msg=None)
               assertLessEqual(first, second, msg=None)
1142

1143
      Test that *first* is respectively >, >=, < or <= than *second* depending
1144
      on the method name.  If not, the test will fail::
1145 1146 1147

         >>> self.assertGreaterEqual(3, 4)
         AssertionError: "3" unexpectedly not greater than or equal to "4"
Benjamin Peterson's avatar
Benjamin Peterson committed
1148

Georg Brandl's avatar
Georg Brandl committed
1149
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
1150 1151


1152 1153
   .. method:: assertRegex(text, regex, msg=None)
               assertNotRegex(text, regex, msg=None)
Benjamin Peterson's avatar
Benjamin Peterson committed
1154

1155
      Test that a *regex* search matches (or does not match) *text*.  In case
1156
      of failure, the error message will include the pattern and the *text* (or
1157
      the pattern and the part of *text* that unexpectedly matched).  *regex*
1158 1159
      may be a regular expression object or a string containing a regular
      expression suitable for use by :func:`re.search`.
1160

1161 1162
      .. versionadded:: 3.1
         under the name ``assertRegexpMatches``.
1163
      .. versionchanged:: 3.2
1164 1165 1166 1167
         The method ``assertRegexpMatches()`` has been renamed to
         :meth:`.assertRegex`.
      .. versionadded:: 3.2
         :meth:`.assertNotRegex`.
1168 1169


1170
   .. method:: assertCountEqual(first, second, msg=None)
1171

1172
      Test that sequence *first* contains the same elements as *second*,
1173 1174 1175
      regardless of their order. When they don't, an error message listing the
      differences between the sequences will be generated.

1176 1177
      Duplicate elements are *not* ignored when comparing *first* and
      *second*. It verifies whether each element has the same count in both
1178
      sequences. Equivalent to:
1179
      ``assertEqual(Counter(list(first)), Counter(list(second)))``
1180
      but works with sequences of unhashable objects as well.
1181

1182 1183
      .. versionadded:: 3.2

1184

1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203
   .. _type-specific-methods:

   The :meth:`assertEqual` method dispatches the equality check for objects of
   the same type to different type-specific methods.  These methods are already
   implemented for most of the built-in types, but it's also possible to
   register new methods using :meth:`addTypeEqualityFunc`:

   .. method:: addTypeEqualityFunc(typeobj, function)

      Registers a type-specific method called by :meth:`assertEqual` to check
      if two objects of exactly the same *typeobj* (not subclasses) compare
      equal.  *function* must take two positional arguments and a third msg=None
      keyword argument just as :meth:`assertEqual` does.  It must raise
      :data:`self.failureException(msg) <failureException>` when inequality
      between the first two parameters is detected -- possibly providing useful
      information and explaining the inequalities in details in the error
      message.

      .. versionadded:: 3.1
1204

1205 1206 1207
   The list of type-specific methods automatically used by
   :meth:`~TestCase.assertEqual` are summarized in the following table.  Note
   that it's usually not necessary to invoke these methods directly.
1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232

   +-----------------------------------------+-----------------------------+--------------+
   | Method                                  | Used to compare             | New in       |
   +=========================================+=============================+==============+
   | :meth:`assertMultiLineEqual(a, b)       | strings                     | 3.1          |
   | <TestCase.assertMultiLineEqual>`        |                             |              |
   +-----------------------------------------+-----------------------------+--------------+
   | :meth:`assertSequenceEqual(a, b)        | sequences                   | 3.1          |
   | <TestCase.assertSequenceEqual>`         |                             |              |
   +-----------------------------------------+-----------------------------+--------------+
   | :meth:`assertListEqual(a, b)            | lists                       | 3.1          |
   | <TestCase.assertListEqual>`             |                             |              |
   +-----------------------------------------+-----------------------------+--------------+
   | :meth:`assertTupleEqual(a, b)           | tuples                      | 3.1          |
   | <TestCase.assertTupleEqual>`            |                             |              |
   +-----------------------------------------+-----------------------------+--------------+
   | :meth:`assertSetEqual(a, b)             | sets or frozensets          | 3.1          |
   | <TestCase.assertSetEqual>`              |                             |              |
   +-----------------------------------------+-----------------------------+--------------+
   | :meth:`assertDictEqual(a, b)            | dicts                       | 3.1          |
   | <TestCase.assertDictEqual>`             |                             |              |
   +-----------------------------------------+-----------------------------+--------------+



1233
   .. method:: assertMultiLineEqual(first, second, msg=None)
1234

1235
      Test that the multiline string *first* is equal to the string *second*.
1236 1237 1238 1239 1240 1241 1242
      When not equal a diff of the two strings highlighting the differences
      will be included in the error message. This method is used by default
      when comparing strings with :meth:`assertEqual`.

      .. versionadded:: 3.1


1243
   .. method:: assertSequenceEqual(first, second, msg=None, seq_type=None)
1244 1245

      Tests that two sequences are equal.  If a *seq_type* is supplied, both
1246
      *first* and *second* must be instances of *seq_type* or a failure will
1247 1248 1249
      be raised.  If the sequences are different an error message is
      constructed that shows the difference between the two.

1250 1251
      This method is not called directly by :meth:`assertEqual`, but
      it's used to implement :meth:`assertListEqual` and
1252 1253 1254 1255 1256
      :meth:`assertTupleEqual`.

      .. versionadded:: 3.1


1257 1258
   .. method:: assertListEqual(first, second, msg=None)
               assertTupleEqual(first, second, msg=None)
1259

Ezio Melotti's avatar
Ezio Melotti committed
1260
      Tests that two lists or tuples are equal.  If not, an error message is
1261 1262 1263 1264 1265 1266 1267 1268
      constructed that shows only the differences between the two.  An error
      is also raised if either of the parameters are of the wrong type.
      These methods are used by default when comparing lists or tuples with
      :meth:`assertEqual`.

      .. versionadded:: 3.1


1269
   .. method:: assertSetEqual(first, second, msg=None)
1270 1271 1272 1273 1274

      Tests that two sets are equal.  If not, an error message is constructed
      that lists the differences between the sets.  This method is used by
      default when comparing sets or frozensets with :meth:`assertEqual`.

1275
      Fails if either of *first* or *second* does not have a :meth:`set.difference`
1276 1277 1278 1279 1280
      method.

      .. versionadded:: 3.1


1281
   .. method:: assertDictEqual(first, second, msg=None)
1282 1283 1284 1285 1286 1287 1288 1289 1290 1291

      Test that two dictionaries are equal.  If not, an error message is
      constructed that shows the differences in the dictionaries. This
      method will be used by default to compare dictionaries in
      calls to :meth:`assertEqual`.

      .. versionadded:: 3.1



1292 1293
   .. _other-methods-and-attrs:

1294
   Finally the :class:`TestCase` provides the following methods and attributes:
1295

1296

1297
   .. method:: fail(msg=None)
1298

1299
      Signals a test failure unconditionally, with *msg* or ``None`` for
1300
      the error message.
1301 1302


1303
   .. attribute:: failureException
1304

1305 1306 1307 1308 1309
      This class attribute gives the exception raised by the test method.  If a
      test framework needs to use a specialized exception, possibly to carry
      additional information, it must subclass this exception in order to "play
      fair" with the framework.  The initial value of this attribute is
      :exc:`AssertionError`.
1310

1311 1312 1313

   .. attribute:: longMessage

1314
      If set to ``True`` then any explicit failure message you pass in to the
1315 1316 1317 1318
      :ref:`assert methods <assert-methods>` will be appended to the end of the
      normal failure message.  The normal messages contain useful information
      about the objects involved, for example the message from assertEqual
      shows you the repr of the two unequal objects. Setting this attribute
1319
      to ``True`` allows you to have a custom error message in addition to the
1320
      normal one.
1321

1322 1323
      This attribute defaults to ``True``. If set to False then a custom message
      passed to an assert method will silence the normal message.
1324 1325

      The class setting can be overridden in individual tests by assigning an
1326
      instance attribute to ``True`` or ``False`` before calling the assert methods.
1327

1328
      .. versionadded:: 3.1
1329 1330


1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345
   .. attribute:: maxDiff

      This attribute controls the maximum length of diffs output by assert
      methods that report diffs on failure. It defaults to 80*8 characters.
      Assert methods affected by this attribute are
      :meth:`assertSequenceEqual` (including all the sequence comparison
      methods that delegate to it), :meth:`assertDictEqual` and
      :meth:`assertMultiLineEqual`.

      Setting ``maxDiff`` to None means that there is no maximum length of
      diffs.

      .. versionadded:: 3.2


1346 1347
   Testing frameworks can use the following methods to collect information on
   the test:
1348 1349


1350
   .. method:: countTestCases()
1351

1352 1353
      Return the number of tests represented by this test object.  For
      :class:`TestCase` instances, this will always be ``1``.
1354 1355


1356
   .. method:: defaultTestResult()
1357

1358 1359 1360
      Return an instance of the test result class that should be used for this
      test case class (if no other result instance is provided to the
      :meth:`run` method).
1361

1362 1363 1364
      For :class:`TestCase` instances, this will always be an instance of
      :class:`TestResult`; subclasses of :class:`TestCase` should override this
      as necessary.
1365 1366


1367
   .. method:: id()
1368

1369 1370
      Return a string identifying the specific test case.  This is usually the
      full name of the test method, including the module and class name.
1371 1372


1373
   .. method:: shortDescription()
1374

1375
      Returns a description of the test, or ``None`` if no description
1376 1377
      has been provided.  The default implementation of this method
      returns the first line of the test method's docstring, if available,
1378
      or ``None``.
1379

1380
      .. versionchanged:: 3.1
1381
         In 3.1 this was changed to add the test name to the short description
1382
         even in the presence of a docstring.  This caused compatibility issues
1383
         with unittest extensions and adding the test name was moved to the
1384
         :class:`TextTestResult` in Python 3.2.
1385

1386

1387
   .. method:: addCleanup(function, *args, **kwargs)
Benjamin Peterson's avatar
Benjamin Peterson committed
1388 1389 1390 1391 1392 1393 1394 1395 1396 1397

      Add a function to be called after :meth:`tearDown` to cleanup resources
      used during the test. Functions will be called in reverse order to the
      order they are added (LIFO). They are called with any arguments and
      keyword arguments passed into :meth:`addCleanup` when they are
      added.

      If :meth:`setUp` fails, meaning that :meth:`tearDown` is not called,
      then any cleanup functions added will still be called.

1398
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
1399 1400 1401 1402


   .. method:: doCleanups()

Barry Warsaw's avatar
typo  
Barry Warsaw committed
1403
      This method is called unconditionally after :meth:`tearDown`, or
Benjamin Peterson's avatar
Benjamin Peterson committed
1404 1405 1406 1407 1408 1409 1410 1411 1412 1413
      after :meth:`setUp` if :meth:`setUp` raises an exception.

      It is responsible for calling all the cleanup functions added by
      :meth:`addCleanup`. If you need cleanup functions to be called
      *prior* to :meth:`tearDown` then you can call :meth:`doCleanups`
      yourself.

      :meth:`doCleanups` pops methods off the stack of cleanup
      functions one at a time, so it can be called at any time.

1414
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
1415 1416


1417
.. class:: FunctionTestCase(testFunc, setUp=None, tearDown=None, description=None)
1418

1419
   This class implements the portion of the :class:`TestCase` interface which
1420 1421 1422 1423
   allows the test runner to drive the test, but does not provide the methods
   which test code can use to check and report errors.  This is used to create
   test cases using legacy test code, allowing it to be integrated into a
   :mod:`unittest`-based test framework.
1424 1425


1426 1427
.. _deprecated-aliases:

1428 1429 1430 1431 1432 1433 1434
Deprecated aliases
##################

For historical reasons, some of the :class:`TestCase` methods had one or more
aliases that are now deprecated.  The following table lists the correct names
along with their deprecated aliases:

1435 1436 1437 1438 1439 1440
   ==============================  ====================== ======================
    Method Name                     Deprecated alias       Deprecated alias
   ==============================  ====================== ======================
    :meth:`.assertEqual`            failUnlessEqual        assertEquals
    :meth:`.assertNotEqual`         failIfEqual            assertNotEquals
    :meth:`.assertTrue`             failUnless             assert\_
1441 1442
    :meth:`.assertFalse`            failIf
    :meth:`.assertRaises`           failUnlessRaises
1443 1444
    :meth:`.assertAlmostEqual`      failUnlessAlmostEqual  assertAlmostEquals
    :meth:`.assertNotAlmostEqual`   failIfAlmostEqual      assertNotAlmostEquals
1445 1446
    :meth:`.assertRegex`                                   assertRegexpMatches
    :meth:`.assertRaisesRegex`                             assertRaisesRegexp
1447 1448
   ==============================  ====================== ======================

1449
   .. deprecated:: 3.1
1450 1451 1452
         the fail* aliases listed in the second column.
   .. deprecated:: 3.2
         the assert* aliases listed in the third column.
1453 1454 1455
   .. deprecated:: 3.2
         ``assertRegexpMatches`` and ``assertRaisesRegexp`` have been renamed to
         :meth:`.assertRegex` and :meth:`.assertRaisesRegex`
1456 1457


1458
.. _testsuite-objects:
1459

1460 1461
Grouping tests
~~~~~~~~~~~~~~
1462

1463
.. class:: TestSuite(tests=())
1464

1465 1466 1467 1468
   This class represents an aggregation of individual tests cases and test suites.
   The class presents the interface needed by the test runner to allow it to be run
   as any other test case.  Running a :class:`TestSuite` instance is the same as
   iterating over the suite, running each test individually.
1469

1470 1471 1472
   If *tests* is given, it must be an iterable of individual test cases or other
   test suites that will be used to build the suite initially. Additional methods
   are provided to add test cases and suites to the collection later on.
1473

1474 1475 1476 1477
   :class:`TestSuite` objects behave much like :class:`TestCase` objects, except
   they do not actually implement a test.  Instead, they are used to aggregate
   tests into groups of tests that should be run together. Some additional
   methods are available to add tests to :class:`TestSuite` instances:
1478 1479


1480
   .. method:: TestSuite.addTest(test)
1481

1482
      Add a :class:`TestCase` or :class:`TestSuite` to the suite.
1483 1484


1485
   .. method:: TestSuite.addTests(tests)
1486

1487 1488
      Add all the tests from an iterable of :class:`TestCase` and :class:`TestSuite`
      instances to this test suite.
1489

1490 1491
      This is equivalent to iterating over *tests*, calling :meth:`addTest` for
      each element.
1492

1493
   :class:`TestSuite` shares the following methods with :class:`TestCase`:
1494 1495


1496
   .. method:: run(result)
1497

1498 1499 1500 1501
      Run the tests associated with this suite, collecting the result into the
      test result object passed as *result*.  Note that unlike
      :meth:`TestCase.run`, :meth:`TestSuite.run` requires the result object to
      be passed in.
1502 1503


1504
   .. method:: debug()
1505

1506 1507 1508
      Run the tests associated with this suite without collecting the
      result. This allows exceptions raised by the test to be propagated to the
      caller and can be used to support running tests under a debugger.
1509 1510


1511
   .. method:: countTestCases()
1512

1513 1514
      Return the number of tests represented by this test object, including all
      individual tests and sub-suites.
1515

Benjamin Peterson's avatar
Benjamin Peterson committed
1516 1517 1518 1519 1520

   .. method:: __iter__()

      Tests grouped by a :class:`TestSuite` are always accessed by iteration.
      Subclasses can lazily provide tests by overriding :meth:`__iter__`. Note
1521 1522 1523 1524 1525 1526 1527
      that this method may be called several times on a single suite (for
      example when counting tests or comparing for equality) so the tests
      returned by repeated iterations before :meth:`TestSuite.run` must be the
      same for each call iteration. After :meth:`TestSuite.run`, callers should
      not rely on the tests returned by this method unless the caller uses a
      subclass that overrides :meth:`TestSuite._removeTestAtIndex` to preserve
      test references.
Benjamin Peterson's avatar
Benjamin Peterson committed
1528

1529
      .. versionchanged:: 3.2
Benjamin Peterson's avatar
Benjamin Peterson committed
1530 1531 1532 1533
         In earlier versions the :class:`TestSuite` accessed tests directly rather
         than through iteration, so overriding :meth:`__iter__` wasn't sufficient
         for providing tests.

1534 1535 1536 1537 1538
      .. versionchanged:: 3.4
         In earlier versions the :class:`TestSuite` held references to each
         :class:`TestCase` after :meth:`TestSuite.run`. Subclasses can restore
         that behavior by overriding :meth:`TestSuite._removeTestAtIndex`.

1539 1540
   In the typical usage of a :class:`TestSuite` object, the :meth:`run` method
   is invoked by a :class:`TestRunner` rather than by the end-user test harness.
1541 1542


1543 1544
Loading and running tests
~~~~~~~~~~~~~~~~~~~~~~~~~
1545

1546
.. class:: TestLoader()
1547

1548 1549 1550
   The :class:`TestLoader` class is used to create test suites from classes and
   modules.  Normally, there is no need to create an instance of this class; the
   :mod:`unittest` module provides an instance that can be shared as
Ezio Melotti's avatar
Ezio Melotti committed
1551 1552
   :data:`unittest.defaultTestLoader`.  Using a subclass or instance, however,
   allows customization of some configurable properties.
1553

1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567
   :class:`TestLoader` objects have the following attributes:


   .. attribute:: errors

      A list of the non-fatal errors encountered while loading tests. Not reset
      by the loader at any point. Fatal errors are signalled by the relevant
      a method raising an exception to the caller. Non-fatal errors are also
      indicated by a synthetic test that will raise the original error when
      run.

      .. versionadded:: 3.5


1568
   :class:`TestLoader` objects have the following methods:
1569

1570

1571
   .. method:: loadTestsFromTestCase(testCaseClass)
1572

1573 1574
      Return a suite of all tests cases contained in the :class:`TestCase`\ -derived
      :class:`testCaseClass`.
1575

1576 1577 1578 1579 1580 1581
      A test case instance is created for each method named by
      :meth:`getTestCaseNames`. By default these are the method names
      beginning with ``test``. If :meth:`getTestCaseNames` returns no
      methods, but the :meth:`runTest` method is implemented, a single test
      case is created for that method instead.

1582

1583
   .. method:: loadTestsFromModule(module, pattern=None)
1584

1585 1586 1587 1588
      Return a suite of all tests cases contained in the given module. This
      method searches *module* for classes derived from :class:`TestCase` and
      creates an instance of the class for each test method defined for the
      class.
1589

1590
      .. note::
1591

1592 1593 1594 1595 1596
         While using a hierarchy of :class:`TestCase`\ -derived classes can be
         convenient in sharing fixtures and helper functions, defining test
         methods on base classes that are not intended to be instantiated
         directly does not play well with this method.  Doing so, however, can
         be useful when the fixtures are different and defined in subclasses.
1597

1598 1599
      If a module provides a ``load_tests`` function it will be called to
      load the tests. This allows modules to customize test loading.
1600 1601
      This is the `load_tests protocol`_.  The *pattern* argument is passed as
      the third argument to ``load_tests``.
1602

1603
      .. versionchanged:: 3.2
1604 1605
         Support for ``load_tests`` added.

1606 1607 1608 1609 1610 1611
      .. versionchanged:: 3.5
         The undocumented and unofficial *use_load_tests* default argument is
         deprecated and ignored, although it is still accepted for backward
         compatibility.  The method also now accepts a keyword-only argument
         *pattern* which is passed to ``load_tests`` as the third argument.

1612

1613
   .. method:: loadTestsFromName(name, module=None)
1614

1615
      Return a suite of all tests cases given a string specifier.
1616

1617 1618 1619 1620 1621 1622 1623
      The specifier *name* is a "dotted name" that may resolve either to a
      module, a test case class, a test method within a test case class, a
      :class:`TestSuite` instance, or a callable object which returns a
      :class:`TestCase` or :class:`TestSuite` instance.  These checks are
      applied in the order listed here; that is, a method on a possible test
      case class will be picked up as "a test method within a test case class",
      rather than "a callable object".
1624

1625 1626 1627
      For example, if you have a module :mod:`SampleTests` containing a
      :class:`TestCase`\ -derived class :class:`SampleTestCase` with three test
      methods (:meth:`test_one`, :meth:`test_two`, and :meth:`test_three`), the
1628 1629 1630 1631 1632 1633
      specifier ``'SampleTests.SampleTestCase'`` would cause this method to
      return a suite which will run all three test methods. Using the specifier
      ``'SampleTests.SampleTestCase.test_two'`` would cause it to return a test
      suite which will run only the :meth:`test_two` test method. The specifier
      can refer to modules and packages which have not been imported; they will
      be imported as a side-effect.
1634

1635
      The method optionally resolves *name* relative to the given *module*.
1636

1637 1638 1639 1640 1641 1642
   .. versionchanged:: 3.5
      If an :exc:`ImportError` or :exc:`AttributeError` occurs while traversing
      *name* then a synthetic test that raises that error when run will be
      returned. These errors are included in the errors accumulated by
      self.errors.

1643

1644
   .. method:: loadTestsFromNames(names, module=None)
1645

1646 1647 1648
      Similar to :meth:`loadTestsFromName`, but takes a sequence of names rather
      than a single name.  The return value is a test suite which supports all
      the tests defined for each name.
1649 1650


1651
   .. method:: getTestCaseNames(testCaseClass)
1652

1653 1654
      Return a sorted sequence of method names found within *testCaseClass*;
      this should be a subclass of :class:`TestCase`.
1655

1656 1657 1658

   .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None)

1659 1660 1661 1662 1663
      Find all the test modules by recursing into subdirectories from the
      specified start directory, and return a TestSuite object containing them.
      Only test files that match *pattern* will be loaded. (Using shell style
      pattern matching.) Only module names that are importable (i.e. are valid
      Python identifiers) will be loaded.
1664 1665 1666 1667 1668

      All test modules must be importable from the top level of the project. If
      the start directory is not the top level directory then the top level
      directory must be specified separately.

1669 1670 1671 1672
      If importing a module fails, for example due to a syntax error, then
      this will be recorded as a single error and discovery will continue.  If
      the import failure is due to :exc:`SkipTest` being raised, it will be
      recorded as a skip instead of an error.
Benjamin Peterson's avatar
Benjamin Peterson committed
1673

1674 1675
      If a package (a directory containing a file named :file:`__init__.py`) is
      found, the package will be checked for a ``load_tests`` function. If this
1676 1677 1678 1679 1680
      exists then it will be called
      ``package.load_tests(loader, tests, pattern)``. Test discovery takes care
      to ensure that a package is only checked for tests once during an
      invocation, even if the load_tests function itself calls
      ``loader.discover``.
1681

1682 1683 1684
      If ``load_tests`` exists then discovery does *not* recurse into the
      package, ``load_tests`` is responsible for loading all tests in the
      package.
1685 1686 1687 1688 1689 1690

      The pattern is deliberately not stored as a loader attribute so that
      packages can continue discovery themselves. *top_level_dir* is stored so
      ``load_tests`` does not need to pass this argument in to
      ``loader.discover()``.

Benjamin Peterson's avatar
Benjamin Peterson committed
1691 1692
      *start_dir* can be a dotted module name as well as a directory.

1693 1694
      .. versionadded:: 3.2

1695
      .. versionchanged:: 3.4
Ezio Melotti's avatar
Ezio Melotti committed
1696
         Modules that raise :exc:`SkipTest` on import are recorded as skips,
1697 1698 1699 1700 1701
           not errors.
         Discovery works for :term:`namespace packages <namespace package>`.
         Paths are sorted before being imported so that execution order is
           the same even if the underlying file system's ordering is not
           dependent on file name.
1702

1703 1704 1705 1706 1707
      .. versionchanged:: 3.5
         Found packages are now checked for ``load_tests`` regardless of
         whether their path matches *pattern*, because it is impossible for
         a package name to match the default pattern.

1708

1709 1710
   The following attributes of a :class:`TestLoader` can be configured either by
   subclassing or assignment on an instance:
1711 1712


1713
   .. attribute:: testMethodPrefix
1714

1715 1716
      String giving the prefix of method names which will be interpreted as test
      methods.  The default value is ``'test'``.
1717

1718 1719
      This affects :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*`
      methods.
1720 1721


1722
   .. attribute:: sortTestMethodsUsing
1723

1724 1725
      Function to be used to compare method names when sorting them in
      :meth:`getTestCaseNames` and all the :meth:`loadTestsFrom\*` methods.
1726 1727


1728
   .. attribute:: suiteClass
1729

1730 1731 1732
      Callable object that constructs a test suite from a list of tests. No
      methods on the resulting object are needed.  The default value is the
      :class:`TestSuite` class.
1733

1734
      This affects all the :meth:`loadTestsFrom\*` methods.
1735 1736


1737
.. class:: TestResult
1738

1739 1740
   This class is used to compile information about which tests have succeeded
   and which have failed.
1741

1742 1743 1744 1745
   A :class:`TestResult` object stores the results of a set of tests.  The
   :class:`TestCase` and :class:`TestSuite` classes ensure that results are
   properly recorded; test authors do not need to worry about recording the
   outcome of tests.
1746

1747 1748 1749 1750
   Testing frameworks built on top of :mod:`unittest` may want access to the
   :class:`TestResult` object generated by running a set of tests for reporting
   purposes; a :class:`TestResult` instance is returned by the
   :meth:`TestRunner.run` method for this purpose.
1751

1752 1753
   :class:`TestResult` instances have the following attributes that will be of
   interest when inspecting the results of running a set of tests:
1754 1755


1756
   .. attribute:: errors
1757

1758 1759 1760
      A list containing 2-tuples of :class:`TestCase` instances and strings
      holding formatted tracebacks. Each tuple represents a test which raised an
      unexpected exception.
1761

1762
   .. attribute:: failures
1763

1764 1765
      A list containing 2-tuples of :class:`TestCase` instances and strings
      holding formatted tracebacks. Each tuple represents a test where a failure
1766
      was explicitly signalled using the :meth:`TestCase.assert\*` methods.
1767

1768
   .. attribute:: skipped
1769

1770 1771
      A list containing 2-tuples of :class:`TestCase` instances and strings
      holding the reason for skipping the test.
1772

1773
      .. versionadded:: 3.1
1774

1775
   .. attribute:: expectedFailures
1776

Georg Brandl's avatar
Georg Brandl committed
1777 1778
      A list containing 2-tuples of :class:`TestCase` instances and strings
      holding formatted tracebacks.  Each tuple represents an expected failure
1779
      of the test case.
1780

1781
   .. attribute:: unexpectedSuccesses
1782

1783 1784
      A list containing :class:`TestCase` instances that were marked as expected
      failures, but succeeded.
1785

1786
   .. attribute:: shouldStop
1787

1788
      Set to ``True`` when the execution of tests should stop by :meth:`stop`.
1789

1790
   .. attribute:: testsRun
1791

1792
      The total number of tests run so far.
1793

1794
   .. attribute:: buffer
Benjamin Peterson's avatar
Benjamin Peterson committed
1795 1796 1797 1798 1799 1800

      If set to true, ``sys.stdout`` and ``sys.stderr`` will be buffered in between
      :meth:`startTest` and :meth:`stopTest` being called. Collected output will
      only be echoed onto the real ``sys.stdout`` and ``sys.stderr`` if the test
      fails or errors. Any output is also attached to the failure / error message.

1801
      .. versionadded:: 3.2
Benjamin Peterson's avatar
Benjamin Peterson committed
1802 1803 1804 1805 1806 1807

   .. attribute:: failfast

      If set to true :meth:`stop` will be called on the first failure or error,
      halting the test run.

1808
      .. versionadded:: 3.2
Benjamin Peterson's avatar
Benjamin Peterson committed
1809

1810 1811 1812 1813 1814
   .. attribute:: tb_locals

      If set to true then local variables will be shown in tracebacks.

      .. versionadded:: 3.5
Benjamin Peterson's avatar
Benjamin Peterson committed
1815

1816
   .. method:: wasSuccessful()
1817

1818 1819
      Return ``True`` if all tests run so far have passed, otherwise returns
      ``False``.
1820

1821 1822 1823 1824
      .. versionchanged:: 3.4
         Returns ``False`` if there were any :attr:`unexpectedSuccesses`
         from tests marked with the :func:`expectedFailure` decorator.

1825
   .. method:: stop()
1826

1827
      This method can be called to signal that the set of tests being run should
1828
      be aborted by setting the :attr:`shouldStop` attribute to ``True``.
1829 1830
      :class:`TestRunner` objects should respect this flag and return without
      running any additional tests.
1831

1832 1833 1834 1835
      For example, this feature is used by the :class:`TextTestRunner` class to
      stop the test framework when the user signals an interrupt from the
      keyboard.  Interactive tools which provide :class:`TestRunner`
      implementations can use this in a similar manner.
1836

1837 1838 1839 1840
   The following methods of the :class:`TestResult` class are used to maintain
   the internal data structures, and may be extended in subclasses to support
   additional reporting requirements.  This is particularly useful in building
   tools which support interactive reporting while tests are being run.
1841

1842

1843
   .. method:: startTest(test)
1844

1845
      Called when the test case *test* is about to be run.
1846

1847
   .. method:: stopTest(test)
1848

1849 1850
      Called after the test case *test* has been executed, regardless of the
      outcome.
1851

1852
   .. method:: startTestRun()
Benjamin Peterson's avatar
Benjamin Peterson committed
1853 1854 1855

      Called once before any tests are executed.

1856
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
1857 1858


1859
   .. method:: stopTestRun()
Benjamin Peterson's avatar
Benjamin Peterson committed
1860

1861
      Called once after all tests are executed.
Benjamin Peterson's avatar
Benjamin Peterson committed
1862

1863
      .. versionadded:: 3.1
Benjamin Peterson's avatar
Benjamin Peterson committed
1864 1865


1866
   .. method:: addError(test, err)
1867

1868
      Called when the test case *test* raises an unexpected exception. *err* is a
1869 1870
      tuple of the form returned by :func:`sys.exc_info`: ``(type, value,
      traceback)``.
1871

1872 1873 1874
      The default implementation appends a tuple ``(test, formatted_err)`` to
      the instance's :attr:`errors` attribute, where *formatted_err* is a
      formatted traceback derived from *err*.
1875 1876


1877
   .. method:: addFailure(test, err)
1878

1879 1880
      Called when the test case *test* signals a failure. *err* is a tuple of
      the form returned by :func:`sys.exc_info`: ``(type, value, traceback)``.
1881

1882 1883 1884
      The default implementation appends a tuple ``(test, formatted_err)`` to
      the instance's :attr:`failures` attribute, where *formatted_err* is a
      formatted traceback derived from *err*.
1885 1886


1887
   .. method:: addSuccess(test)
1888

1889
      Called when the test case *test* succeeds.
1890

1891
      The default implementation does nothing.
1892 1893


1894
   .. method:: addSkip(test, reason)
1895

1896 1897
      Called when the test case *test* is skipped.  *reason* is the reason the
      test gave for skipping.
1898

1899 1900
      The default implementation appends a tuple ``(test, reason)`` to the
      instance's :attr:`skipped` attribute.
1901 1902


1903
   .. method:: addExpectedFailure(test, err)
1904

1905 1906
      Called when the test case *test* fails, but was marked with the
      :func:`expectedFailure` decorator.
1907

1908 1909 1910
      The default implementation appends a tuple ``(test, formatted_err)`` to
      the instance's :attr:`expectedFailures` attribute, where *formatted_err*
      is a formatted traceback derived from *err*.
1911 1912


1913
   .. method:: addUnexpectedSuccess(test)
1914

1915 1916
      Called when the test case *test* was marked with the
      :func:`expectedFailure` decorator, but succeeded.
1917

1918 1919
      The default implementation appends the test to the instance's
      :attr:`unexpectedSuccesses` attribute.
1920

1921

1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937
   .. method:: addSubTest(test, subtest, outcome)

      Called when a subtest finishes.  *test* is the test case
      corresponding to the test method.  *subtest* is a custom
      :class:`TestCase` instance describing the subtest.

      If *outcome* is :const:`None`, the subtest succeeded.  Otherwise,
      it failed with an exception where *outcome* is a tuple of the form
      returned by :func:`sys.exc_info`: ``(type, value, traceback)``.

      The default implementation does nothing when the outcome is a
      success, and records subtest failures as normal failures.

      .. versionadded:: 3.4


1938 1939
.. class:: TextTestResult(stream, descriptions, verbosity)

1940 1941 1942 1943 1944 1945
   A concrete implementation of :class:`TestResult` used by the
   :class:`TextTestRunner`.

   .. versionadded:: 3.2
      This class was previously named ``_TextTestResult``. The old name still
      exists as an alias but is deprecated.
1946

1947

1948
.. data:: defaultTestLoader
1949

1950 1951 1952
   Instance of the :class:`TestLoader` class intended to be shared.  If no
   customization of the :class:`TestLoader` is needed, this instance can be used
   instead of repeatedly creating new instances.
1953 1954


1955
.. class:: TextTestRunner(stream=None, descriptions=True, verbosity=1, failfast=False, \
1956
                          buffer=False, resultclass=None, warnings=None, *, tb_locals=False)
1957

1958
   A basic test runner implementation that outputs results to a stream. If *stream*
1959
   is ``None``, the default, :data:`sys.stderr` is used as the output stream. This class
1960
   has a few configurable parameters, but is essentially very simple.  Graphical
1961 1962 1963
   applications which run test suites should provide alternate implementations. Such
   implementations should accept ``**kwargs`` as the interface to construct runners
   changes when features are added to unittest.
1964

1965
   By default this runner shows :exc:`DeprecationWarning`,
1966
   :exc:`PendingDeprecationWarning`, :exc:`ResourceWarning` and
1967 1968 1969 1970 1971 1972 1973
   :exc:`ImportWarning` even if they are :ref:`ignored by default
   <warning-ignored>`. Deprecation warnings caused by :ref:`deprecated unittest
   methods <deprecated-aliases>` are also special-cased and, when the warning
   filters are ``'default'`` or ``'always'``, they will appear only once
   per-module, in order to avoid too many warning messages.  This behavior can
   be overridden using the :option:`-Wd` or :option:`-Wa` options and leaving
   *warnings* to ``None``.
1974

1975 1976 1977 1978
   .. versionchanged:: 3.2
      Added the ``warnings`` argument.

   .. versionchanged:: 3.2
1979
      The default stream is set to :data:`sys.stderr` at instantiation time rather
1980 1981
      than import time.

1982 1983 1984
   .. versionchanged:: 3.5
      Added the tb_locals parameter.

Benjamin Peterson's avatar
Benjamin Peterson committed
1985 1986 1987 1988 1989
   .. method:: _makeResult()

      This method returns the instance of ``TestResult`` used by :meth:`run`.
      It is not intended to be called directly, but can be overridden in
      subclasses to provide a custom ``TestResult``.
1990

1991 1992
      ``_makeResult()`` instantiates the class or callable passed in the
      ``TextTestRunner`` constructor as the ``resultclass`` argument. It
Benjamin Peterson's avatar
Benjamin Peterson committed
1993
      defaults to :class:`TextTestResult` if no ``resultclass`` is provided.
1994 1995 1996
      The result class is instantiated with the following arguments::

        stream, descriptions, verbosity
Benjamin Peterson's avatar
Benjamin Peterson committed
1997

1998 1999 2000 2001 2002 2003 2004 2005
   .. method:: run(test)

      This method is the main public interface to the `TextTestRunner`. This
      method takes a :class:`TestSuite` or :class:`TestCase` instance. A
      :class:`TestResult` is created by calling
      :func:`_makeResult` and the test(s) are run and the
      results printed to stdout.

2006

2007
.. function:: main(module='__main__', defaultTest=None, argv=None, testRunner=None, \
2008
                   testLoader=unittest.defaultTestLoader, exit=True, verbosity=1, \
2009
                   failfast=None, catchbreak=None, buffer=None, warnings=None)
2010

2011 2012 2013 2014
   A command-line program that loads a set of tests from *module* and runs them;
   this is primarily for making test modules conveniently executable.
   The simplest use for this function is to include the following line at the
   end of a test script::
2015 2016 2017 2018

      if __name__ == '__main__':
          unittest.main()

2019 2020 2021 2022 2023 2024
   You can run tests with more detailed information by passing in the verbosity
   argument::

      if __name__ == '__main__':
          unittest.main(verbosity=2)

2025 2026 2027 2028
   The *defaultTest* argument is either the name of a single test or an
   iterable of test names to run if no test names are specified via *argv*.  If
   not specified or ``None`` and no test names are provided via *argv*, all
   tests found in *module* are run.
2029

2030 2031 2032 2033
   The *argv* argument can be a list of options passed to the program, with the
   first element being the program name.  If not specified or ``None``,
   the values of :data:`sys.argv` are used.

2034
   The *testRunner* argument can either be a test runner class or an already
Benjamin Peterson's avatar
Benjamin Peterson committed
2035 2036 2037
   created instance of it. By default ``main`` calls :func:`sys.exit` with
   an exit code indicating success or failure of the tests run.

2038 2039 2040
   The *testLoader* argument has to be a :class:`TestLoader` instance,
   and defaults to :data:`defaultTestLoader`.

Benjamin Peterson's avatar
Benjamin Peterson committed
2041 2042 2043 2044 2045 2046 2047
   ``main`` supports being used from the interactive interpreter by passing in the
   argument ``exit=False``. This displays the result on standard output without
   calling :func:`sys.exit`::

      >>> from unittest import main
      >>> main(module='test_module', exit=False)

2048
   The *failfast*, *catchbreak* and *buffer* parameters have the same
2049
   effect as the same-name `command-line options`_.
Benjamin Peterson's avatar
Benjamin Peterson committed
2050

2051 2052 2053 2054 2055
   The *warning* argument specifies the :ref:`warning filter <warning-filter>`
   that should be used while running the tests.  If it's not specified, it will
   remain ``None`` if a :option:`-W` option is passed to :program:`python`,
   otherwise it will be set to ``'default'``.

Benjamin Peterson's avatar
Benjamin Peterson committed
2056 2057 2058
   Calling ``main`` actually returns an instance of the ``TestProgram`` class.
   This stores the result of the tests run as the ``result`` attribute.

Éric Araujo's avatar
Éric Araujo committed
2059
   .. versionchanged:: 3.1
2060
      The *exit* parameter was added.
Éric Araujo's avatar
Éric Araujo committed
2061

2062
   .. versionchanged:: 3.2
2063 2064
      The *verbosity*, *failfast*, *catchbreak*, *buffer*
      and *warnings* parameters were added.
2065

2066 2067 2068 2069
   .. versionchanged:: 3.4
      The *defaultTest* parameter was changed to also accept an iterable of
      test names.

2070 2071 2072 2073

load_tests Protocol
###################

2074
.. versionadded:: 3.2
Benjamin Peterson's avatar
Benjamin Peterson committed
2075

2076 2077 2078 2079 2080 2081
Modules or packages can customize how tests are loaded from them during normal
test runs or test discovery by implementing a function called ``load_tests``.

If a test module defines ``load_tests`` it will be called by
:meth:`TestLoader.loadTestsFromModule` with the following arguments::

2082 2083 2084 2085
    load_tests(loader, standard_tests, pattern)

where *pattern* is passed straight through from ``loadTestsFromModule``.  It
defaults to ``None``.
2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106

It should return a :class:`TestSuite`.

*loader* is the instance of :class:`TestLoader` doing the loading.
*standard_tests* are the tests that would be loaded by default from the
module. It is common for test modules to only want to add or remove tests
from the standard set of tests.
The third argument is used when loading packages as part of test discovery.

A typical ``load_tests`` function that loads tests from a specific set of
:class:`TestCase` classes may look like::

    test_cases = (TestCase1, TestCase2, TestCase3)

    def load_tests(loader, tests, pattern):
        suite = TestSuite()
        for test_class in test_cases:
            tests = loader.loadTestsFromTestCase(test_class)
            suite.addTests(tests)
        return suite

2107 2108 2109 2110 2111 2112
If discovery is started in a directory containing a package, either from the
command line or by calling :meth:`TestLoader.discover`, then the package
:file:`__init__.py` will be checked for ``load_tests``.  If that function does
not exist, discovery will recurse into the package as though it were just
another directory.  Otherwise, discovery of the package's tests will be left up
to ``load_tests`` which is called with the following arguments::
2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129

    load_tests(loader, standard_tests, pattern)

This should return a :class:`TestSuite` representing all the tests
from the package. (``standard_tests`` will only contain tests
collected from :file:`__init__.py`.)

Because the pattern is passed into ``load_tests`` the package is free to
continue (and potentially modify) test discovery. A 'do nothing'
``load_tests`` function for a test package would look like::

    def load_tests(loader, standard_tests, pattern):
        # top level directory cached on loader instance
        this_dir = os.path.dirname(__file__)
        package_tests = loader.discover(start_dir=this_dir, pattern=pattern)
        standard_tests.addTests(package_tests)
        return standard_tests
Benjamin Peterson's avatar
Benjamin Peterson committed
2130

2131 2132 2133 2134 2135
.. versionchanged:: 3.5
   Discovery no longer checks package names for matching *pattern* due to the
   impossibility of package names matching the default pattern.


Benjamin Peterson's avatar
Benjamin Peterson committed
2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195

Class and Module Fixtures
-------------------------

Class and module level fixtures are implemented in :class:`TestSuite`. When
the test suite encounters a test from a new class then :meth:`tearDownClass`
from the previous class (if there is one) is called, followed by
:meth:`setUpClass` from the new class.

Similarly if a test is from a different module from the previous test then
``tearDownModule`` from the previous module is run, followed by
``setUpModule`` from the new module.

After all the tests have run the final ``tearDownClass`` and
``tearDownModule`` are run.

Note that shared fixtures do not play well with [potential] features like test
parallelization and they break test isolation. They should be used with care.

The default ordering of tests created by the unittest test loaders is to group
all tests from the same modules and classes together. This will lead to
``setUpClass`` / ``setUpModule`` (etc) being called exactly once per class and
module. If you randomize the order, so that tests from different modules and
classes are adjacent to each other, then these shared fixture functions may be
called multiple times in a single test run.

Shared fixtures are not intended to work with suites with non-standard
ordering. A ``BaseTestSuite`` still exists for frameworks that don't want to
support shared fixtures.

If there are any exceptions raised during one of the shared fixture functions
the test is reported as an error. Because there is no corresponding test
instance an ``_ErrorHolder`` object (that has the same interface as a
:class:`TestCase`) is created to represent the error. If you are just using
the standard unittest test runner then this detail doesn't matter, but if you
are a framework author it may be relevant.


setUpClass and tearDownClass
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These must be implemented as class methods::

    import unittest

    class Test(unittest.TestCase):
        @classmethod
        def setUpClass(cls):
            cls._connection = createExpensiveConnectionObject()

        @classmethod
        def tearDownClass(cls):
            cls._connection.destroy()

If you want the ``setUpClass`` and ``tearDownClass`` on base classes called
then you must call up to them yourself. The implementations in
:class:`TestCase` are empty.

If an exception is raised during a ``setUpClass`` then the tests in the class
are not run and the ``tearDownClass`` is not run. Skipped classes will not
2196
have ``setUpClass`` or ``tearDownClass`` run. If the exception is a
2197
:exc:`SkipTest` exception then the class will be reported as having been skipped
2198
instead of as an error.
Benjamin Peterson's avatar
Benjamin Peterson committed
2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212


setUpModule and tearDownModule
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These should be implemented as functions::

    def setUpModule():
        createConnection()

    def tearDownModule():
        closeConnection()

If an exception is raised in a ``setUpModule`` then none of the tests in the
2213
module will be run and the ``tearDownModule`` will not be run. If the exception is a
2214
:exc:`SkipTest` exception then the module will be reported as having been skipped
2215
instead of as an error.
Benjamin Peterson's avatar
Benjamin Peterson committed
2216 2217 2218 2219 2220


Signal Handling
---------------

2221 2222
.. versionadded:: 3.2

2223
The :option:`-c/--catch <unittest -c>` command-line option to unittest,
2224 2225 2226 2227 2228
along with the ``catchbreak`` parameter to :func:`unittest.main()`, provide
more friendly handling of control-C during a test run. With catch break
behavior enabled control-C will allow the currently running test to complete,
and the test run will then end and report all the results so far. A second
control-c will raise a :exc:`KeyboardInterrupt` in the usual way.
Benjamin Peterson's avatar
Benjamin Peterson committed
2229

2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240
The control-c handling signal handler attempts to remain compatible with code or
tests that install their own :const:`signal.SIGINT` handler. If the ``unittest``
handler is called but *isn't* the installed :const:`signal.SIGINT` handler,
i.e. it has been replaced by the system under test and delegated to, then it
calls the default handler. This will normally be the expected behavior by code
that replaces an installed handler and delegates to it. For individual tests
that need ``unittest`` control-c handling disabled the :func:`removeHandler`
decorator can be used.

There are a few utility functions for framework authors to enable control-c
handling functionality within test frameworks.
Benjamin Peterson's avatar
Benjamin Peterson committed
2241 2242 2243 2244 2245 2246 2247

.. function:: installHandler()

   Install the control-c handler. When a :const:`signal.SIGINT` is received
   (usually in response to the user pressing control-c) all registered results
   have :meth:`~TestResult.stop` called.

2248

Benjamin Peterson's avatar
Benjamin Peterson committed
2249 2250 2251 2252 2253 2254
.. function:: registerResult(result)

   Register a :class:`TestResult` object for control-c handling. Registering a
   result stores a weak reference to it, so it doesn't prevent the result from
   being garbage collected.

2255 2256 2257 2258
   Registering a :class:`TestResult` object has no side-effects if control-c
   handling is not enabled, so test frameworks can unconditionally register
   all results they create independently of whether or not handling is enabled.

2259

Benjamin Peterson's avatar
Benjamin Peterson committed
2260 2261 2262 2263 2264 2265
.. function:: removeResult(result)

   Remove a registered result. Once a result has been removed then
   :meth:`~TestResult.stop` will no longer be called on that result object in
   response to a control-c.

2266

2267 2268 2269 2270 2271 2272 2273 2274 2275
.. function:: removeHandler(function=None)

   When called without arguments this function removes the control-c handler
   if it has been installed. This function can also be used as a test decorator
   to temporarily remove the handler whilst the test is being executed::

      @unittest.removeHandler
      def test_signal_handling(self):
          ...