Mock

In this chapter, we will test the CLI. For this, we will use the mock package, which has been delivered as part of the Python standard library under the name unittest.mock since Python 3.3. For older versions of Python, you can install it with :

Linux/macOS

$ . .venv/bin/activate
$ python -m pip install mock

Windows

C:> Scripts\activate.bat
C:> python -m pip install mock

Mock objects are sometimes also referred to as test doubles, fakes or stubs. With pytest’s own monkeypatch fixture and mock, you should have all the functions you need.

Example

Firstly, we wanted to start with a simple example and check whether the working days from Monday to Friday are determined correctly.

1. We import datetime.datetime and Mock:

   from datetime import datetime
   from unittest.mock import Mock
   

2. Then we define two test days:

   monday = datetime(year=2021, month=10, day=11)
   saturday = datetime(year=2021, month=10, day=16)
   

3. Now we define a method for checking the working days, whereby the Python datetime library treats Mondays as 0 and Sundays as 6:

   def is_workingday():
       today = datetime.today()
       return 0 <= today.weekday() < 5
   

4. Then we mock datetime:

   datetime = Mock()
   

5. Finally, we test our two mock objects:

   datetime.today.return_value = monday
   # Test Tuesday is a weekday
   assert is_workingday()
   

   datetime.today.return_value = saturday
   # Test Saturday is not a weekday
   assert not is_workingday()
   

Testing with Typer

For testing the Items CLI, we will also look at how the CliRunner provided by Typer helps with testing. Typer provides a test interface that allows us to call our application without having to rely on subprocess.run() as in the short capsys example. This is good because we cannot simulate what is running in a separate process. So in tests/cli/conftest.py we can just pass our application items.cli.app and a list of strings representing the command to the invoke() function of our runner: more precisely, we use shlex.split(command_string)() to convert the commands, for example list -o "veit" into ["list", "-o", "veit"] and can then intercept and return the output.

 import shlex

 import pytest
 from typer.testing import CliRunner

 import items

 runner = CliRunner()


 @pytest.fixture()
 def items_cli(db_path, monkeypatch, items_db):
     monkeypatch.setenv("ITEMS_DB_DIR", db_path.as_posix())

     def run_cli(command_string):
         command_list = shlex.split(command_string)
         result = runner.invoke(items.cli.app, command_list)
         output = result.stdout.rstrip()
         return output

     return run_cli

We can then simply use this fixture to test the version in tests/cli/test_version.py, for example:

import items


def test_version(items_cli):
    assert items_cli("version") == items.__version__

Mocking of attributes

Let’s take a look at how we can use mocking to ensure that, for example, three-digit version numbers of items.__version__() are also output correctly via the CLI. For this we will use mock.patch.object() as a context manager:

 from unittest import mock

 import items


 def test_mock_version(items_cli):
     with mock.patch.object(items, "__version__", "100.0.0"):
         assert items_cli("version") == items.__version__

In our test code, we import items. The resulting items object is what we will patch. The call to mock.patch.object(), which is used as a context manager within a with block, returns a mock object that is cleaned up after the with block:

1. In this case, the __version__ attribute of items is replaced with "100.0.0" for the duration of the with block.

2. We then use items_cli() to call our CLI application with the "version" command. However, when the version() method is called, the __version__ attribute is not the original string, but the string we replaced with mock.patch.object().

Mocking classes and methods

In src/items/cli.py we have defined config() as follows:

def config():
    """List the path to the Items db."""
    with items_db() as db:
        print(db.path())

items_db() is a context manager that returns an items.ItemsDB object. The returned object is then used as a db to call db.path(). So we should mock two things here: items.ItemsDB and one of its methods, path(). Let’s start with the class:

from unittest import mock

import items


def test_mock_itemsdb(items_cli):
    with mock.patch.object(items, "ItemsDB") as MockItemsDB:
        mock_db_path = MockItemsDB.return_value.path.return_value = "/foo/"
        assert items_cli("config") == str(mock_db_path)

Let’s make sure that it really works:

$ pytest -v -s tests/cli/test_config.py::test_mock_itemsdb
============================= test session starts ==============================
...
configfile: pyproject.toml
plugins: cov-4.1.0, Faker-19.11.0
collected 1 item

tests/cli/test_config.py::test_mock_itemsdb PASSED

============================== 1 passed in 0.04s ===============================

Great, now we just have to move the mock for the database to a fixture, because we will need it in many test methods:

@pytest.fixture()
def mock_itemsdb():
    with mock.patch.object(items="ItemsDB") as MockItemsDB:
        yield MockItemsDB.return_value

This fixture mocks the ItemsDB object and returns the return_value so that tests can use it to replace things like path:

def test_mock_itemsdb(items_cli, mock_itemsdb):
    mock_itemsdb.path.return_value = "/foo/"
    result = runner.invoke(app, ["config"])
    assert result.stdout.rstrip() == "/foo/"

Alternatively, the @mock.patch() decorator can also be used to mock classes or objects. In the following examples, the output of os.listdir is mocked. This does not require db_path to be present in the file system:

import os
from unittest import mock


@mock.patch("os.listdir", mock.MagicMock(return_value="db_path"))
def test_listdir():
    assert "db_path" == os.listdir()

Another alternative is to define the return value separately:

@mock.patch("os.listdir")
def test_listdir(mock_listdir):
    mock_listdir.return_value = "db_path"
    assert "db_path" == os.listdir()

Synchronising mocks with autospec

Mock objects are usually intended as objects that are used instead of the real implementation. By default, however, they will accept any access. For example, if the real object allows start(index)(), our mock objects should also allow start(index)(). However, there is a problem with this. Mock objects are too flexible by default: they would also accept stort() or other misspelled, renamed or deleted methods or parameters. Over time, this can lead to so-called mock drift if the interface you are modelling changes, but your mock in your test code does not. This form of mock drift can be solved by adding autospec=True to the mock during creation:

 @pytest.fixture()
 def mock_itemsdb():
     with mock.patch.object(items, "ItemsDB") as MockItemsDB:
         yield MockItemsDB.return_value

Usually, this protection is always built in with autospec. The only exception I know of is if the class or object being mocked has dynamic methods or if attributes are added at runtime.

See also

The Python documentation has a large section on autospec: Autospeccing.

Check call with assert_called_with()

So far, we have used the return values of a mocking method to ensure that our application code handles the return values correctly. But sometimes there is no useful return value, for example with items add some tasks -o veit. In these cases, we can ask the mock object if it was called correctly. After calling items_cli("add some tasks -o veit")(), the API is not used to check whether the item has entered the database, but a mock is used to ensure that the CLI has called the API method correctly. Finally, the implementation of the add() function calls db.add_item() with an Item object:

 def test_add_with_owner(mock_itemsdb, items_cli):
     items_cli("add some task -o veit")
     expected = items.Item("some task", owner="veit", state="todo")
     mock_itemsdb.add_item.assert_called_with(expected)

If add_item() is not called or is called with the wrong type or the wrong object content, the test fails. For example, if we capitalise the string "Veit" in expected, but not in the CLI call, we get the following output:

 $ pytest -s tests/cli/test_add.py::test_add_with_owner
 ============================= test session starts ==============================
 ...
 configfile: pyproject.toml
 plugins: cov-4.1.0, Faker-19.11.0
 collected 1 item

 tests/cli/test_add.py F
 ...
 >           raise AssertionError(_error_message()) from cause
 E           AssertionError: expected call not found.
 E           Expected: add_item(Item(summary='some task', owner='Veit', state='todo', id=None))
 E           Actual: add_item(Item(summary='some task', owner='veit', state='todo', id=None))
 ...
 =========================== short test summary info ============================
 FAILED tests/cli/test_add.py::test_add_with_owner - AssertionError: expected call not found.
 ============================== 1 failed in 0.08s ===============================

See also

There is a whole range of variants of assert_called(). A complete list and description can be found in unittest.mock.Mock.assert_called.

If the only way to test is to ensure the correct call, the various assert_called*() methods fulfil their purpose.

Wenn die einzige Möglichkeit zum Testen darin besteht, den korrekten Aufruf sicherzustellen, erfüllen die verschiedenen assert_called*()-Methoden ihren Zweck.

Create error conditions

Let’s now check if the Items CLI handles error conditions correctly. For example, here is the implementation of the delete command:

@app.command()
def delete(item_id: int):
    """Remove item in db with given id."""
    with items_db() as db:
        try:
            db.delete_item(item_id)
        except items.InvalidItemId:
            print(f"Error: Invalid item id {item_id}")

To test how the CLI handles an error condition, we can pretend that delete_item() generates an exception by assigning the exception to the side_effect attribute of the mock object, like this:

def test_delete_invalid(mock_itemsdb, items_cli):
    mock_itemsdb.delete_item.side_effect = items.api.InvalidItemId
    out = items_cli("delete 42")
    assert "Error: Invalid item id 42" in out

That’s all we need to test the CLI: mocking return values, checking calls to mock functions and mocking exceptions. However, there is a whole range of other mocking techniques that we have not covered. So be sure to read unittest.mock — mock object library if you want to use mocking extensively.

Limitations of mocking

One of the biggest problems with using mocks is that we are no longer testing the behaviour in a test, but the implementation. However, this is not only time-consuming but also dangerous: a valid refactoring, for example changing a variable name, can cause tests to fail if that particular variable has been mocked. However, we only want our tests to fail when there are breaks in behaviour, not just when there are code changes.

However, sometimes mocking is the easiest way to create exceptions or error conditions and make sure your code handles them correctly. There are also cases where testing behaviour is unreasonable, such as when accessing a payment API or sending emails. In these cases, a good option is to test whether your code calls a specific API method at the right time and with the right parameters.

See also

• Hynek Schlawack: “Don’t Mock What You Don’t Own”

Avoid mocking with tests on multiple levels

We can also test the Items CLI without mocks by also using the API. We will not test the API, but only use it to check the behaviour of actions that are executed via the CLI. We can also test the test_add_with_owner example as follows:

def test_add_with_owner(items_db, items_cli):
    items_cli("add some task -o veit")
    expected = items.Item("some task", owner="veit", state="todo")
    all = items_db.list_items()
    assert len(all) == 1
    assert all[0] == expected

Mocking tests the implementation of the command line interface and ensures that an API call is made with certain parameters. The mixed-layer approach tests the behaviour to ensure that the result meets our expectations. This approach is much less of a change detector and has a greater chance of remaining valid during a refactoring. Interestingly, the tests are also about twice as fast:

$ pytest -s tests/cli/test_add.py::test_add_with_owner
============================= test session starts ==============================
...
configfile: pyproject.toml
plugins: cov-4.1.0, Faker-19.11.0
collected 1 item

tests/cli/test_add.py .

============================== 1 passed in 0.03s ===============================

We could also avoid mocking in another way. We could test the behaviour completely via the CLI. This might require parsing the output of the items list to check the correct database content.

In the API, add_item() returns an index and provides a get_item(index)() method to help with testing. Both methods are not available in the CLI, but could be. We could perhaps add the items get index or items info index commands so we can retrieve an item instead of having to use items list for everything. list also already supports filtering. Maybe filtering by index would work instead of adding a new command. And we could add an output to items add that says something like Item added at index 3. These changes would fall into the Design for Testability category. They also don’t seem to be deep interface interventions and perhaps should be considered in future versions.

Plugins to support mocking

So far we have focussed on the direct use of mock. However, there are many plugins that help with mocking, such as pytest-mock, which provides a mocker fixture. One advantage is that the fixture cleans up after itself, so you don’t need to use a with block like we did in our examples.

There are also some special mocking libraries:

• The following are suitable for mocking database accesses:

  • pytest-postgresql

  • pytest-mongo

  • pytest-mysql

  • pytest-dynamodb.

• You can use pytest-httpserver to test HTTP servers.

• You can use responses or betamax to mock requests.

• Other tools for different requirements are:

  • pytest-rabbitmq

  • pytest-solr

  • pytest-elasticsearch and pytest-redis.