Limitations of mocking

One of the biggest problems with using mocks is that, in a test, we are no longer testing the behaviour but the implementation. However, this is not only time-consuming but also dangerous: a valid refactoring – for example, changing a variable name – can cause tests to fail if that particular variable has been mocked. However, we want our tests to fail only when there are breaks in behaviour, not simply because of code changes.

Sometimes, though, mocking is the easiest way to generate exceptions or error conditions and ensure that your code handles them correctly. There are also cases where testing behaviour is impractical, such as when accessing a payment API or sending emails. In these cases, it is a good option to test whether your code calls a specific API method at the right time and with the correct parameters.

See also

• “Don’t Mock What You Don’t Own” by Hynek Schlawack

Note

Even in agent-based software development, we try to avoid mocking as much as possible:

AGENTS.md

- Prefer testing real code where possible. Use mocks and `monkeypatch` when absolute necessary. Try to avoid mocking as much as possible.

See also

• Testing

Avoiding mocks with multi-level testing

We can also test the Tasks CLI without using mocks by utilising the API. In doing so, we will not be testing the API itself, but simply using it to verify the behaviour of actions executed via the CLI. We can also test the test_add_with_owner example as follows:

def test_add_with_owner(tasks_db, tasks_cli):
    tasks_cli("add some task -o veit")
    expected = tasks.Task("some task", owner="veit", state="todo")
    all = tasks_db.list_tasks()
    assert len(all) == 1
    assert all[0] == expected

Mocking tests the implementation of the command-line interface and ensures that an API call is made with specific parameters. With the mixed-layer approach, the behaviour is tested to ensure that the result meets our expectations. This approach is much less of a change detector and is more likely to remain valid during refactoring. Interestingly, the tests are also about twice as fast:

$ uv run pytest -s tests/cli/test_add.py::test_add_with_owner
============================= test session starts ==============================
…
configfile: pyproject.toml
plugins: cov-4.1.0, Faker-19.11.0
collected 1 item

tests/cli/test_add.py .

============================== 1 passed in 0.03s ===============================

We could also avoid mocking in another way. We could test the behaviour entirely via the CLI. To do this, we might need to parse the output of the tasks list to verify that the database contents are correct.

In the API, add_task() returns an index and provides a get_task(index) method, which helps with testing. Neither of these methods is available in the CLI, but they could be. We could perhaps add the commands tasks get index or tasks info index so that we can retrieve a task without having to use tasks list for everything. list already supports filtering. Perhaps filtering by index would work, rather than adding a new command. And we could add output to tasks add that says something like Task added at index 3. These changes would fall under the category of Design for Testability. They do not appear to involve any major changes to the interface and might be worth considering in future versions.