Testing

reST formatting

Whether the Sphinx documentation is written in valid reStructuredText format can be checked with sphinx-lint. We usually include this in our pre-commit configuration:

.pre-commit-config.yaml
- repo: https://github.com/sphinx-contrib/sphinx-lint
  rev: v1.0.0
  hooks:
    - id: sphinx-lint
      types: [rst]

See also

With Sybil you can not only check reStructuredText, but also Markdown and Myst, for example. Sybil can also check code blocks in the documentation with either pytest or Unittest.

Code

With the built-in Python library Doctest, you can also test code in your documentation with the doctest.testfile() method:

import doctest

doctest.testfile("example.rst")

This short script executes and checks all interactive Python examples contained in the example.rst file. The content of the file is treated as if it were a single huge docstring.

See also

A simple example can be found in the Python documentation: Simple Usage: Checking Examples in a Text File.

Another way to test code in documentation is pytest-doctestplus.

Code formatting

The formatting of code blocks can be checked with blacken-docs, which uses Black. We usually integrate the library via the pre-commit framework:

- repo: https://github.com/adamchainz/blacken-docs
  rev: "v1.12.1"
  hooks:
  - id: blacken-docs
    additional_dependencies:
    - black

blacken-docs currently supports the following black options:

Vale goes beyond spelling and grammar checks. It also checks the language style: Is what is said repeated? Is the language too informal? Is the language inconsistent? Are undesirable clichés being used? Or is the language sexist?

Vale is used by many open source projects, including

The following styles come with Vale itself:

Microsoft

An implementation of the Microsoft Writing Style Guide.

Google

An implementation of the style guide for the Google developer documentation style guide.

write-good

An implementation of the guidelines enforced by the write-good linter.

proselint

An implementation of the guidelines enforced by the proselint linter.

Joblint

An implementation of the directives enforced by the Joblint linter.

Vale is configured in the .vale.ini file:

.vale.ini
StylesPath = styles
MinAlertLevel = suggestion
Packages = https://github.com/cusyio/cusy-vale/archive/refs/tags/v0.1.0.zip

[*.{md,rst}]
BasedOnStyles = cusy-en

See also

You should then update your .gitignore file if necessary:

.gitignore
styles/*

You can configure Vale for the pre-commit framework with:

.pre-commit-config.yaml
- repo: https://github.com/errata-ai/vale
  rev: v3.7.1
  hooks:
  - id: vale sync
    pass_filenames: false
    args: [sync]
  - id: vale
    args: [--output=line, --minAlertLevel=error, .]

Docstrings coverage

interrogate checks your codebase for missing documentation strings and generates a shields.io-like badge.

You can configure interrogate in the pyproject.toml file, for example:

pyproject.toml
[project.optional-dependencies]
tests = [
    "coverage[toml]",
    "interrogate",
    "pytest>=6.0",
]

[tool.interrogate]
ignore-init-method = true
ignore-init-module = false
ignore-magic = false
ignore-semiprivate = false
ignore-private = false
ignore-module = false
ignore-property-decorators = false
fail-under = 95
exclude = ["tests/functional/sample", "setup.py", "docs"]
verbose = 0
omit-covered-files = false
quiet = false
whitelist-regex = []
ignore-regex = []
color = true

See also

You can now insert interrogate into your tox file, for example with

tox.ini
[testenv:doc]
deps = interrogate
skip_install = true
commands =
    interrogate --quiet --fail-under 95 src tests

You can also use interrogate with pre-commit:

.pre-commit-config.yaml
repos:
  - repo: https://github.com/econchick/interrogate
    rev: 1.7.0
    hooks:
      - id: interrogate
        args: [--quiet, --fail-under=95]
        pass_filenames: false