Adding more Python libraries

Although Python’ „Batteries included“ philosophy means that you can already do a lot with the default installation of Python, there will inevitably come a situation where you need functionality that is not included in Python.

If you need a third-party module that is not pre-built for your platform, you must use its source distribution. However, this causes two problems:

  • To install the source distribution, you need to find and download it.

  • Certain Python paths and authorisations of your system are expected.

Python offers pip as a current solution for both problems. pip tries to find the module in the Python Package Index (PyPI), downloads it and all dependencies and takes care of the installation. You can also call pypi.org directly and search for packages or filter the packages by category.

Warning

Never install anything with pip into the global Python, not even with the --user flag. Always use venv. This way you avoid contaminating your Python installation with libraries that you install and then forget about. Every time you need to do something new, you should create a new virtual environment. This will also avoid library conflicts between different projects.

Tip

We recommend that you configure pip so that it is not possible to install Python packages globally. To do this, you can enter the following in your ~/.config/pip/pip.conf:

[global]
require-virtualenv = true

venv

A virtual environment (virtualenv) is a self-contained directory structure that contains both an installation of Python and the additional packages. Since the entire Python environment is contained in this directory, the libraries and modules installed there cannot collide with those in the main system or in other virtual environments, so that different applications can use different versions of Python and its packages. Creating and using a virtual environment is a two-step process:

  1. First we create a project directory and then the virtual environment in it:

    $ mkdir myproj
$ cd myproj
$ python3 -m venv .venv

    > mkdir myproj
> cd myproj
> py -m venv .venv

    This creates the environment with Python and pip in a directory called .venv.

  2. You can then activate this environment so that the next time you call python, it will use the Python from your new environment:

    $ . .venv/bin/activate

    > .venv\Scripts\activate

  3. Install Python packages only for this virtual environment, for example the popular pandas library:

    (.venv) $ python -m pip install pandas

    (.venv) > python.exe -m pip install pandas

  4. If you want to finish your work on this project, you can deactivate the virtual environment again with

    (.venv) $ deactivate

    (.venv) > deactivate

See also

pip

The basic syntax of pip is quite simple:

$ python -m pip install pandas

If you want to specify a particular version of a package, you can simply append the version numbers:

$ python -m pip install pandas==2.2.2

or

$ python -m pip install "pandas>=2"

Proxy server

To install Python packages via a proxy server, you can enter the following:

python -m pip install --proxy http://USER_NAME:{PASSWORD}@PROXYSERVER_NAME:PORT PKG_NAME

You can also save the proxy server permanently as an environment variable:

for example in the ~/.bashrc with:

export HTTP_PROXY=http://{USER_NAME}:{PASSWORD}@{PROXYSERVER_NAME}:{PORT}

Add the following line to the environment variables:

set HTTP_PROXY={PROXYSERVER_NAME}:{PORT}

Pinning the version numbers

… of packages

For a stable environment, it makes sense to specify the version numbers of the dependencies.

Tip

In none of our library projects does so much happen that the Git history should mainly consist of updates. We only restrict the version numbers to be used in the event of problems. For apps, however, we specify the version numbers.

We use PDM to specify the versions for our applications and maintain cross-platform lock files. PDM also supports the management of virtual environments with pdm venv activate.

… of Python

In contrast to applications, our packages usually support more than one Python version. Nevertheless, we usually add the current standard version in .python-version to packages as well:

.python-version
3.12

The nice thing about this is that we can use the same file in GitHub Actions as input for setup-python:

.github/workflows/ci.yml
  docs:
    name: Build docs and check links
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: pandoc/actions/setup@v1
    - uses: actions/setup-python@v5
      with:
        # Keep in sync with .readthedocs.yaml

In our GitLab CI/CD pipelines, however, we use requires-python from the pyproject.toml file to build Docker containers with the appropriate Python version.

uv

uv simplifies the creation of an initial project structure and the management of your dependencies.

Installation

uv uv does not depend on Python. Pre-compiled, standalone binaries can be installed on Linux, macOS and Windows:

$ curl -LsSf https://astral.sh/uv/install.sh | sh

> powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

uv updates itself regularly with this installation.

Automatic shell completion

To activate automatic shell completion for uv commands, carry out one of the following steps:

Specify your shell, for example with echo $SHELL, then execute one of the following commands:

$ echo 'eval "$(uv generate-shell-completion bash)"' >> ~/.bashrc
$ echo 'eval "$(uv generate-shell-completion zsh)"' >> ~/.zshrc
$ echo 'uv generate-shell-completion fish | source' >> ~/.config/fish/config.fish
$ echo 'eval (uv generate-shell-completion elvish | slurp)' >> ~/.elvish/rc.elv

Add-Content -Path $PROFILE -Value '(& uv generate-shell-completion powershell) | Out-String | Invoke-Expression'
Add-Content -Path $PROFILE -Value '(& uvx --generate-shell-completion powershell) | Out-String | Invoke-Expression'

Then restart the shell or call up source with your shell configuration file.

Python installation

With uv not only older CPython versions can be installed, but also, for example, PyPy with uv python install pypy@3.12 or free-threaded Python 3.13 with uv python install --python-preference only-managed 3.13t.

Create project structure

Depending on whether you want to create a library or an application, uv can create a suitable project structure.