Packaging Python tools

Last update: 2021-10-20.

So you wrote a nice Python program and would like to share it, but setuptools documentation is giving you headaches? Let me try to help. It's normal to feel a bit lost because the Python packaging world has been a bit of a mess for a long time and is currently undergoing a lot of changes to modernise it and provide a more stable interface to developers.

This document assumes you are familiar with the Python language. It is for SIMPLE programs and libraries: for complex projects using multiple languages or other complicated setup process it will obviously not be sufficient. That's all!

We will review some concepts, see how you can package a program or library using the common setuptools package, and how modern packaging requires changing a few things.

Lexicon

Here are brief descriptions of the high-level elements of Python packaging. I do not go into details of what build back-ends are or what is the exact timeline that lead to this or that, it's just to clear things up a bit.

Repositories

PyPI is a public repository where people upload Python programs and libraries. Anyone can post their own programs there for everyone else, and by default the pip tool installs packages from PyPI.

Note that PyPI is not strictly required for package utilities as it is possible to install a Python package from other sources (filesystem, Git repo, etc).

Packages

distutils is a Python package to build and install other packages. You rarely have to deal with it directly and mostly use setuptools, which is another, more user-friendly Python package to build, install and distribute other packages. You generally use it to create OR install a package, so you need it installed with your Python distribution. On Debian, it's the python3-setuptools package. Even though it is not part of the standard library, setuptools has been the de facto utility to produce and distribute Python packages for years now.

Files

Let's review files commonly found in packaging discussions.

`setup.py` is a legacy way people used to provide to distribute packages. It's a script using either distutils, setuptools or another build backend and running a few commands to install your package in a hopefully standard way. Due to being a Python script, it's cumbersome to detect what are the build time dependencies, what back-end it uses, using which options so it is becoming less and less popular.

Its static brother `setup.cfg` is a config file describing the package declaratively so it's easier for build systems to process it. Despite the generic name, this file is meant to be used with setuptools.

Finally, `pyproject.toml` is a config file, made standard by PEP 518, defining how a project should be built: which build back-end to use, what are the dependencies to *build* a project (not even to run it, just to produce a working package), etc.

And last but not least, wheels (with the whl extension) are packages that can be shared and installed. When you download a package from PyPI, you usually receive the source and/or a wheel and install it. This is a generally easy and portable solution so that's probably what you want to produce.

Packaging 101 using setuptools

OK all of that is great, but how do I package my Python program?! For most needs, you can still use setuptools with a simple setup.cfg script to achieve the following:

Create wheels to share;

Describe runtime dependencies;

Embed resources like images;

Provide “entry points”, scripts that can be provided to your user's path;

Add metadata for PyPI, and more.

In this section, I will not cover the pyproject.toml file and assume you just want to use setuptools to get the job done, with a simple example. To be clear, this is not the “standard” or “clean” way to do it (there is not one simple answer) but rather a straightforward way to use setuptools without thinking too much about corner cases. In the next section we will cover what's the modern and proper way to specify your build system using pyproject.toml.

Project structure

Let's say I wrote a fancy Gemini browser named Baboop and I'm eager to share it. The project is structured as follows:

To package it with setuptools, we will need additional files:

An entry point (often called __main__.py);

A setup.cfg.

The entry point

In Python you do not have to specify an entry point to your program, but when you wish to control execution you often use this weird snippet:

This is good practice because the `main` function will be executed only if the script is executed, but not if it's imported by another module, and we will use exactly this for our tool. Put your entry point code (the main function and the if block) in a __main__.py module, in the root package baboop, and you are done!

setup.cfg

At the root of the project, let's create a simple setup.cfg file. It looks a bit like INI files, and in our case will contain three sections:

Let's go with the metadata section first. Nothing too special, `name` is your package name, `long_description` here uses the special format `file: ` to tell the build back-end that the long description of the project is in fact the readme file, and `classifiers` are optional tags used on PyPI.

The options section is more interesting:

`packages` is a list of source packages to embed in your package. Here we have the `baboop` package AND its subpackage `baboop.utils`. For small projects, specifying by hand all the packages to embed is not an issue, but for larger projects you should check the automatic package discovery features of setuptools or whatever build system you end up using.

`python_requires` specifies the Python versions you support.

`setup_requires` specifies the build tools required to build the package, here setuptools is enough.

`install_requires` specifies the dependencies of your package. The syntax to express exact deps' version ranges is too complex to describe here; in the example we require the requests package, version 2.25 or more (but not 3.x).

And finally comes our entry points section where we specify through the `console_scripts` list the name of the executable we want to ship with our package: a script named “baboop”, that will run the `main` function of the `baboop.__main__` module. When installed, our package should provide the “baboop” script in the user's path.

Create the wheel

To continue, ensure your environment have both setuptools and wheel installed, using `pip3 list`. As we're going the quick way, we know we want to use setuptools as build back-end and that a wheel is convenient both for us and for our users, whether we want to use PyPI or not.

We have two quick ways of invoking setuptools: create a dummy script or just use a self-sufficient Python command.

The dummy setup.py script that follows can be called with the command `python3 setup.py bdist_wheel` which will create a wheel, which is the simplest way to use setuptools as our build back-end:

To keep the repository clean and avoiding creating a scarily deprecated setup.py we can use this command instead, which does exactly the same thing:

If everything goes right, a bunch of new folders should have appeared, and your newly created wheel will be in the dist folder. That's it, you have your package!

Install the wheel

You probably want to install it, maybe in a virtualenv, to check if the installation process works smoothly. Installing a wheel is as simple as this:

And your package is installed. Check that baboop (or the less silly name of your own tool) is now in your path, and you can now share it with friends and give them this command, or upload the wheel to PyPI.

Modern packaging

The previous section covers a simple way to use setuptools but in the future developers want to move to an installation process description independent of the build back-end. This means tools such as pip will have a standard way to build your package when someone wants to install it, and not simply run some setup.py with fingers crossed, hoping that the target system has setuptools or any other build dependency installed.

In this section, we will reuse our previous example but port it to the modern workflow. Nothing to be afraid of!

Express build requirements

It is important to note that setuptools itself is not deprecated or anything, just that having a setup.py in your source tree is not recommended. Indeed, if that script has `import setuptools`, there is no simple way for tools like pip to know that setuptools is required before running that script, so the idea is to express in a static file that building your project requires setuptools, and this is what the pyproject.toml file is for.

In our example, this can be specified like so:

That's it! This file is enough for PEP 517 compliant build front-ends to know how to build your package, which means that pip now knows how to install it without the need for the dummy setup.py (you can delete it if you created it) or the arcane Python command. The build front-end will see that both setuptools and wheel are required, and do what's needed to use them (ask the user to install them, do it automatically in a separate env, etc).

Build the wheel

You can now build your project's wheel without explicitely calling setuptools, using a PEP 517 compliant build front-end. A simple one is PyPA's “build”, which you can find on PyPI, used like this:

PyPA's “build” documentation

On PyPI

Upload the wheel to PyPI

This is out of the scope of this document, but you can find good instructions in the relevant section of this link to setup your PyPI account and run upload commands against a test instance of PyPI before doing it on the real repository.

Packaging Python programs: Uploading the distribution archives

Expressing requirements without PyPI

If for some reason you do not want to use PyPI to store your packages and reuse them, and there are a lot of valid reasons, you may wonder how to work out dependencies between your projects because asking users to manually install dependencies through “git+https” URLs can be a bit deterrent.

Imagine we have a library testlib_high that does fancy high level stuff using another library testlib_low. In testlib_high we want to express the requirements for testlib_low:

Package and install… Woops!

pip can't find your package because it is not on PyPI. What can we do? Until recently it was possible to define “dependency links” which are URLs to use when looking up dependencies, but it is now deprecated and will soon be removed. Fortunately with recent versions of pip we can directly put our “git+https” as the place to fetch a specific package:

This should both work with git+https and git+ssh URLs. The egg fragment part of the URL is necessary, it's usually safe to use the same name as the package.

Note that this method requires you to express, in the testlib_low repository, your build system or at least have a dummy setup.py script (as in the previous section) because pip will need one of them to know how to process this dependency URL.

Building and installing the wheel should now automatically fetch and install your dependency from the repository! 🙌

References

PEP 518

Building and distributing packages with setuptools

Why you shouldn't invoke setup.py directly?