Embracing the pyproject.toml future of packaging Python projects with setuptools

Last week I worked on a new template for PyPI python projects for SQuaRE, my team at Rubin Observatory.¹ One of the banner features in this new template is that we did away with setup.cfg and went all-in with pyproject.toml for defining our projects' packaging.

PEPs 517, 518 and 621

First, a reminder of what’s going on with Python packaging. PEP 517 and PEP 518 introduced the idea of a “build-system independent format for source trees.” Up to that point, the assumption was that all Python projects would be built with setuptools.² A build-system independent format lets you specify an alternative build backend, but in a standardized way, so that a front-end like pip can build and install the package regardless. In the SQuaRE team, though we use setuptools, we embraced PEP 517 and added pyroject.toml files containing a [build-system] table. However, our project metadata remained in setup.cfg, where setuptools expects it.

Then in PEP 621, Python introduced a standardized schema for storing a project’s metadata (everything from its name to its dependencies) in a [project] table of project.toml files. At this point, it seemed as if we could finally drop setup.cfg files from our projects entirely.

Well, almost. The last straw was for setuptools itself to support pyproject.toml configurations for aspects of the build that are specific to setuptools, like package discovery. Currently, the support for a [tool.setuptools] table in pyproject.toml is in beta, but it exists and that was good enough for our team to start supporting it.

And with that, we can now package our Python projects with only a pyproject.toml file, dropping setup.cfg and (almost) setup.py (more on this later).

Our template pyproject.toml

This is the example pyproject.toml from our template (and here’s the related Jinja2/cookiecutter template for it). You can also see it in action with Kafkit, one of our Python packages for Kafka.

[project]
# https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
name = "example"
description = "Short one-sentence description of the package"
license = {file = "LICENSE"}
readme= "README.md"
keywords = [
    "rubin",
    "lsst",
]
# https://pypi.org/classifiers/
classifiers = [
    "Development Status :: 4 - Beta",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Intended Audience :: Developers",
    "Natural Language :: English",
    "Operating System :: POSIX",
    "Typing :: Typed",
]
requires-python = ">=3.8"
dependencies = []
dynamic = ["version"]

[project.optional-dependencies]
dev = [
    # Testing
    "coverage[toml]",
    "pytest",
    "pytest-asyncio",
    "pre-commit",
    "mypy",
    # Documentation
    "sphinx",
    "documenteer",
    "lsst-sphinx-bootstrap-theme",
    "sphinx-prompt",
    "sphinx-automodapi",
    "myst-parser",
    "markdown-it-py[linkify]",
]

[project.urls]
# Homepage = "https://example.lsst.io"
Source = "https://github.com/lsst-sqre/example"

[build-system]
requires = [
    "setuptools>=61",
    "wheel",
    "setuptools_scm[toml]>=6.2"
]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]

[tool.setuptools.packages.find]
# https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html
where = ["src"]
include = ["example*"]

[tool.coverage.run]
parallel = true
branch = true
source = ["example"]

[tool.coverage.paths]
source = ["src", ".tox/*/site-packages"]

[tool.coverage.report]
show_missing = true
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "if self.debug:",
    "if settings.DEBUG",
    "raise AssertionError",
    "raise NotImplementedError",
    "if 0:",
    "if __name__ == .__main__.:",
    "if TYPE_CHECKING:"
]

[tool.black]
line-length = 79
target-version = ["py38"]
exclude = '''
/(
    \.eggs
  | \.git
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | build
  | dist
)/
'
# Use single-quoted strings so TOML treats the string like a Python r-string
#  Multi-line strings are implicitly treated by black as regular expressions

[tool.pydocstyle]
# Reference: http://www.pydocstyle.org/en/stable/error_codes.html
convention = "numpy"
add_select = [
    "D212" # Multi-line docstring summary should start at the first line
]
add-ignore = [
    "D105", # Missing docstring in magic method
    "D102", # Missing docstring in public method (needed for docstring inheritance)
    "D100", # Missing docstring in public module
    # Below are required to allow multi-line summaries.
    "D200", # One-line docstring should fit on one line with quotes
    "D205", # 1 blank line required between summary line and description
    "D400", # First line should end with a period
    # Properties shouldn't be written in imperative mode. This will be fixed
    # post 6.1.1, see https://github.com/PyCQA/pydocstyle/pull/546
    "D401",
]

[tool.isort]
profile = "black"
line_length = 79
known_first_party = ["example", "tests"]
skip = ["docs/conf.py"]

[tool.pytest.ini_options]
asyncio_mode = "strict"
python_files = [
    "tests/*.py",
    "tests/*/*.py"
]

[tool.mypy]
disallow_untyped_defs = true
disallow_incomplete_defs = true
ignore_missing_imports = true
strict_equality = true
warn_redundant_casts = true
warn_unreachable = true
warn_unused_ignores = true
# plugins =

This pyproject.toml is built for a “src/” Python package layout. The [tool.setuptools.packages.find] table is where we configure this.

setup.py is still necessary, for now

Our original goal with this new template was to do away with the “legacy” packaging files: setup.cfg and setup.py. We did succeed in doing away with setup.cfg since not only has setuptools moved to pyproject.toml, but nearly every other tool using setup.cfg has moved their configurations as well.³

It might seem like setup.py is also obsolete. However, in our experience, some projects still require it for editable installs pip install -e .. We often use editable installs during development to rapidly test and prototype, so this is a useful feature for us. For that reason, we’ve had to re-add a basic setup.py file to our projects:

from setuptools import setup

setup()

What’s surprising is that, anecdotally, not all setuptools-based projects seem to need a setup.py for their editable installs.

Regardless, setuptools issue 2816 is worth a subscription to learn when this has changed.

Concatenated READMEs with Markdown may not work

With the new [project] table in pyproject.toml, you can set a readme field that links to your project’s README file. This effectively replaces the old “long description” field, where, back in the setup.py days, we’d write a small function to open and insert text from the README file. This is a nice, clean way of standardizing that practice.

Something I liked to do in our setup.cfg files was to actually concatenated multiple files into the long description:

[metadata]
name = documenteer
long_description = file: README.rst, CHANGELOG.rst, LICENSE
long_description_content_type = text/x-rst

This way, the project’s page on PyPI would include not only the README but also the change log, followed by the license. Nice.

In pyproject.toml, the same can be accomplished by making the README dynamic:

[project]
dynamic = ["readme"]

[tool.setuptools.dynamic]
readme = {file = ["README.rst", "CHANGELOG.rst"]}

And this works. Where things went wrong for me is using Markdown for the README and CHANGELOG:

[project]
dynamic = ["readme"]

[tool.setuptools.dynamic]
readme = {file = ["README.md", "CHANGELOG.md"]}

When I looked inside the built wheel’s METADATA file, the Description-Content-Type would be text/re-structured-text instead of text/markdown. Normally, the tooling successfully detects the content type for the description, but this may not be working for dynamic READMEs. Something to be aware of, as setuptools' pyproject.toml support is being developed.

Jonathan Sick’s Timeline