QA checks and warnings

This section explains Python-related QA checks and the resulting QA warnings that can be output while running the package manager or related tooling.

Improved QA warning reporting in Portage

Normally, Portage outputs QA warnings at specific phases of the build process. They are usually interspersed with other verbose output, and they are easy to miss, especially when building multiple packages in a single batch.

To make them easier to catch, Portage’s elog system can be used to repeat all the QA warnings once emerge exits. The required “echo” module is already enabled by default, however it skips QA warnings by default. To change that, set in your make.conf:

PORTAGE_ELOG_CLASSES="log warn error qa"

For more information on using Portage’s elog system, please refer to make.conf.example included in the Portage distribution.

Compiled bytecode-related warnings

To improve performance, the Python interpreter compiles Python sources into bytecode. CPython and PyPy3 feature three optimization levels that impact the bytecode size:

1. no optimizations (the default)

2. -O that removes assert statements from code

3. -OO that removes assert statements and docstrings

Normally, the compiled bytecode is stored on disk in .pyc files (historically, Python 2 has also used .pyo files). When these files are present and up-to-date, Python loads the precompiled bytecode from them rather than creating it from the source code, improving module loading time. When they are missing, Python normally creates them automatically if it has write permissions to the respective directory.

In Gentoo, we aim for packages to always byte-compile all their Python files to all supported optimization levels. Besides improving module loading performance, it ensures that the Python interpreters will not attempt to write compiled bytecode themselves, effectively creating files that are not monitored by the package manager (and therefore e.g. are not removed when the respective package is uninstalled) or causing sandbox violations while building other packages.

The Gentoo repository features a QA check to ensure that all installed Python modules are byte-compiled for all optimization levels supported by the respective Python interpreter, and that no stray compiled files exist. This check is implemented using an auxiliary command found in app-portage/gpep517-7 and newer. This check can also be run manually (with machine-readable output) using e.g.:

$ python3.10 -m gpep517 verify-pyc --destdir=/tmp/portage/dev-python/trimesh-3.12.9/image
missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py
missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py
missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py

Modules are not byte-compiled

The most common QA warning that can be noticed while building packages indicates that at least some of the expected .pyc files are missing. For example:

* QA Notice: This package installs one or more Python modules that are
* not byte-compiled.
* The following files are missing:
*
*   /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc
*   /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc
*   /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc

* QA Notice: This package installs one or more Python modules that are
* not byte-compiled.
* The following files are missing:
*
*   /usr/lib/python3.10/site-packages/blueman/__pycache__/Constants.cpython-310.opt-2.pyc
*   /usr/lib/python3.10/site-packages/blueman/__pycache__/DeviceClass.cpython-310.opt-2.pyc
*   /usr/lib/python3.10/site-packages/blueman/__pycache__/Functions.cpython-310.opt-2.pyc
*   /usr/lib/python3.10/site-packages/blueman/__pycache__/Sdp.cpython-310.opt-2.pyc
[...]

There are three common causes for these warnings:

1. The package’s build system nor the ebuild do not byte-compile installed Python modules — the warning lists all optimization levels for all installed modules.

2. The package’s build system byte-compiles installed modules only for a subset of optimization levels — the warning lists all modules but only for a subset of levels (the second example in the example snippet).

3. The package installs .py files with incorrect syntax that can not be byte-compiled and usually trigger syntax errors during the install phase (the first example in the above snippet).

For the first two cases, the better solution is to patch the respective build system to perform byte-compilation for all optimization levels. An acceptable workaround is to call python_optimize from the ebuild (note that in PEP517 mode, distutils-r1 does that unconditionally).

For the third case, the only real solution is to submit a fix upstream that renames files that do not contain valid Python modules to use another suffix. For example, the template triggering the QA warning in trimesh package could be renamed from .py to .py.tmpl.

Stray compiled bytecode

The following QA warning indicates that there are stray .pyc files that are not clearly matching any installed Python module-implementation pair:

* QA Notice: This package installs one or more compiled Python modules
* that do not match installed modules (or their implementation).
* The following files are stray:
*
*   /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-35.pyc
*   /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-36.pyc
*   /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-38.pyc

There are two common causes for this:

1. The package is shipping precompiled .pyc files and installing them along with .py modules. The ebuild should remove the stray files in src_prepare then.

2. The ebuild is attempting to remove some .py files after they have been byte-compiled. It needs to be modified to either remove them prior to the byte-compilation stage, or to fix the build system not to install them in the first place.

Stray top-level files in site-packages

distutils-r1 checks for the common mistake of installing unexpected files that are installed top-level into the site-packages directory. An example error due to that looks like the following:

* The following unexpected files/directories were found top-level
* in the site-packages directory:
*
*   /usr/lib/python3.10/site-packages/README.md
*   /usr/lib/python3.10/site-packages/LICENSE
*   /usr/lib/python3.10/site-packages/CHANGELOG
*
* This is most likely a bug in the build system.  More information
* can be found in the Python Guide:
* https://projects.gentoo.org/python/guide/qawarn.html#stray-top-level-files-in-site-packages

In general, it is desirable to prepare a fix for the build system and submit it upstream. However, it is acceptable to remove the files locally in the ebuild while waiting for a release with the fix.

The subsequent sections describe the common causes and the suggested fixes.

Example or test packages installed by setuptools

Many packages using the setuptools build system utilize the convenient find_packages() method to locate the Python sources. In some cases, this method also wrongly grabs top-level test directories or other files that were not intended to be installed.

For example, the following invocation will install everything that looks like a Python package from the source tree:

setup(
    packages=find_packages())

The correct fix for this problem is to add an exclude parameter that restricts the installed package list, for example:

setup(
    packages=find_packages(exclude=["tests", "tests.*"]))

Note that if the top-level tests package has any subpackages, both tests and tests.* need to be listed.

If setup.cfg is used instead, the excludes are specified as follows:

[options.packages.find]
exclude =
    tests
    tests.*

If pyproject.toml is used:

[tool.setuptools.packages.find]
exclude = [
    "tests",
    "tests.*",
]

For reference, see custom discovery in setuptools documentation.

Documentation files installed by Poetry

It is a relatively common problem that packages using the Poetry build system are installing documentation files (such as README) to the site-packages directory. This is because of incorrect include use in pyproject.toml. For example, consider the following configuration:

include = [
    "CHANGELOG",
    "README.md",
    "LICENSE"
]

The author meant to include these files in the source distribution packages. However, the include key applies to wheels as well, effectively including them in files installed into site-packages.

To fix that, you need to specify file formats explicitly, for every entry:

include = [
    { path = "CHANGELOG", format = "sdist" },
    { path = "README.md", format = "sdist" },
    { path = "LICENSE", format = "sdist" },
]

For reference, see include and exclude in Poetry documentation.

Deprecated PEP 517 backends

flit.buildapi

Some packages are still found using the historical flit build backend. Their pyproject.toml files contain a section similar to the following:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

This backend requires installing the complete flit package manager. Instead, the package should be fixed upstream to use flit_core per flit build system section documentation instead:

[build-system]
requires = ["flit_core"]
build-backend = "flit_core.buildapi"

flit_core produces identical artifacts to flit. At the same time, it reduces the build-time dependency footprint and therefore makes isolated PEP 517 builds faster.

poetry.masonry.api

A similar problem applies to the packages using poetry. The respective pyproject.toml files contain:

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Instead, the lightweight poetry-core module should be used per poetry PEP-517 documentation:

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

poetry-core produces identical artifacts to poetry. It has smaller dependency footprint and makes isolated builds much faster.

setuptools.build_meta:__legacy__

Some packages using setuptools specify the following:

[build-system]
requires = ["setuptools>=40.8.0", "wheel"]
build-backend = "setuptools.build_meta:__legacy__"

This is incorrect, as the legacy backend is intended to be used only as an implicit fallback. All packages should be using the regular backend instead:

[build-system]
requires = ["setuptools>=40.8.0"]
build-backend = "setuptools.build_meta"

Please also note that the wheel package should not be listed as a dependency, as it is an implementation detail and it was always implicitly returned by the backend. Unfortunately, due to prolonged documentation error, a very large number of packages still specifies it, and other packages tend to copy that mistake.