When %{py_reproducible_pyc_path} is used explicitly, then error our if
marshalparser returns an error. But when running in the automatic mode, only
warn. marshalparser fails for example for python2 pyc files.
Also, filter out the python2.7 directory. In Fedora, it's the major and only
source of python2 pyc files and it doesn't make much sense to try to do anything
with them.
Architecture-specific pyc files cause two problems:
- noarch packages may cause noarch packages to differ between different
architectures in the case where we build an archful package which has some
noarch subpackages,
- package rebuilds report a difference when build reproducibility is checked
and the build happened on a different architecture.
Both problems can be resolved by using the existing %py_reproducible_pyc_path
macro, but this requires maintainers to opt-in. Let's just extend this operation
to all files under /usr, to make life easier for maintainers and for folks
working on reproducible builds.
"find /usr/ -name '*.pyc'" reveals that .pyc files are installed in quite a few
locations, not just the python site and arch directories, so just apply the
fixer to the whole /usr.
%py_fix_reproducibility can be unset to opt out.
%py_reproducible_pyc_path retains its function. We can probably drop all or
almost all of its uses in Fedora, but it might be useful for people doing some
special things.
Fixes https://pagure.io/fedora-reproducible-builds/project/issue/12,
https://bugzilla.redhat.com/show_bug.cgi?id=2266767.
To deal with bootstrapping, the script does nothing if marshalparser is
not installed. python-srpm-macros Require marshalparser, but not when built
with bootstrap.
The call to marshalparser takes a bit of time. I ran it on all .pyc files on my
system and that took a few minutes. But for each single package, the time is
usually < 1 s. There are some exceptions, for example libvirt.cpython-312.*.pyc
take ~15 s each. But I don't think this really matters: the files come from a
huge package which takes a long time to compile, and adding a few dozen seconds
at the end doesn't change much. I expect that marshalparser itself might need
some optimizations. We can do that at some later point if the slowdowns become
noticeable.
1. Error out on unset variables.
2. Implement suggestions made by shellcheck: use "||" instead of "-o"" in test,
use -print0 to avoid ambiguity with filenames with embedded newlines.
(Those issues are unlikely to cause problems in the rpm environment,
but it's nice to be shellcheck-clean to use shellcheck during development.)
This adds a new `%_python_memoize` macro that caches the values of
various python macros in a lua table based on the current value of
`%__python` / `%__python3`. The Python macros are adjusted to use this
macro for memoization instead of the current approach. This way, the
macros will return values that apply to the _current_ `%__python3` value
when packagers create multi-python specfiles that toggle the value of
`%python3_pkgversion`.
Relates: https://bugzilla.redhat.com/2209055
Co-Authored-By: Miro Hrončok <miro@hroncok.cz>
%py3_install_egg is nonfunctional; setuptools removed the easy_install
entrypoint years ago.
%py3_build_egg is technically functional but has been superseded by
newer macros. Calling setup.py directly is deprecated, and
building/installing eggs to begin with is deprecated.
The macros are not used in any Fedora packages and are broken.
It's time to remove them.
When we added %_smp_mflags here, Petr Viktorin asked the question:
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/154#comment-124613
> I couldn't find docs for %_smp_mflags.
> How much of a guarantee is there that it contains no other flags than -j?
My answer was:
> %_smp_mflags is documented in https://rpm-packaging-guide.github.io/
> and used in many other RPM macros in Fedora and upstream everywhere.
> There is no official guarantee that it will never contain anything else,
> but if it does, I assume multiple things would burn.
> I am willing to take that risk.
Turns out, the world did not burn, but packagers do set %_smp_mflags to -lX,
which does not work with compileall.
Fixes https://bugzilla.redhat.com/2179149
Macros like %{python3_sitelib} were evaluated at every instance in the spec
(each time, Python was started, sysconfig was imported...).
When there were many instances (40+), it might have taken more than a minute to parse the spec file.
This way, the macros are defined via %global on first usage.
Every repetitive usage reuses the actual value.
When reviewing https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/291
we have discovered that there is no macronized way to get this part of some paths
and that packagers need to hardcode it as cpython-%{python3_version_nodots}.
This way, we have a standardized macro packagers (and other macros) can use.
Implemented in: https://github.com/python/cpython/pull/27926
We keep using it thou, because this is Python version agnostic.
Once we drop support for anything older than 3.11, we can remove it.
That'll be around ~2030. Assuming the world still exists by then.
In Koji, python3-rpm-generators are not installed during the build.
However, packagers can have them installed locally, in mock or in Copr.
This way, we make sure the automatic provides (and obsoletes)
do not magically appear only in some environments.
Since python3-rpm-macros actually requires python-rpm-macros,
the requirement is self-satisfied when the automatic provides are generated.
I.e. when %python3_pkgversion is 3.12, %__python3 is /usr/bin/python3.12
We assume that when packagers pacakge for Python 3.X, they want to change both
%python3_pkgversion and %__python3 value.
Hence instead of copy-pasting this:
%global python3_pkgversion 3.X
%global __python3 /usr/bin/python3.X
They just need to do:
%global python3_pkgversion 3.X
Packagers who want to change the value of %__python3 without touching
%python3_pkgversion can still do it:
%global __python3 /usr/bin/pypy3
Related to https://bugzilla.redhat.com/1821489
%%py{3}_check_import now respects the custom setting of %%py{3}_shebang_flags
and invokes Python with the respective values.
If %%py{3}_shebang_flags is undefined or set to no value,
there no flags are passed to Python on invoke.
Resolves: rhbz#2018615
-f: optionally read a file with module names to test
-t: bool flag - if set, filter only top-level modules
-e: optionally exclude module names matching the given glob (Unix
shell-style wildcards)
Importing all modules may cause bogus failures in some cases,
eg. when the imported code assumes there is an existing graphical window.
Such behaviour may be by design, hence for automatic processing it's
more convinient to - in some cases - check only for top-level modules
or filter out the troublemakers.
Our Pythons currently patches distutils to install packages to
/usr/lib(64)/pythonX.Y/site-packages when the $RPM_BUILD_ROOT environment
variable is set (and to /usr/local/lib(64)/pythonX.Y/site-packages otherwise).
With the deprecation of distutils [1] we want to change the patch to create
and use a different sysconfig install scheme [2].
However, we have realized that macros defined as %(%{__python3} ...) don't
"see" the environment variables set by rpmbuild because they are expanded earlier
and hence e.g. %{python3_sitelib} evaluates to
/usr/local/lib/python3.X/site-packages -- which is not desired.
To be able to reliably detect an RPM build environment by checking
the presence of the $RPM_BUILD_ROOT environment variable,
we manually set it in the macro definitions.
Since %{buildroot} in not fully populated
(e.g. it can expand literally to
/home/anna/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64),
we don't use it here.
The variable simply needs to present in the environment.
See also the analysis of the build failures when this is not done [3].
[1] https://www.python.org/dev/peps/pep-0632/
[2] https://bugs.python.org/issue43976
[3] https://src.fedoraproject.org/rpms/python3.10/pull-request/63#comment-79042
