When %{py_reproducible_pyc_path} is used explicitly, then error our if
marshalparser returns an error. But when running in the automatic mode, only
warn. marshalparser fails for example for python2 pyc files.
Also, filter out the python2.7 directory. In Fedora, it's the major and only
source of python2 pyc files and it doesn't make much sense to try to do anything
with them.
Architecture-specific pyc files cause two problems:
- noarch packages may cause noarch packages to differ between different
architectures in the case where we build an archful package which has some
noarch subpackages,
- package rebuilds report a difference when build reproducibility is checked
and the build happened on a different architecture.
Both problems can be resolved by using the existing %py_reproducible_pyc_path
macro, but this requires maintainers to opt-in. Let's just extend this operation
to all files under /usr, to make life easier for maintainers and for folks
working on reproducible builds.
"find /usr/ -name '*.pyc'" reveals that .pyc files are installed in quite a few
locations, not just the python site and arch directories, so just apply the
fixer to the whole /usr.
%py_fix_reproducibility can be unset to opt out.
%py_reproducible_pyc_path retains its function. We can probably drop all or
almost all of its uses in Fedora, but it might be useful for people doing some
special things.
Fixes https://pagure.io/fedora-reproducible-builds/project/issue/12,
https://bugzilla.redhat.com/show_bug.cgi?id=2266767.
To deal with bootstrapping, the script does nothing if marshalparser is
not installed. python-srpm-macros Require marshalparser, but not when built
with bootstrap.
The call to marshalparser takes a bit of time. I ran it on all .pyc files on my
system and that took a few minutes. But for each single package, the time is
usually < 1 s. There are some exceptions, for example libvirt.cpython-312.*.pyc
take ~15 s each. But I don't think this really matters: the files come from a
huge package which takes a long time to compile, and adding a few dozen seconds
at the end doesn't change much. I expect that marshalparser itself might need
some optimizations. We can do that at some later point if the slowdowns become
noticeable.
1. Error out on unset variables.
2. Implement suggestions made by shellcheck: use "||" instead of "-o"" in test,
use -print0 to avoid ambiguity with filenames with embedded newlines.
(Those issues are unlikely to cause problems in the rpm environment,
but it's nice to be shellcheck-clean to use shellcheck during development.)
A new script brp-fix-pyc-reproducibility creates an opt-in way of how to fix
problems with the reproducibility of byte-compiled Python files. The script
uses marshalparser [0] which currently doesn't provide solutions for all issues
but can fix at least problems with reference flags. For more info see
this Bugzilla [1].
If you want to use this new feature, you need to define
`%py_reproducible_pyc_path` to specify a path you want to fix `.pyc`
files in (recursively) and build-require /usr/bin/marshalparser.
if you forget to build-require the parser. The error message is:
```
+ /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0
Bytecompiling .py files below /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64/usr/lib/python3.9 using /usr/bin/python3.9
+ /usr/lib/rpm/redhat/brp-fix-pyc-reproducibility /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64
ERROR: If %py_reproducible_pyc_path is defined, you have to also BuildRequire: /usr/bin/marshalparser !
error: Bad exit status from /var/tmp/rpm-tmp.UUJr4v (%install)
```
A build fails if the parser is not able to parse any of the `.pyc` files.
And finally, if a build is properly configured it produces fixed `.pyc` files.
Currently, `.pyc` files in the tldr package contain a lot of unused reference flags:
```
$ dnf install -y tldr
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
… long output …
190 - Flag_ref(byte=9610, type='TYPE_SHORT_ASCII_INTERNED', content=b'init', usages=0)
191 - Flag_ref(byte=9633, type='TYPE_SHORT_ASCII_INTERNED', content=b'source', usages=0)
192 - Flag_ref(byte=9651, type='TYPE_SHORT_ASCII_INTERNED', content=b'argv', usages=0)
193 - Flag_ref(byte=9657, type='TYPE_SHORT_ASCII_INTERNED', content=b'print_help', usages=0)
194 - Flag_ref(byte=9669, type='TYPE_SHORT_ASCII_INTERNED', content=b'stderr', usages=0)
195 - Flag_ref(byte=9682, type='TYPE_SHORT_ASCII_INTERNED', content=b'parse_args', usages=0)
196 - Flag_ref(byte=9737, type='TYPE_SHORT_ASCII_INTERNED', content=b'encode', usages=0)
197 - Flag_ref(byte=9782, type='TYPE_SHORT_ASCII_INTERNED', content=b'parser', usages=0)
198 - Flag_ref(byte=9790, type='TYPE_SHORT_ASCII_INTERNED', content=b'options', usages=0)
199 - Flag_ref(byte=9799, type='TYPE_SHORT_ASCII_INTERNED', content=b'rest', usages=0)
200 - Flag_ref(byte=9821, type='TYPE_SHORT_ASCII_INTERNED', content=b'result', usages=0)
202 - Flag_ref(byte=10022, type='TYPE_SHORT_ASCII_INTERNED', content=b'__main__', usages=0)
203 - Flag_ref(byte=10102, type='TYPE_SHORT_ASCII_INTERNED', content=b'argparse', usages=0)
204 - Flag_ref(byte=10433, type='TYPE_SHORT_ASCII_INTERNED', content=b'__name__', usages=0)
205 - Flag_ref(byte=10463, type='TYPE_SHORT_ASCII_INTERNED', content=b'<module>', usages=0)
```
This new feature fixes them:
```
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
<empty output>
```
[0] https://github.com/fedora-python/marshalparser
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1686078
