fully automate repository changes

Instead of performing error-prone manual changes in .repo files several times
per cycle, this patchset brings fully automated handling of such edits. All the
releng person needs to do is to set `rawhide_release` and
`updates_testing_enabled` variables (and of course adjust `Version` and
`Release`), and all the repos will be auto-magically populated with the correct
values during build (which includes `enabled=0/1` and `metadata_expire=6h/7d`).
The intention is to avoid human errors which inevitably happen (an example [1]).

This means:

* Rawhide/ELN repo files will get enabled for Rawhide builds, disabled otherwise.
* Standard repo files will get enabled for non-Rawhide builds, disabled otherwise.
* Updates-testing repo will get enabled per specified configuration (ignored on
  Rawhide).
* Base repo files will have short metadata expiration during development period
  ("Branched"), long expiration during stable period.

Everything is covered with tests, to prevent humans and automatons from doing
mistakes. That covers the automated changes to repo files, but also e.g. ensuring
that updates-testing is not enabled in a Final release by mistake.

Coupled with this changeset is an adjustment to certain repo files to make sure
all the sections defined in a single repo file have the same value
`metadata_expire=`, as it makes sense. And also in the spec file the rawhide
subpackage description was moved next to the subpackage definition, they were
split by mistake in the past, it seems.

If this gets merged, it will massively simplify Releng's Mass Branching SOP in
the future [2].

[1] https://src.fedoraproject.org/rpms/fedora-repos/c/08819dbf9428d57eedbe5cd978b516f995bb8b6a?branch=f34
[2] https://docs.pagure.org/releng/sop_mass_branching.html#fedora-repos
This commit is contained in:
Kamil Páral 2021-02-23 15:05:55 +01:00
commit 1db958c4fb
10 changed files with 143 additions and 46 deletions

View file

@ -11,18 +11,19 @@
# testing and troubleshooting for development packages in conjunction
# with new test releases.
#
# More information is available at http://fedoraproject.org/wiki/Testing
# More information is available at http://fedoraproject.org/wiki/Testing
#
# Reproducible and reportable issues should be filed at
# http://bugzilla.redhat.com/.
#
# Product: Fedora
# Version: eln
[eln-baseos]
name=Fedora - ELN BaseOS - Developmental packages for the next Enterprise Linux release
baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/$basearch/os/
#metalink=https://mirrors.fedoraproject.org/metalink?repo=eln&arch=$basearch
enabled=1
enabled=AUTO_VALUE
countme=1
metadata_expire=6h
repo_gpgcheck=0
@ -36,6 +37,7 @@ name=Fedora - ELN BaseOS - Debug
baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/$basearch/debug/tree
#metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-debug&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -47,6 +49,7 @@ name=Fedora - ELN BaseOS - Source
baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/source/tree/
#metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-source&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -59,7 +62,7 @@ skip_if_unavailable=False
name=Fedora - ELN AppStream - Developmental packages for the next Enterprise Linux release
baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/AppStream/$basearch/os/
#metalink=https://mirrors.fedoraproject.org/metalink?repo=eln&arch=$basearch
enabled=1
enabled=AUTO_VALUE
countme=1
metadata_expire=6h
repo_gpgcheck=0
@ -73,6 +76,7 @@ name=Fedora - ELN AppStream - Debug
baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/AppStream/$basearch/debug/tree
#metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-debug&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -84,9 +88,9 @@ name=Fedora - ELN AppStream - Source
baseurl=https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/AppStream/source/tree/
#metalink=https://mirrors.fedoraproject.org/metalink?repo=eln-source&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

View file

@ -2,9 +2,9 @@
name=Fedora Modular $releasever - $basearch
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Modular/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-$releasever&arch=$basearch
enabled=0
enabled=AUTO_VALUE
countme=1
#metadata_expire=7d
metadata_expire=AUTO_VALUE
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -16,7 +16,7 @@ name=Fedora Modular $releasever - $basearch - Debug
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Modular/$basearch/debug/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-debug-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
metadata_expire=AUTO_VALUE
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -28,7 +28,7 @@ name=Fedora Modular $releasever - Source
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Modular/source/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-source-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
metadata_expire=AUTO_VALUE
repo_gpgcheck=0
type=rpm
gpgcheck=1

View file

@ -11,7 +11,7 @@
# testing and troubleshooting for development packages in conjunction
# with new test releases.
#
# More information is available at http://fedoraproject.org/wiki/Testing
# More information is available at http://fedoraproject.org/wiki/Testing
#
# Reproducible and reportable issues should be filed at
# http://bugzilla.redhat.com/.
@ -23,13 +23,13 @@
name=Fedora - Modular Rawhide - Developmental packages for the next Fedora release
#baseurl=http://download.example/pub/fedora/linux/development/rawhide/Modular/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular&arch=$basearch
enabled=1
enabled=AUTO_VALUE
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE
skip_if_unavailable=False
[rawhide-modular-debuginfo]
@ -37,10 +37,11 @@ name=Fedora - Modular Rawhide - Debug
#baseurl=http://download.example/pub/fedora/linux/development/rawhide/Modular/$basearch/debug/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular-debug&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE
skip_if_unavailable=False
[rawhide-modular-source]
@ -48,9 +49,9 @@ name=Fedora - Modular Rawhide - Source
#baseurl=http://download.example/pub/fedora/linux/development/rawhide/Modular/source/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-modular-source&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE
skip_if_unavailable=False

View file

@ -11,7 +11,7 @@
# testing and troubleshooting for development packages in conjunction
# with new test releases.
#
# More information is available at http://fedoraproject.org/wiki/Testing
# More information is available at http://fedoraproject.org/wiki/Testing
#
# Reproducible and reportable issues should be filed at
# http://bugzilla.redhat.com/.
@ -23,13 +23,13 @@
name=Fedora - Rawhide - Developmental packages for the next Fedora release
#baseurl=http://download.example/pub/fedora/linux/development/rawhide//Everything/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch
enabled=1
enabled=AUTO_VALUE
countme=1
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE
skip_if_unavailable=False
[rawhide-debuginfo]
@ -37,10 +37,11 @@ name=Fedora - Rawhide - Debug
#baseurl=http://download.example/pub/fedora/linux/development/rawhide/Everything/$basearch/debug/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE
skip_if_unavailable=False
[rawhide-source]
@ -48,9 +49,9 @@ name=Fedora - Rawhide - Source
#baseurl=http://download.example/pub/fedora/linux/development/rawhide/Everything/source/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-source&arch=$basearch
enabled=0
metadata_expire=6h
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch AUTO_VALUE
skip_if_unavailable=False

View file

@ -1,4 +1,5 @@
%global rawhide_release 35
%global updates_testing_enabled 0
Summary: Fedora package repositories
Name: fedora-repos
@ -106,6 +107,9 @@ Summary: Rawhide repo definitions
Requires: fedora-repos = %{version}-%{release}
Obsoletes: fedora-repos-rawhide < 33-0.7
%description rawhide
This package provides the rawhide repo definitions.
%package archive
Summary: Fedora updates archive package repository
Requires: fedora-repos = %{version}-%{release}
@ -115,9 +119,6 @@ This package provides the repo definition for the updates archive repo.
It is a package repository that contains any RPM that has made it to
stable in Bodhi and been available in the Fedora updates repo in the past.
%description rawhide
This package provides the rawhide repo definitions.
%package rawhide-modular
Summary: Rawhide modular repo definitions
Requires: fedora-repos = %{version}-%{release}
@ -187,20 +188,51 @@ done
ln -s RPM-GPG-KEY-fedora-%{version}-primary RPM-GPG-KEY-%{version}-fedora
popd
# Install repo files
install -d -m 755 $RPM_BUILD_ROOT/etc/yum.repos.d
for file in %{_sourcedir}/fedora*repo ; do
install -m 644 $file $RPM_BUILD_ROOT/etc/yum.repos.d
done
# Enable or disable repos based on current release cycle state.
%if %{rawhide_release} == %{version}
rawhide_enabled=1
stable_enabled=0
testing_enabled=0
%else
rawhide_enabled=0
stable_enabled=1
testing_enabled=%{updates_testing_enabled}
%endif
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-{rawhide,eln}*.repo; do
sed -i "s/^enabled=AUTO_VALUE$/enabled=${rawhide_enabled}/" $repo || exit 1
done
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora{,-modular,-updates,-updates-modular}.repo; do
sed -i "s/^enabled=AUTO_VALUE$/enabled=${stable_enabled}/" $repo || exit 1
done
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-updates-testing{,-modular}.repo; do
sed -i "s/^enabled=AUTO_VALUE$/enabled=${testing_enabled}/" $repo || exit 1
done
# Adjust Rawhide repo files to include Rawhide+1 GPG key.
# This is necessary for the period when Rawhide gets bumped to N+1 and packages
# start to be signed with a newer key. Without having the key specified in the
# repo file, the system would consider the new packages as untrusted.
rawhide_next=$((%{rawhide_release}+1))
for repo in %{_sourcedir}/fedora-rawhide*.repo; do
sed -ir "s@^gpgkey=.*@& file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-${rawhide_next}-\$basearch@" \
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-rawhide*.repo; do
sed -i "/^gpgkey=/ s@AUTO_VALUE@file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-${rawhide_next}-\$basearch@" \
$repo || exit 1
done
# Install repo files
install -d -m 755 $RPM_BUILD_ROOT/etc/yum.repos.d
for file in %{_sourcedir}/fedora*repo ; do
install -m 644 $file $RPM_BUILD_ROOT/etc/yum.repos.d
# Set appropriate metadata_expire in base repo files (6h before Final, 7d after)
%if "%{release}" < "1"
expire_value='6h'
%else
expire_value='7d'
%endif
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora{,-modular}.repo; do
sed -i "/^metadata_expire=/ s/AUTO_VALUE/${expire_value}/" \
$repo || exit 1
done
# Install ostree remote config
@ -210,23 +242,82 @@ install -m 644 %{_sourcedir}/fedora-compose.conf $RPM_BUILD_ROOT/etc/ostree/remo
%check
# assert all rawhide/eln repos are set to enabled only when this is rawhide
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-{rawhide,eln}*.repo; do
%if %{rawhide_release} == %{version}
grep 'enabled=1' $repo
%else
grep 'enabled=1' $repo && exit 1 || :
%endif
# Make sure all repo variables were substituted
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/*.repo; do
if grep -q AUTO_VALUE $repo; then
echo "ERROR: Repo $repo contains an unsubstituted placeholder value"
exit 1
fi
done
# make sure the Rawhide+1 key wasn't forgotten to be created
# Make sure correct repos were enabled/disabled
enabled_repos=(fedora-cisco-openh264)
disabled_repos=(fedora-updates-archive)
%if %{rawhide_release} == %{version}
enabled_repos+=(fedora-rawhide fedora-rawhide-modular fedora-eln)
disabled_repos+=(fedora fedora-modular fedora-updates fedora-updates-modular \
fedora-updates-testing fedora-updates-testing-modular)
%else
enabled_repos+=(fedora fedora-modular fedora-updates fedora-updates-modular)
disabled_repos+=(fedora-rawhide fedora-rawhide-modular fedora-eln)
%if %{updates_testing_enabled}
enabled_repos+=(fedora-updates-testing fedora-updates-testing-modular)
%else
disabled_repos+=(fedora-updates-testing fedora-updates-testing-modular)
%endif
%endif
for repo in ${enabled_repos[@]}; do
if ! grep -q 'enabled=1' $RPM_BUILD_ROOT/etc/yum.repos.d/${repo}.repo; then
echo "ERROR: Repo $repo should have been enabled, but it isn't"
exit 1
fi
done
for repo in ${disabled_repos[@]}; do
if grep -q 'enabled=1' $RPM_BUILD_ROOT/etc/yum.repos.d/${repo}.repo; then
echo "ERROR: Repo $repo should have been disabled, but it isn't"
exit 1
fi
done
# Make sure updates-testing is not enabled in a Final (stable) release
%if "%{release}" >= "1"
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-updates-testing{,-modular}.repo; do
if grep -q 'enabled=1' $repo; then
echo "ERROR: Repo $repo should be disabled in a stable release, but it isn't"
exit 1
fi
done
%endif
# Make sure metadata_expire was correctly set
%if "%{release}" < "1"
expire_value='6h'
%else
expire_value='7d'
%endif
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora{,-modular}.repo; do
lines=$(grep '^metadata_expire=' $repo | sort | uniq)
if [ "$(echo "$lines" | wc -l)" -ne 1 ]; then
echo "ERROR: Non-matching metadata_expire lines in $repo: $lines"
exit 1
fi
if test "$lines" != "metadata_expire=${expire_value}"; then
echo "ERROR: Wrong metadata_expire value in $repo: $lines"
exit 1
fi
done
# Make sure the Rawhide+1 key wasn't forgotten to be created
rawhide_next=$((%{rawhide_release}+1))
test -n "$rawhide_next" || exit 1
if ! test -f $RPM_BUILD_ROOT/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-${rawhide_next}-primary; then
echo "ERROR: GPG key for Fedora ${rawhide_next} is not present"
exit 1
fi
# make sure the Rawhide+1 key is present in Rawhide repo files
# Make sure the Rawhide+1 key is present in Rawhide repo files
for repo in $RPM_BUILD_ROOT/etc/yum.repos.d/fedora-rawhide*.repo; do
gpg_lines=$(grep '^gpgkey=' $repo)
if test -z "$gpg_lines"; then

View file

@ -2,7 +2,7 @@
name=Fedora Modular $releasever - $basearch - Updates
#baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Modular/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-f$releasever&arch=$basearch
enabled=0
enabled=AUTO_VALUE
countme=1
repo_gpgcheck=0
type=rpm

View file

@ -2,7 +2,7 @@
name=Fedora Modular $releasever - $basearch - Test Updates
#baseurl=http://download.example/pub/fedora/linux/updates/testing/$releasever/Modular/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-modular-f$releasever&arch=$basearch
enabled=0
enabled=AUTO_VALUE
countme=1
repo_gpgcheck=0
type=rpm

View file

@ -2,7 +2,7 @@
name=Fedora $releasever - $basearch - Test Updates
#baseurl=http://download.example/pub/fedora/linux/updates/testing/$releasever/Everything/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch
enabled=0
enabled=AUTO_VALUE
countme=1
repo_gpgcheck=0
type=rpm

View file

@ -2,7 +2,7 @@
name=Fedora $releasever - $basearch - Updates
#baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
enabled=0
enabled=AUTO_VALUE
countme=1
repo_gpgcheck=0
type=rpm

View file

@ -2,9 +2,9 @@
name=Fedora $releasever - $basearch
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/$basearch/os/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
enabled=0
enabled=AUTO_VALUE
countme=1
#metadata_expire=7d
metadata_expire=AUTO_VALUE
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -16,7 +16,7 @@ name=Fedora $releasever - $basearch - Debug
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
metadata_expire=AUTO_VALUE
repo_gpgcheck=0
type=rpm
gpgcheck=1
@ -28,7 +28,7 @@ name=Fedora $releasever - Source
#baseurl=http://download.example/pub/fedora/linux/releases/$releasever/Everything/source/tree/
metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
metadata_expire=AUTO_VALUE
repo_gpgcheck=0
type=rpm
gpgcheck=1