llvm Tests

This repository contains tests for llvm.

Usage

Tests are executed using TMT (Test Management Tool). The repository includes a TMT plan that runs all the tests by default. All the tests support the following context dimensions:

• distro: The distribution version running the tests (e.g. rhel-9.6, fedora-42, centos-stream-10)
• arch: The hardware architecture of the host running the tests (e.g. aarch64, s390x, x86_64)
• compat: Indicates a run for fedora compat-versioned packages (e.g. llvm18)
• snapshot: Indicates a run for llvm snapshots copr build.

To run the tests:

# Run in the default virtual environment
tmt -c distro=<distro> -c arch=<arch> run -avv

# Run on a scheduled system
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname>

# Run specific tests
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname> test -n <regex>

Adding New Tests

Please consider the following when adding new tests:

• Test scripts must be named test.sh. If a different test is needed, you need to overwrite the test: key in the test main.fmf.
• Tests must be placed under tests/<test_name>. Additionally they can be placed into subdirectories if grouping make sense. E.g. tests/regression/<test_name>.
• This repository has a common configuration under tests/main.fmf, inherited by all tests. When adding new ones, be careful not to overwrite existing keys from the inherited tests/main.fmf.
• When setting FMF metadata keys, do merge values instead overwriting them. E.g. use require+: instead require:. For advanced use and more examples check FMF documentation.

After creating the new test, consider running tmt <context> tests show <test-name> to review that the resulting metadata such as requires, enable or test keys are as expected. For example:

$ tmt -c distro=fedora-38 -c arch=aarch64 tests show binutils-plugin-ar
/tests/binutils-plugin-ar
                 summary Make sure the the LLVM plugin for binutils' ar works
             description Test that ar picks the correct plugin when archiving
                         LLVM bitcode.
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'llvm'
                    test ./test.sh
                    path /tests/binutils-plugin-ar
               framework shell
                  manual false
                     tty false
                 require binutils
                         llvm
                         llvm-libs
                         llvm-devel
                         clang
                duration 1m
                 enabled true
                  result respect
                    tier 1

Templates

Use the following templates when creating new tests.

FMF template. Uncomment needed fields, remove the rest.

summary: Descriptive summary for the test
# Maximum duration for the test
duration: 2m
# Short tests should be tier <= 1 so they're used for gating.
tier: 1
# Add an additional contact(s) if applicable
# contact+:
#   - Somebody <somebody@redhat.com>
# Additional requires
# require+:
#   - rpm-build
# Adjustments based on context dimensions
# adjust+:
#   - when: collection is defined
#     enabled: false
#     because: Test not supported in collections
# link+:
#   - verifies: https://issues.redhat.com/browse/...
# tag+:
#   - sometag

Test Tags

The following tags are used to control which tests are included or excluded in TMT plans. Please be cautious when adding or removing these tags, as they are used by plans in multiple repositories.

• fedora-only: The test applies to Fedora distros only.
• not-in-default: Excludes the test from the default test plan. This is useful for resource-intensive tests that are better suited for a dedicated test plan or could potentially break the test system.
• multilib: Requires 32-bit (i686) package dependencies.

Avoid common errors using pre-commit

This project uses pre-commit

• to lint fmf files
• to check that you don't have trailing whitespaces
• to check that files end with a newline

Please install pre-commit using pip install pre-commit as described here. Then navigate to your clone of this project and install the git hook scripts using pre-commit install. This will run pre-commit on every git commit that you make in this repository from now on.

Test specific documentation

kernel-ark-rebuild

The kernel-ark-rebuild test is a special test that we use to ensure that clang, llvm, and lld builds do not break Always Ready Kernel builds, which are done in Fedora Rawhide. This is a virtual test comprised of 4 tests corresponding to different build configurations for the kernel:

• base
• base/lto
• debug
• debug/lto

Due to this, this test is not regularly run on PRs opened in tests/clang, as running the 4 tests can take about 6 hours. Instead, if you modify the test you are expected to run it on your own and provide the results in the Pull Request. A specific tmt test plan named kernel-ark-gating.fmf is provided for that. It can be run using Testing Farm with this command:

testing-farm request -c initiator=human -c distro=fedora-rawhide -c arch=<your-arch> \
    --compose Fedora-Rawhide --git-url <url-to-your-fork> --git-ref <your-pr-branch> \
    --plan kernel-ark-gating

You need an API key to be able to use Testing Farm. If you don't have on, please ask for help running the test when opening a Pull Request.