llvm Tests

This repository contains tests for llvm.

Usage

Tests are executed using TMT (Test Management Tool). The repository includes a TMT plan that runs all the tests by default. All the tests support the following context dimensions:

• distro
• arch
• collection (only applicable to RHEL 7)

To run the tests:

# Run in the default virtual environment
tmt -c distro=<distro> -c arch=<arch> run -avv

# Run on a scheduled system
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname>

# Run specific tests
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname> test -n <regex>

Adding New Tests

Please consider the following when adding new tests:

• Test scripts must be named test.sh. If a different test is needed, you need to overwrite the test: key in the test main.fmf.
• Tests must be placed under tests/<test_name>. Additionally they can be placed into subdirectories if grouping make sense. E.g. tests/regression/<test_name>.
• This repository has a common configuration under tests/main.fmf, inherited by all tests. When adding new ones, be careful not to overwrite existing keys from the inherited tests/main.fmf.
• When setting FMF metadata keys, do merge values instead overwriting them. E.g. use require+: instead require:. For advanced use and more examples check FMF documentation.

After creating the new test, consider running tmt <context> tests show <test-name> to review that the resulting metadata such as requires, enable or test keys are as expected. For example:

$ tmt -c distro=fedora-38 -c arch=aarch64 tests show binutils-plugin-ar
/tests/binutils-plugin-ar
                 summary Make sure the the LLVM plugin for binutils' ar works
             description Test that ar picks the correct plugin when archiving
                         LLVM bitcode.
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'llvm'
                    test ./test.sh
                    path /tests/binutils-plugin-ar
               framework shell
                  manual false
                     tty false
                 require binutils
                         llvm
                         llvm-libs
                         llvm-devel
                         clang
                duration 1m
                 enabled true
                  result respect
                    tier 1

Templates

Use the following templates when creating new tests.

FMF template. Uncomment needed fields, remove the rest.

summary: Descriptive summary for the test
# Maximum duration for the test
duration: 2m
# Short tests should be tier <= 1 so they're used for gating.
tier: 1
# Add an additional contact(s) if applicable
# contact+:
#   - Somebody <somebody@redhat.com>
# Additional requires
# require+:
#   - rpm-build
# Adjustments based on context dimensions
# adjust+:
#   - when: collection is defined
#     enabled: false
#     because: Test not supported in collections
# link+:
#   - verifies: https://issues.redhat.com/browse/...
# tag+:
#   - sometag

Avoid common errors using pre-commit

This project uses pre-commit

• to lint fmf files
• to check that you don't have trailing whitespaces
• to check that files end with a newline

Please install pre-commit using pip install pre-commit as described here. Then navigate to your clone of this project and install the git hook scripts using pre-commit install. This will run pre-commit on every git commit that you make in this repository from now on.

Test specific documentation

kernel-ark-rebuild

The kernel-ark-rebuild test is a special test that we use to ensure that clang, llvm, and lld builds do not break Always Ready Kernel builds, which are done in Fedora Rawhide. This is a virtual test comprised of 4 tests corresponding to different build configurations for the kernel:

• base
• base/lto
• debug
• debug/lto

Due to this, this test is not regularly run on PRs opened in tests/clang, as running the 4 tests can take about 6 hours. Instead, if you modify the test you are expected to run it on your own and provide the results in the Pull Request. A specific tmt test plan named kernel-ark-gating.fmf is provided for that. It can be run using Testing Farm with this command:

testing-farm request -c initiator=human -c distro=fedora-rawhide -c arch=<your-arch> \
    --compose Fedora-Rawhide --git-url <url-to-your-fork> --git-ref <your-pr-branch> \
    --plan kernel-ark-gating

You need an API key to be able to use Testing Farm. If you don't have on, please ask for help running the test when opening a Pull Request.