llvm Tests

This repository contains tests for llvm.

Usage

Tests are executed using TMT (Test Management Tool). The repository includes a TMT plan that runs all the tests by default. All the tests support the following context dimensions:

• distro
• arch
• collection (only applicable to RHEL 7)

To run the tests:

# Run in the default virtual environment
tmt -c distro=<distro> -c arch=<arch> run -avv

# Run on a scheduled system
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname>

# Run specific tests
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname> test -n <regex>

Adding New Tests

Please consider the following when adding new tests:

• Test scripts must be named test.sh. If a different test is needed, you need to overwrite the test: key in the test main.fmf.
• Tests must be placed under tests/<test_name>. Additionally they can be placed into subdirectories if grouping make sense. E.g. tests/regression/<test_name>.
• This repository has a common configuration under tests/main.fmf, inherited by all tests. When adding new ones, be careful not to overwrite existing keys from the inherited tests/main.fmf.
• When setting FMF metadata keys, do merge values instead overwriting them. E.g. use require+: instead require:. For advanced use and more examples check FMF documentation.

After creating the new test, consider running tmt <context> tests show <test-name> to review that the resulting metadata such as requires, enable or test keys are as expected. For example:

$ tmt -c distro=fedora-38 -c arch=aarch64 tests show binutils-plugin-ar
/tests/binutils-plugin-ar
                 summary Make sure the the LLVM plugin for binutils' ar works
             description Test that ar picks the correct plugin when archiving
                         LLVM bitcode.
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'llvm'
                    test ./test.sh
                    path /tests/binutils-plugin-ar
               framework shell
                  manual false
                     tty false
                 require binutils
                         llvm
                         llvm-libs
                         llvm-devel
                         clang
                duration 1m
                 enabled true
                  result respect
                    tier 1

Templates

Use the following templates when creating new tests.

FMF template. Uncomment needed fields, remove the rest.

summary: Descriptive summary for the test
# Maximum duration for the test
duration: 2m
# Short tests should be tier <= 1 so they're used for gating.
tier: 1
# Add an additional contact(s) if applicable
# contact+:
#   - Somebody <somebody@redhat.com>
# Additional requires
# require+:
#   - rpm-build
# Adjustments based on context dimensions
# adjust+:
#   - when: collection is defined
#     enabled: false
#     because: Test not supported in collections
# link+:
#   - verifies: https://issues.redhat.com/browse/...
# tag+:
#   - sometag

Avoid common errors using pre-commit

Have you ever been asked to add a newline to the end of a file in your pull request? Don't worry no more.

This project uses pre-commit to validate that certain files (e.g. fmf) are not broken.

Pre-commit also checks that you don't have trailing whitespaces and that every file ends with a newline.

Please install pre-commit using pip install pre-commit as described here. Then navigate to your clone of this project and install the git hook scripts using pre-commit install. This will run pre-commit on every git commit that you make in this repository from now on.

The nice part is this: pre-commit will not only complain but fix your code when it runs and if your git commit is rejected because pre-commit complains, you can just run git add -p to check and pick the changes that it made in order to fix your commit. Then just re-run your git commit command.

From time to time you might want to run pre-commit autoupdate to refresh config to the latest version.

Example of tmt linting

Let's take the tests/llvm-config/main.fmf file:

summary: Make sure llvm-config symlink is properly set up
tier: 1
duration: 1m
extra-summary: /tools/llvm/llvm-config
extra-task: /tools/llvm/llvm-config
extra-nitrate: TC#0614096

And change duration: 1m to durations: 1m.

Now add this change with git add -p and then run git commit to see how pre-commit complains:

$ git commit
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
tmt lint.................................................................Failed
- hook id: tmt-lint
- exit code: 1

.fmf/version
        warn: /tests/llvm-config: - 'durations' does not match any of the regexes: '^extra-'
/tests/llvm-config
warn C000 key "durations" not recognized by schema, and does not match "^extra-" pattern
warn C000 fmf node failed schema validation
warn C001 summary key is missing
fail T001 unknown key "durations" is used
pass T002 test script is defined
pass T003 directory path is absolute
pass T004 test path '/home/kkleine/src/tests/llvm/tests/llvm-config' does exist
skip T005 legacy relevancy not detected
skip T006 legacy 'coverage' field not detected
skip T007 not a manual test
skip T008 not a manual test
skip T009 library/file requirements not used

Notice the error unknown key "durations" is used. Isn't that nice?