# llvm Tests

This repository contains tests for llvm.

## Usage
Tests are executed using TMT (Test Management Tool). The repository includes a
TMT plan that runs all the tests by default. All the tests support the following
context dimensions:

* `distro`: The distribution version running the tests (e.g. `rhel-9.6`,
  `fedora-42`, `centos-stream-10`)
* `arch`: The hardware architecture of the host running the tests (e.g. `aarch64`,
`s390x`, `x86_64`)
* `compat`: Indicates a run for fedora compat-versioned packages (e.g. `llvm18`)
* `snapshot`: Indicates a run for [llvm snapshots](
  https://github.com/fedora-llvm-team/llvm-snapshots) copr build.

To run the tests:
```bash
# Run in the default virtual environment
tmt -c distro=<distro> -c arch=<arch> run -avv

# Run on a scheduled system
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname>

# Run specific tests
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname> test -n <regex>
```

## Adding New Tests
Please consider the following when adding new tests:

* Test scripts must be named `test.sh`. If a different test is needed, you need
to overwrite the `test:` key in the test `main.fmf`.
* Tests must be placed under `tests/<test_name>`. Additionally they can be
  placed into subdirectories if grouping make sense. E.g.
  `tests/regression/<test_name>`.
* This repository has a common configuration under `tests/main.fmf`, inherited
  by all tests. When adding new ones, be careful not to overwrite existing keys
  from the inherited `tests/main.fmf`.
* When setting FMF metadata keys, do merge values instead overwriting them. E.g.
use `require+:` instead `require:`. For advanced use and more examples check
[FMF documentation](https://fmf.readthedocs.io/en/stable/features.html#merging).

After creating the new test, consider running `tmt <context> tests show
<test-name>` to review that the resulting metadata such as requires, enable or
test keys are as expected. For example:
```
$ tmt -c distro=fedora-38 -c arch=aarch64 tests show binutils-plugin-ar
/tests/binutils-plugin-ar
                 summary Make sure the the LLVM plugin for binutils' ar works
             description Test that ar picks the correct plugin when archiving
                         LLVM bitcode.
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'llvm'
                    test ./test.sh
                    path /tests/binutils-plugin-ar
               framework shell
                  manual false
                     tty false
                 require binutils
                         llvm
                         llvm-libs
                         llvm-devel
                         clang
                duration 1m
                 enabled true
                  result respect
                    tier 1

```

### Templates
Use the following templates when creating new tests.

FMF template. Uncomment needed fields, remove the rest.
```yaml
summary: Descriptive summary for the test
# Maximum duration for the test
duration: 2m
# Short tests should be tier <= 1 so they're used for gating.
tier: 1
# Add an additional contact(s) if applicable
# contact+:
#   - Somebody <somebody@redhat.com>
# Additional requires
# require+:
#   - rpm-build
# Adjustments based on context dimensions
# adjust+:
#   - when: collection is defined
#     enabled: false
#     because: Test not supported in collections
# link+:
#   - verifies: https://issues.redhat.com/browse/...
# tag+:
#   - sometag
```

### Test Tags

The following tags are used to control which tests are included or excluded in
TMT plans. Please be cautious when adding or removing these tags, as they are
used by plans in multiple repositories.

* `fedora-only`: The test applies to Fedora distros only.
* `not-in-default`: Excludes the test from the default test plan. This is useful
  for resource-intensive tests that are better suited for a dedicated test plan
  or could potentially break the test system.
* `multilib`: Requires 32-bit (i686) package dependencies.

### Avoid common errors using pre-commit

This project uses [`pre-commit`](https://pre-commit.com/)

* to [lint fmf files](https://tmt.readthedocs.io/en/latest/guide.html#lint)
* to check that you don't have [trailing
  whitespaces](https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#trailing-whitespace)
* to check that [files end with a
  newline](https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#end-of-file-fixer)

Please install `pre-commit` using `pip install pre-commit` as described
[here](https://pre-commit.com/#install). Then navigate to your clone of this
project and install the git hook scripts using `pre-commit install`. This will
run pre-commit on every `git commit` that you make in this repository from now
on.

## Test specific documentation

### kernel-ark-rebuild
The `kernel-ark-rebuild` test is a special test that we use to ensure that clang,
llvm, and lld builds do not break [Always Ready Kernel](
https://gitlab.com/cki-project/kernel-ark) builds, which are done in Fedora
Rawhide. This is a virtual test comprised of 4 tests corresponding to different
build configurations for the kernel:

* base
* base/lto
* debug
* debug/lto

Due to this, this test is not regularly run on PRs opened in tests/clang, as
running the 4 tests can take about 6 hours. Instead, if you modify the test
you are expected to run it on your own and provide the results in the Pull
Request. A specific tmt test plan named `kernel-ark-gating.fmf` is provided for
that. It can be run using Testing Farm with this command:
```
testing-farm request -c initiator=human -c distro=fedora-rawhide -c arch=<your-arch> \
    --compose Fedora-Rawhide --git-url <url-to-your-fork> --git-ref <your-pr-branch> \
    --plan kernel-ark-gating
```
You need an [API key](https://docs.testing-farm.io/Testing%20Farm/0.1/onboarding.html)
to be able to use Testing Farm. If you don't have on, please ask for help running
the test when opening a Pull Request.