# clang tests

This repository contains tests for clang.

## Usage
Tests are executed using TMT (Test Management Tool). The repository includes a
TMT plan that runs all the tests by default. All the tests support the following
context dimensions:

* `distro`
* `arch`
* `collection` (only applicable to RHEL 7)

To run the tests:
```bash
# Run in the default virtual environment
tmt -c distro=<distro> -c arch=<arch> run -avv

# Run on a scheduled system
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname>

# Run specific tests
tmt -c distro=<distro> -c arch=<arch> run -avv provision -h connect -g <ip_or_hostname> test -n <regex>
```

## Adding New Tests
Please consider the following when adding new tests:

* Test scripts must be named `runtest.sh`. If a different test is needed, you need
to overwrite the `test:` key in the test `main.fmf`.
* Tests must be placed under `tests/<test_name>`. Additionally they can be
  placed into subdirectories if grouping make sense. E.g.
  `tests/regression/<test_name>`.
* This repository has a common configuration under `tests/main.fmf`, inherited
  by all tests. When adding new ones, be careful not to overwrite existing keys
  from the inherited `tests/main.fmf`.
* When setting FMF metadata keys, do merge values instead overwriting them. E.g.
use `require+:` instead `require:`. For advanced use and more examples check
[FMF documentation](https://fmf.readthedocs.io/en/stable/features.html#merging).

After creating the new test, consider running `tmt <context> tests show
<test-name>` to review that the resulting metadata such as requires, enable or
test keys are as expected. For example:
```
$ tmt -c distro=fedora-38 tests show ucrt64-toolchain
/tests/ucrt64-toolchain
                 summary Test that ucrt64 toolchain is detected
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'clang'
                    test ./runtest.sh
                    path /tests/ucrt64-toolchain
               framework shell
                  manual false
                     tty false
                 require 'ucrt64-gcc-c++' and 'clang'
                duration 5m
                 enabled true
                  result respect

$ tmt -c distro=rhel-9.3 tests show ucrt64-toolchain
/tests/ucrt64-toolchain
                 summary Test that ucrt64 toolchain is detected
                 contact Jesus Checa Hidalgo <jcheca@redhat.com>
               component 'llvm-toolset' and 'clang'
                    test ./runtest.sh
                    path /tests/ucrt64-toolchain
               framework shell
                  manual false
                     tty false
                 require 'ucrt64-gcc-c++' and 'clang'
                duration 5m
                 enabled false
                  result respect
```

### Templates
Use the following templates when creating new tests.

FMF template. Uncomment needed fields, remove the rest.
```yaml
summary: Descriptive summary for the test
# Maximum duration for the test
duration: 2m
# Short tests should be tier <= 1 so they're used for gating.
tier: 1
# Add an additional contact(s) if applicable
# contact+:
#   - Somebody <somebody@redhat.com>
# Additional requires
# require+:
#   - rpm-build
# Adjustments based on context dimensions
# adjust+:
#   - when: collection is defined
#     enabled: false
#     because: Test not supported in collections
# link+:
#   - verifies: https://issues.redhat.com/browse/...
# tag+:
#   - sometag
```

### Avoid common errors using pre-commit

This project uses [`pre-commit`](https://pre-commit.com/)

* to [lint fmf files](https://tmt.readthedocs.io/en/latest/guide.html#lint)
* to check that you don't have [trailing
  whitespaces](https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#trailing-whitespace)
* to check that [files end with a
  newline](https://github.com/pre-commit/pre-commit-hooks?tab=readme-ov-file#end-of-file-fixer)

Please install `pre-commit` using `pip install pre-commit` as described
[here](https://pre-commit.com/#install). Then navigate to your clone of this
project and install the git hook scripts using `pre-commit install`. This will
run pre-commit on every `git commit` that you make in this repository from now
on.

## Test specific documentation

### kernel-ark-rebuild
The `kernel-ark-rebuild` test is a special test that we use to ensure that clang,
llvm, and lld builds do not break [Always Ready Kernel](
https://gitlab.com/cki-project/kernel-ark) builds, which are done in Fedora
Rawhide. This is a virtual test comprised of 4 tests corresponding to different
build configurations for the kernel:

* base
* base/lto
* debug
* debug/lto

Due to this, this test is not regularly run on PRs opened in tests/clang, as
running the 4 tests can take about 6 hours. Instead, if you modify the test
you are expected to run it on your own and provide the results in the Pull
Request. A specific tmt test plan named `kernel-ark-gating.fmf` is provided for
that. It can be run using Testing Farm with this command:
```
testing-farm request -c initiator=human -c distro=fedora-rawhide -c arch=<your-arch> \
    --compose Fedora-Rawhide --git-url <url-to-your-fork> --git-ref <your-pr-branch> \
    --plan kernel-ark-gating
```
You need an [API key](https://docs.testing-farm.io/Testing%20Farm/0.1/onboarding.html)
to be able to use Testing Farm. If you don't have on, please ask for help running
the test when opening a Pull Request.