diff --git a/README.md b/README.md index 23d921a..760e85e 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,104 @@ -# compiler-rt +# compiler-rt Tests -Tests for compiler-rt \ No newline at end of file +This repository contains tests for compiler-rt. + +## Usage +Tests are executed using TMT (Test Management Tool). The repository includes a +TMT plan that runs all the tests by default. All the tests support the following +context dimensions: +* `distro` +* `arch` +* `collection` (only applicable to RHEL 7) + +To run the tests: +```bash +# Run in the default virtual environment +tmt -c distro= -c arch= run -avv + +# Run on a scheduled system +tmt -c distro= -c arch= run -avv provision -h connect -g + +# Run specific tests +tmt -c distro= -c arch= run -avv provision -h connect -g test -n +``` + +## Adding New Tests +Please consider the following when adding new tests: +* Test scripts must be named `test.sh`. If a different test is needed, you need +to overwrite the `test:` key in the test `main.fmf`. +* Tests must be placed under `tests/`. Additionally they can be + placed into subdirectories if grouping make sense. E.g. + `tests/regression/`. +* This repository has a common configuration under `tests/main.fmf`, inherited + by all tests. When adding new ones, be careful not to overwrite existing keys + from the inherited `tests/main.fmf`. +* When setting FMF metadata keys, do merge values instead overwriting them. E.g. +use `require+:` instead `require:`. For advanced use and more examples check +[FMF documentation](https://fmf.readthedocs.io/en/stable/features.html#merging). + +After creating the new test, consider running `tmt tests show +` to review that the resulting metadata such as requires, enable or +test keys are as expected. For example: +``` +$ tmt -c distro=fedora-39 tests show sanity +/tests/sanity + summary compiler-rt sanity test with signed int overflow + contact Jesus Checa Hidalgo + component 'llvm-toolset' and 'compiler-rt' + test ./test.sh + path /tests/sanity + framework shell + manual false + tty false + require 'clang' and 'compiler-rt' + duration 5m + enabled true + result respect + tier 1 + +$ $ tmt -c collection=llvm-toolset-15.0 tests show sanity +/tests/sanity + summary compiler-rt sanity test with signed int overflow + contact Jesus Checa Hidalgo + component 'llvm-toolset' and 'compiler-rt' + test scl enable llvm-toolset-15.0 -- ./test.sh + path /tests/sanity + framework shell + manual false + tty false + require llvm-toolset-15.0-clang + llvm-toolset-15.0-compiler-rt + duration 5m + enabled true + result respect + tier 1 + + +``` + +### Templates +Use the following templates when creating new tests. + +FMF template. Uncomment needed fields, remove the rest. +```yaml +summary: Descriptive summary for the test +# Maximum duration for the test +duration: 2m +# Short tests should be tier <= 1 so they're used for gating. +tier: 1 +# Add an additional contact(s) if applicable +# contact+: +# - Somebody +# Additional requires +# require+: +# - rpm-build +# Adjustments based on context dimensions +# adjust+: +# - when: collection is defined +# enabled: false +# because: Test not supported in collections +# link+: +# - verifies: https://issues.redhat.com/browse/... +# tag+: +# - sometag +```