rubygem-cucumber stack test suite

Test suite for th rubygem-cucumber stack to ensure functional Ruby cucumber test runner.

Using test plans for rubygem packages

Simply use the plan import TMT feature.

Let's say we want to test rubygem-cucumber-gherkin package, we will use the following TMT plan:

# plans/all.fmf
plan:
  import:
    url: https://src.fedoraproject.org/tests/rubygem-cucumber
    name: /plans/full/all

This will use the generic plan that runs all the tests described in the Layout section.

Next, after the plan is comitted, we run the in the command-line:

$ tmt run --environment KOJI_TASK_ID=120746946 --all

This will run all steps with global defaults for the build 120746946 which is a regular Koji build where we can source required package for reverse dependency build and isntall tests.

Layout

The BUILD_PACKAGE variable

Some of the tests use BUILD_PACKAGE variable.

This variable is mainly used in tests to define which package to build/install as part of the test to verify the behavior.

Nearly all metadata is inherited via tmt inheritance so individual tests only need to define the BUILD_PACKAGE variable in the main.fmf test plan.

Example:

# file rubygem-rspec.fmf
environment:
  BUILD_PACKAGE: rubygem-rspec

Basic Test Suites

Basic high-level tests of ensuring functional rubygem-cucumber. Tests include running a test suite of an upstream project or running a test suite containing Aruba test cases.

Reverse Dependencies Build

Set of tests ensuring that the main users of rubygem-cucumber (such as rubygem-rspec) continue building after a rubygem-cucumber stack's dependency update.

A failure in this class means we broke build of reverse dependencies.

Reverse Dependencies Install

Ensure that the main users of the rubygem-cucumber dependencies at runtime (rubygem-cucumber and rubygem-aruba) still install with the updated dependency.

A failure in this class means we broke ability to use cucumber in any context with the update.

Running

Most tests, especially reverse_dependencies* tests rely on an environment variable:

• KOJI_TASK_ID -- build result (built RPMs) from koji build. Usually provided by Testing Farm in Fedora CI

The KOJI_TASK_ID variable

This variable is expected to be provided via Testing Farm executed in a Fedora PR after a passing scratch build. It has to be a koji task ID, so for local runs, any Task ID is enough.

To execute tests locally we will either need to have a scratch build task ID can utilize production builds.

See section above Using test plans for rubygem packages for practical example.