Passing parameters to tmt

[Venefilyn]

When it comes to tmt, you have the ability to pass name, filter to reduce the test suite overall. But this is not something you can pass when doing /packit test. It would be great to have the ability to use filters this way so that we can have a limited plan running for every PR but trigger different plans depending on the command sent

If we have to test for kernel modules in a specific PR, I'd want the ability to do /packit test --filter "tag:kmods" and have integration tests started for that

[TomasTomecek]

Thank you for opening the RFE!

Would such a test run be required to pass in order to merge a pull request?

We can easily implement the command arguments and forward them to TF - the untrivial part is the reporting: what would be the name for the check (should it be a new one? should we rewrite the generic?); how should these be treated when a new push happens, etc.

[Venefilyn]

Would such a test run be required to pass in order to merge a pull request?

I'd say it depends. I don't want to run all the integration tests we have for each PR, it's becoming too much for that. If we reduce to having a limited test suite for PRs, we'd want to run all of the integration tests for the main branch instead. But ideally, if a feature is touching kmods or similar, it would be a nice to have feature to be able to run tests that hasn't been run before in the PR based on a filter

That being said, I don't think it's a requirement for merging PRs. But it does give ease of mind when a PR is a bit more complex to be able to run specific tags etc.

what would be the name for the check (should it be a new one? should we rewrite the generic?)

IMO it should be a new one if possible, run existing plans with the filter and possibly prepend the filter to the name

how should these be treated when a new push happens

If the filter is more for creating new checks that run. I wouldn't have it run on consequent commits, I don't think it would make sense as you it's an extra thing for ease of mind rather than a requirement. But maybe it's possible to allow for both somehow? Labels? Different command? /packit test-filter-persistent "tag:kmods" and /packit test-filter "tag:kmods" for instance

[TomasTomecek]

Would such a test run be required to pass in order to merge a pull request?

I'd say it depends. I don't want to run all the integration tests we have for each PR, it's becoming too much for that. If we reduce to having a limited test suite for PRs, we'd want to run all of the integration tests for the main branch instead. But ideally, if a feature is touching kmods or similar, it would be a nice to have feature to be able to run tests that hasn't been run before in the PR based on a filter

That being said, I don't think it's a requirement for merging PRs. But it does give ease of mind when a PR is a bit more complex to be able to run specific tags etc.

Makes sense to me. So it would be an interface to run a specific set of tests in CI for your convenience: use upstream pull request as a venue to integrate the work rather than running those tests locally & set up the environment manually. Agreed, I think that with Packit and Testing Farm, we're heading in this direction of utilizing GitHub and services to make smooth development experience.

how should these be treated when a new push happens

If the filter is more for creating new checks that run. I wouldn't have it run on consequent commits, I don't think it would make sense as you it's an extra thing for ease of mind rather than a requirement. But maybe it's possible to allow for both somehow? Labels? Different command? /packit test-filter-persistent "tag:kmods" and /packit test-filter "tag:kmods" for instance

Oki, let's treat this as a subsequent RFE.

Thank you Eric for the clarifications! I applied the "discuss" label to discuss this at the next Packit architecture meeting.

[TomasTomecek]

Related to #1627

[kkaarreell]

Maybe custom tmt context can help with the filtering now when #1797 has been implemented.

[psss]

This feature would be very useful. We discussed something similar today with @lukaszachy and agreed it would be nice to have something like an alias for an extra job which would hold additional configuration. The config could look like this:

- job: tests
  name: full
  manual: true
  trigger: pull_request
  targets:
    - fedora-all
  tf_extra_params:
    environments:
      - tmt:
          context:
            how: "full"

This test job would not be executed by default but would be manual, triggered using the following comment:

/packit test full

Results would be reported as a separate check using the provided name. This would allow to define additional tests jobs for common scenarios and easily trigger them when desired using a concise command. Thoughts?