`max-parallel` in `matrix`

[pritidesai]

Configuring a matrix to generate combinations can be done in two different ways:

1. Pipeline governed combinations

The pipeline author specifies values of each matrix parameter in the definition. Eg.:

    matrix:
      params:
        - name: platform
          value:
          - linux
          - mac
        - name: browser
          value:
          - safari
          - chrome
  ...

This matrix specification generates four different combinations i.e. it creates four different taskRuns in parallel.

3. User specified combinations

    matrix:
      params:
        - name: platform
          value: $(params.platform[*])
        - name: browser
          value: $(params.browser[*])
  ...

This matrix specification can generate any number of combinations with a minimum of 1 to maximum of X (which is enforced by the controller level configuration default-max-matrix-combinations-count). The pipeline author does not have control over the parameters. If a user sets these params which generates some higher number of combinations (but under the enforced configuration), the pipelineRun will create those many parallel taskRuns. And a user can create multiple such pipelineRuns. This can potentially bring down the entire cluster if many such pipelineRuns are created at the same time.

As a pipeline author, I am looking for a solution where the number of parallel taskRuns does not exceed certain limit and the combinations are run in batches to avoid impacting other pipelines. For example, for a pipelineRun creating 12 parallel taskRuns as part of matrix, I would like to honor the pipeline level enforcement of executing 4 parallel taskRuns. The next 4 are scheduled after the first 4 are completed. And the last 4 are scheduled after first 8 are completed.

As a cluster operator, I am looking for a solution where one pipeline can be executed without impacting other pipelines in a cluster.

default-max-matrix-combinations-count cause the pipelineRun to fail if the number of combinations exceeds the configuration specified in default-max-matrix-combinations-count but does not control maximum parallel jobs running simultaneously.

Potential Feature Request:

    matrix:
      max-parallel: 2
      params:
        - name: platform
          value:
          - linux
          - mac
        - name: browser
          value:
          - safari
          - chrome
  ...

    matrix:
      max-parallel: 4
      params:
        - name: platform
          value: $(params.platform[*])
        - name: browser
          value: $(params.browser[*])
  ...

Reference:

max-parallel in matrix: https://cbrgm.net/post/2022-01-12-github-actions-environments/

/kind feature

[jerop]

+1 to adding this feature as was previously discussed in https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md#concurrency-control

for reference, here are the docs for max-parallel in github actions: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymax-parallel

[jwx0925]

Any update for this issue?