dynamic matrix

[pritidesai]

Feature request

Be able to author a sharable pipeline with matrix when a task needs to fan out using dynamic combinations.

Use case

As described in one of the comments in #7069 (comment), we have a use case where the combination of params could be dynamic. These dynamic combinations could be specified through pipelineRun or could be coming from a result of a parent task.

Reiterating the requirement in detail here:

---

matrix works great with an implicit and explicit combinations. But matrix does not support dynamic list of combinations specified by the users through the pipelineRun or a trigger.

For example, with a shared pipeline, some users would like to specify 2 combinations while some would like to initiate the same pipeline using 3 combinations.

The following example works great where the pipeline controller generates all possible combinations of both platform and browser. The users can specify a list of platforms and browsers at run time. The same pipeline can be invoked for 2 platforms and also for 3 platforms.

    matrix:
      params:
        - name: platform
          value: $(params.platforms[*])
        - name: browser
          value: $(params.browsers[*])
  ...

We are missing the same level of dynamism with an explicit combinations. Taking an example of matrix to build an image with a specific path for each image would not work if the list of images are specified at the pipelineRun level. The following example is part of the pipeline and the pipeline will always create three instances of the same build task.

    matrix:
        include:
          - name: build-1
            params:
              - name: IMAGE
                value: "image-1"
              - name: DOCKERFILE
                value: "path/to/Dockerfile1"
          - name: build-2
            params:
              - name: IMAGE
                value: "image-2"
              - name: DOCKERFILE
                value: "path/to/Dockerfile2"
          - name: build-3
            params:
              - name: IMAGE
                value: "image-3"
              - name: DOCKERFILE
                value: "path/to/Dockerfile3"
  ...

---

The idea with dynamic matrix is to introduce a new section under matrix along with params and include. This new section can only be used in isolation.

e.g.

apiVersion: tekton.dev/v1
kind: Pipeline
spec:
  tasks:
    - name: clone
      taskRef:
        name: clone
    - name: build
       matrix:
         strategy: $(tasks.setup.results.matrix) # <JSON payload of a list of params and values>

where $(tasks.setup.results.matrix) could be set to:

{"include":  [{"IMAGE": "image", "DOCKERFILE": "path/to/Dockerfile"}]}

or

{"include":  [{"IMAGE": "image-1", "DOCKERFILE": "path/to/Dockerfile1"},
                    {"IMAGE": "image-2", "DOCKERFILE": "path/to/Dockerfile2"},
                    {"IMAGE": "image-3", "DOCKERFILE": "path/to/Dockerfile3"}]}

References:

• https://tomasvotruba.com/blog/2020/11/16/how-to-make-dynamic-matrix-in-github-actions/
• https://stackoverflow.com/questions/65056670/is-it-possible-to-have-a-dynamic-strategy-matrix-in-a-workflow-in-github-actions
• https://docs.github.com/en/actions/learn-github-actions/expressions#example-returning-a-json-object