Skip to content

Conversation

@potiuk
Copy link
Member

@potiuk potiuk commented Mar 29, 2024

Depending on selective checks, but also on the job executed, we choose whether to run job on public runners or self-hosted runners. So far the set of labels to select the runners were passed in a bit inconsistent way. Outputs of selective checks can only be strings and the run-as accepts array of strings (labels) - so we were using fromJSON to convert between the two. And we used runs-on inputs on a number of our workflows to pass the selection.

However this meant that runs-on could be either string or array and that sometimes we passed public/self-hosted labels as strings directly and some of those were hard-coded.

This PR changes it consistently across the board to introduce consistent approach:

  • build info have no selective checks results yet, so for them runs-on is hardcoded
  • similarly for "windows" and release jobs that are manually run without running selective checks
  • selective checks will produce three outuputs - JSON stringiified array of labels:
    • default (one that is selected depending on who runs the build)
    • public (for cases where we want to force the builds to use public runners
    • self-hosted (for cases where we want to force the builds to use self-hosted runners
  • all the outputs are named <type>-runs-on-as-string to make it clear they are all strings
  • all inputs of workflows expectings strings are named the same (with as-string suffix and prefix)
  • whenever a job is run, we pass "runs-on" parameter to be fromJSON with appropriate type we want to use passed as input

This will make it easier to reason on which job is using which type of runner and it will make it easier in the future to make it more flexible when we add ASF self-hosted runners and possibly our own K8S runners, or when we would want to change labels for public runners or self-hosted runners.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk added canary When set on PR running from apache repo - behave as canary run default versions only When assigned to PR - only default python version is used for CI tests labels Mar 29, 2024
@potiuk potiuk closed this Mar 29, 2024
@potiuk potiuk reopened this Mar 29, 2024
@potiuk
Copy link
Member Author

potiuk commented Mar 29, 2024

same as #38601 but run as canary run

@potiuk potiuk added the upgrade to newer dependencies If set, upgrade to newer dependencies is forced label Mar 29, 2024
@potiuk potiuk closed this Mar 29, 2024
@potiuk potiuk reopened this Mar 29, 2024
@potiuk
Copy link
Member Author

potiuk commented Mar 29, 2024

Running with "upgrade to newer dependencies" as well. So far all looks good.

Depending on selective checks, but also on the job executed, we
choose whether to run job on public runners or self-hosted runners.
So far the set of labels to select the runners were passed in a bit
inconsistent way. Outputs of selective checks can only be strings
and the `run-as` accepts array of strings (labels) - so we were
using fromJSON to convert between the two. And we used runs-on
inputs on a number of our workflows to pass the selection.

However this meant that runs-on could be either string or array and
that sometimes we passed public/self-hosted labels as strings
directly and some of those were hard-coded.

This PR changes it consistently across the board to introduce
consistent approach:

* build info have no selective checks results yet, so for them
  runs-on is hardcoded
* similarly for "windows" and release jobs that are manually run
  without running selective checks
* selective checks will produce three outuputs - JSON stringiified
  array of labels:
   * default (one that is selected depending on who runs the build)
   * public (for cases where we want to force the builds to use public
     runners
   * self-hosted (for cases where we want to force the builds to use
     self-hosted runners
* all the outputs are named `<type>-runs-on-as-string` to make it clear
  they are all strings
* all inputs of workflows expectings strings are named the same (with
  as-string suffix and <type> prefix)
* whenever a job is run, we pass "runs-on" parameter to be `fromJSON`
  with appropriate type we want to use passed as input

This will make it easier to reason on which job is using which type
of runner and it will make it easier in the future to make it more
flexible when we add ASF self-hosted runners and possibly our own
K8S runners, or when we would want to change labels for public runners
or self-hosted runners.
@potiuk potiuk force-pushed the consistent-approach-for-runs-on-usage branch from b8056fa to cad7587 Compare March 29, 2024 13:22
@potiuk potiuk closed this Mar 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools canary When set on PR running from apache repo - behave as canary run default versions only When assigned to PR - only default python version is used for CI tests upgrade to newer dependencies If set, upgrade to newer dependencies is forced

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant