Skip to content

Switch build-info and build images to run on public runners #31451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
May 22, 2023
Merged

Switch build-info and build images to run on public runners #31451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build-images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,10 @@ jobs:
build-info:
timeout-minutes: 10
name: "Build Info"
runs-on: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
runs-on: 'ubuntu-20.04'
env:
TARGET_BRANCH: ${{ github.event.pull_request.base.ref }}
outputs:
runs-on: ${{ github.repository == 'apache/airflow' && 'self-hosted' || 'ubuntu-20.04' }}
python-versions: "${{ steps.selective-checks.python-versions }}"
upgrade-to-newer-dependencies: ${{ steps.selective-checks.outputs.upgrade-to-newer-dependencies }}
all-python-versions-list-as-string: >-
Expand All @@ -66,6 +65,7 @@ jobs:
cache-directive: ${{ steps.selective-checks.outputs.cache-directive }}
default-branch: ${{ steps.selective-checks.outputs.default-branch }}
default-constraints-branch: ${{ steps.selective-checks.outputs.default-constraints-branch }}
runs-on: ${{ steps.selective-checks.outputs.runs-on }}
Copy link
Member Author

@potiuk potiuk May 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also save quite a lot of "self-hosted" time - building images will now be done on public runners for non-commiter runs.

target-commit-sha: "${{steps.discover-pr-merge-commit.outputs.target-commit-sha ||
github.event.pull_request.head.sha ||
github.sha
Expand Down
70 changes: 3 additions & 67 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,64 +65,7 @@ jobs:
# to find the list of authors and replace them, so any changes to the
# formatting of the contains(fromJSON()) structure below will need to be
# reflected in that workflow too.
runs-on: >-
Copy link
Member Author

@potiuk potiuk May 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally we will be able to get rid of this list in yaml (@ashb - ci-infra instruction fix will follow once we merge it).

${{ (
(
github.event_name == 'push' ||
github.event_name == 'schedule' ||
contains(fromJSON('[
"BasPH",
"Fokko",
"KevinYang21",
"Taragolis",
"XD-DENG",
"aijamalnk",
"alexvanboxel",
"aoen",
"artwr",
"ashb",
"bbovenzi",
"bolkedebruin",
"criccomini",
"dimberman",
"dstandish",
"eladkal",
"ephraimbuddy",
"feluelle",
"feng-tao",
"houqp",
"hussein-awala",
"jedcunningham",
"jgao54",
"jghoman",
"jhtimmins",
"jmcarp",
"josh-fell",
"kaxil",
"leahecole",
"malthe",
"mik-laj",
"milton0825",
"mistercrunch",
"msumit",
"o-nikolas",
"pierrejeambrun",
"pingzh",
"potiuk",
"r39132",
"ryanahamilton",
"ryw",
"saguziel",
"sekikn",
"turbaszek",
"uranusjr",
"vikramkoka",
"xinbinhuang",
"yuqian90",
"zhongjiajie"
]'), github.event.pull_request.user.login)
) && github.repository == 'apache/airflow'
) && 'self-hosted' || 'ubuntu-20.04' }}
runs-on: "ubuntu-20.04"
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
outputs:
Expand Down Expand Up @@ -171,11 +114,11 @@ jobs:
skip-pre-commits: ${{ steps.selective-checks.outputs.skip-pre-commits }}
helm-test-packages: ${{ steps.selective-checks.outputs.helm-test-packages }}
debug-resources: ${{ steps.selective-checks.outputs.debug-resources }}
runs-on: ${{ steps.selective-checks.outputs.runs-on }}
source-head-repo: ${{ steps.source-run-info.outputs.source-head-repo }}
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
in-workflow-build: ${{ steps.source-run-info.outputs.in-workflow-build }}
build-job-description: ${{ steps.source-run-info.outputs.build-job-description }}
runs-on: ${{ steps.source-run-info.outputs.runs-on }}
canary-run: ${{ steps.source-run-info.outputs.canary-run }}
run-coverage: ${{ steps.source-run-info.outputs.run-coverage }}
steps:
Expand Down Expand Up @@ -525,7 +468,7 @@ jobs:
wait-for-ci-images:
timeout-minutes: 120
name: "Wait for CI images"
runs-on: "${{needs.build-info.outputs.runs-on}}"
Copy link
Member Author

@potiuk potiuk May 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will save quite a lot of "self-hosted" time for committer builds this way.

runs-on: "ubuntu-20.04"
needs: [build-info, build-ci-images]
if: needs.build-info.outputs.image-build == 'true'
env:
Expand All @@ -548,13 +491,6 @@ jobs:
env:
PYTHON_VERSIONS: ${{ needs.build-info.outputs.python-versions-list-as-string }}
DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
- name: "Tests Pytest collection"
Copy link
Member Author

@potiuk potiuk May 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wil save ~ 4 minutes of "feedback time" as this test-pytest-collection is also repeated in each test job.

run: breeze testing tests --run-in-parallel --collect-only
if: needs.build-info.outputs.run-tests == 'true'
env:
PYTHON_MAJOR_MINOR_VERSION: "${{needs.build-info.outputs.default-python-version}}"
BACKEND: sqlite
PARALLEL_TEST_TYPES: "${{needs.build-info.outputs.parallel-test-types-list-as-string}}"
- name: "Fix ownership"
run: breeze ci fix-ownership
if: always()
Expand Down
13 changes: 13 additions & 0 deletions dev/breeze/src/airflow_breeze/commands/ci_commands.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from airflow_breeze.utils.common_options import (
option_answer,
option_dry_run,
option_github_repository,
option_verbose,
)
from airflow_breeze.utils.confirm import Answer, user_confirm
Expand Down Expand Up @@ -220,6 +221,14 @@ def get_changed_files(commit_ref: str | None) -> tuple[str, ...]:
envvar="GITHUB_EVENT_NAME",
show_default=True,
)
@option_github_repository
@click.option(
"--github-actor",
help="Actor that triggered the event (Github user)",
envvar="GITHUB_ACTOR",
type=str,
default="",
)
@option_verbose
@option_dry_run
def selective_check(
Expand All @@ -228,6 +237,8 @@ def selective_check(
default_branch: str,
default_constraints_branch: str,
github_event_name: str,
github_repository: str,
github_actor: str,
):
from airflow_breeze.utils.selective_checks import SelectiveChecks

Expand All @@ -243,6 +254,8 @@ def selective_check(
default_constraints_branch=default_constraints_branch,
pr_labels=tuple(ast.literal_eval(pr_labels)) if pr_labels else (),
github_event=github_event,
github_repository=github_repository,
github_actor=github_actor,
)
print(str(sc), file=sys.stderr)

Expand Down
9 changes: 8 additions & 1 deletion dev/breeze/src/airflow_breeze/commands/ci_commands_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,16 @@
"--pr-labels",
"--default-branch",
"--default-constraints-branch",
],
},
{
"name": "Github parameters",
"options": [
"--github-event-name",
"--github-repository",
"--github-actor",
],
}
},
],
"breeze ci get-workflow-info": [
{
Expand Down
54 changes: 53 additions & 1 deletion dev/breeze/src/airflow_breeze/global_constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,59 @@ def get_default_platform_machine() -> str:
BREEZE_INIT_COMMAND = ""
DRY_RUN_DOCKER = False
INSTALL_AIRFLOW_VERSION = ""
SQLITE_URL = "sqlite:////root/airflow/airflow.db"


COMMITTERS = [
"BasPH",
"Fokko",
"KevinYang21",
"Taragolis",
"XD-DENG",
"aijamalnk",
"alexvanboxel",
"aoen",
"artwr",
"ashb",
"bbovenzi",
"bolkedebruin",
"criccomini",
"dimberman",
"dstandish",
"eladkal",
"ephraimbuddy",
"feluelle",
"feng-tao",
"houqp",
"hussein-awala",
"jedcunningham",
"jgao54",
"jghoman",
"jhtimmins",
"jmcarp",
"josh-fell",
"kaxil",
"leahecole",
"malthe",
"mik-laj",
"milton0825",
"mistercrunch",
"msumit",
"o-nikolas",
"pierrejeambrun",
"pingzh",
"potiuk",
"r39132",
"ryanahamilton",
"ryw",
"saguziel",
"sekikn",
"turbaszek",
"uranusjr",
"vikramkoka",
"xinbinhuang",
"yuqian90",
"zhongjiajie",
]


def get_airflow_version():
Expand Down
24 changes: 22 additions & 2 deletions dev/breeze/src/airflow_breeze/utils/selective_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,15 @@
from re import match
from typing import Any, Dict, List, TypeVar

from typing_extensions import Literal
if sys.version_info >= (3, 9):
from typing import Literal
else:
from typing_extensions import Literal

from airflow_breeze.global_constants import (
ALL_PYTHON_MAJOR_MINOR_VERSIONS,
APACHE_AIRFLOW_GITHUB_REPOSITORY,
COMMITTERS,
CURRENT_KUBERNETES_VERSIONS,
CURRENT_MSSQL_VERSIONS,
CURRENT_MYSQL_VERSIONS,
Expand All @@ -59,6 +64,8 @@
DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
HELM_VERSION,
KIND_VERSION,
RUNS_ON_PUBLIC_RUNNER,
RUNS_ON_SELF_HOSTED_RUNNER,
GithubEvents,
SelectiveUnitTestTypes,
all_helm_test_packages,
Expand Down Expand Up @@ -295,13 +302,17 @@ def __init__(
commit_ref: str | None = None,
pr_labels: tuple[str, ...] = (),
github_event: GithubEvents = GithubEvents.PULL_REQUEST,
github_repository: str = APACHE_AIRFLOW_GITHUB_REPOSITORY,
github_actor: str = "",
):
self._files = files
self._default_branch = default_branch
self._default_constraints_branch = default_constraints_branch
self._commit_ref = commit_ref
self._pr_labels = pr_labels
self._github_event = github_event
self._github_repository = github_repository
self._github_actor = github_actor

def __important_attributes(self) -> tuple[Any, ...]:
return tuple(getattr(self, f) for f in self.__HASHABLE_FIELDS)
Expand Down Expand Up @@ -618,7 +629,7 @@ def _get_test_types_to_run(self) -> list[str]:
@staticmethod
def _extract_long_provider_tests(current_test_types: set[str]):
"""
In case there are Provider tests in the list of test to run (either in the form of
In case there are Provider tests in the list of test to run - either in the form of
Providers or Providers[...] we subtract them from the test type,
and add them to the list of tests to run individually.

Expand Down Expand Up @@ -765,3 +776,12 @@ def affected_providers_list_as_string(self) -> str | None:
if affected_providers == "ALL_PROVIDERS":
return _ALL_PROVIDERS_LIST
return " ".join(sorted(affected_providers))

@cached_property
def runs_on(self) -> str:
if self._github_repository == APACHE_AIRFLOW_GITHUB_REPOSITORY:
if self._github_event in [GithubEvents.SCHEDULE, GithubEvents.PUSH]:
return RUNS_ON_SELF_HOSTED_RUNNER
if self._github_actor in COMMITTERS:
return RUNS_ON_SELF_HOSTED_RUNNER
return RUNS_ON_PUBLIC_RUNNER
4 changes: 2 additions & 2 deletions images/breeze/output-commands-hash.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ ci:fix-ownership:fee2c9ec9ef19686792002ae054fecdd
ci:free-space:47234aa0a60b0efd84972e6e797379f8
ci:get-workflow-info:01ee34c33ad62fa5dc33e0ac8773223f
ci:resource-check:1d4fe47dff9fc64ac1648ec4beb2d85c
ci:selective-check:3a085894f24cb909812fbc8253a21e13
ci:e51cbc38a202b92b7dc6288f6344c412
ci:selective-check:8a39978ee69d496dae2533d37a48b137
ci:2868dbcdd482663e9d6ccd00055b9cac
ci-image:build:3ffe4dd24ae7090415543e27d8504955
ci-image:pull:c16c6e57c748bfe9b365b4ffafb18472
ci-image:verify:aee88f55e8837028d19316356e29b009
Expand Down
Loading