Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners #4059

nikola-jokic · 2025-04-25T16:42:29Z

…failed ephemeral runners

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

adamcharnock · 2025-05-02T10:29:22Z

This would be a big help to us. We're having some issues with our Kubevirt runners that we need to resolve, but this would help us out until we can nail down the precise problem.

rr-krupesh-savaliya · 2025-05-13T13:24:21Z

Could we please prioritize this PR? We are currently experiencing the same problem as described in #2721, and it is impacting our workflow.

@nikola-jokic @mumoshu @rentziass @toast-gear

nikola-jokic · 2025-05-13T13:29:15Z

Hey @rr-krupesh-savaliya,

I just want to say that depending on the root cause of the failures happening on your cluster, you might still want to inspect them and fix them. This is just a measure that would allow self-healing, but underlying problems should still be fixed, so you can get all the benefits of autoscaling

Link- · 2025-05-14T12:33:31Z

controllers/actions.github.com/ephemeralrunner_controller_test.go

-				}
-				return updated.Status.Reason, nil
-			}, ephemeralRunnerTimeout, ephemeralRunnerInterval).Should(BeEquivalentTo("TooManyPodFailures"), "Reason should be TooManyPodFailures")
+			for i := range 5 {


Do we have coverage for partial failures? Do we want to add one more test case to cover behaviour for less than < 5 failures?

There is a test that covers it on eviction, which kind of covers the case for re-creation, and we have the case that checks the exit code 0 when the runner exists in the service. I feel like these tests are covering the less than 5 failures.

Link-

We already discussed this at length, left a non blocking comment

jjoseph456 · 2025-05-19T16:08:54Z

At @nikola-jokic @Link- customer mentioned this PR in a recent ticket:

I hope this message finds you well.

We are writing with regard to GitHub Self-Hosted Runners. We have noticed that the bug fix we have been awaiting was recently merged into the main branch.

Could you please let us know the approximate ETA for a release that includes this fix?
In addition, might you have a published release schedule that we can rely on for our own update planning?

Thank you very much for your assistance.

* Updates: runner to v2.318.0 container-hooks to v0.6.1 (actions#3684) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Updates: runner to v2.319.0 (actions#3702) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Updates: runner to v2.319.1 (actions#3708) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Add exponential backoff when generating runner reg tokens (actions#3724) * Updates: runner to v2.320.0 (actions#3763) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Updates: runner to v2.321.0 container-hooks to v0.6.2 (actions#3809) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix ARC e2e tests (actions#3836) * Make EphemeralRunnerController MaxConcurrentReconciles configurable (actions#3832) Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Make EphemeralRunnerReconciler create runner pods earlier (actions#3831) Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Bump github.com/bradleyfalzon/ghinstallation/v2 from 2.8.0 to 2.12.0 (actions#3837) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Update docs with details for the dashboard visualizations (actions#3696) Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Make k8s client rate limiter parameters configurable (actions#3848) Co-authored-by: Taketoshi Fujiwara <t-b-fujiwara@mercari.com> * Bump golang.org/x/crypto from 0.22.0 to 0.31.0 (actions#3844) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Prepare `0.10.0` release (actions#3849) * Fix helm chart bug related to `runnerMaxConcurrentReconciles` (actions#3858) * Prepare `0.10.1` release (actions#3859) * Update dependabot config to group packages (& include actions eco) (actions#3880) * Fix template tests and add go test on gha-validate-chart (actions#3886) * cmd/ghalistener/config: export Validate (actions#3870) Co-authored-by: Han-Wen Nienhuys <hanwenn@gmail.com> * Updated dead link (actions#3830) Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * docs: end markdown code block correctly (actions#3736) * Clarify syntax for `githubConfigSecret` (actions#3812) Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Bump golang.org/x/net from 0.25.0 to 0.33.0 (actions#3881) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Updates: runner to v2.322.0 (actions#3893) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Sanitize labels ending in hyphen, underscore, and dot (actions#3664) * metrics cardinality for ghalistener (actions#3671) Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Rename log from target/actual to build/autoscalingRunnerSet version (actions#3957) * Use Ready from the pod conditions when setting it to the EphemeralRunner (actions#3891) * AutoscalingRunnerSet env: not Rendering correctly (actions#3826) Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Drop verbose flag from runner scale set init-dind-externals copy (actions#3805) * Include custom annotations and labels to all resources created by `gha-runner-scale-set` chart (actions#3934) * Remove old githubrunnerscalesetlistener, remove warning and fix config bug (actions#3937) * Wrap errors in controller helper methods and swap logic in cleanups (actions#3960) * Clean up as much as possible in a single pass for the EphemeralRunner reconciler (actions#3941) * Use gha-runner-scale-set-controller.chart instead of .Chart.Version (actions#3729) Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Trim volume and container helpers in gha-runner-scale-set (actions#3807) Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> * Small readme updates for readability (actions#3860) * Update all dependencies, conforming to the new controller-runtime API (actions#3949) * feat: allow namespace overrides (actions#3797) Signed-off-by: Jesús Fernández <7312236+fernandezcuesta@users.noreply.github.com> Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * chore: Added `OwnerReferences` during resource creation for `EphemeralRunnerSet`, `EphemeralRunner`, and `EphemeralRunnerPod` (actions#3575) * Updates: runner to v2.323.0 (actions#3976) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2 (actions#3984) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add events role permission to leader_election_role (actions#3988) * Create configurable metrics (actions#3975) * Prepare 0.11.0 release (actions#3992) * Fix busy runners metric (actions#4016) * Bump the gomod group across 1 directory with 7 updates (actions#4008) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Include more context to errors raised by github/actions client (actions#4032) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Pin third party actions (actions#3981) * upgrade(golangci-lint): v2.1.2 (actions#4023) Signed-off-by: karamaru-alpha <mrnk3078@gmail.com> * Revised dashboard (actions#4022) * feat(helm): move dind to sidecar (actions#3842) * Fix code block fences (actions#3140) Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com> * Add missing backtick to metrics.serviceMonitor.namespace line to correct formatting (actions#3790) * Bump go version (actions#4075) * Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners (actions#4059) * Updates: runner to v2.324.0 container-hooks to v0.7.0 (actions#4086) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Fix docker lint warnings (actions#4074) * Relax version requirements to allow patch version mismatch (actions#4080) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Refactor resource naming removing unnecessary calculations (actions#4076) * Allow use of client id as an app id (actions#4057) * Updates: runner to v2.325.0 (actions#4109) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Add job_workflow_ref label to listener metrics (actions#4054) Signed-off-by: rskmm0chang <rskmm0chang@hatena.ne.jp> * Bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 (actions#4118) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add startup probe to dind side-car (actions#4117) * Avoid nil point when config.Metrics is nil and expose all metrics if none are configured (actions#4101) Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Add response body to error when fetching access token (actions#4005) Co-authored-by: mluffman <mluffman@thoughtmachine.net> Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Delete config secret when listener pod gets deleted (actions#4033) Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Azure Key Vault integration to resolve secrets (actions#4090) * Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 (actions#4120) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Prepare 0.12.0 release (actions#4122) * Bump build-push-action to 6.18.0 (actions#4123) * Remove cache for build-push-action (actions#4124) * Fix indentation of startupProbe attributes in dind sidecar (actions#4126) * Fix dind sidecar template (actions#4128) * Remove duplicate float64 call (actions#4139) * Remove check if runner exists after exit code 0 (actions#4142) * Explicitly requeue during backoff ephemeral runner (actions#4152) * Prepare 0.12.1 release (actions#4153) * Update CodeQL workflow for v3 (global-run-codeql.yaml) (actions#4157) * Bump the actions group across 1 directory with 5 updates (actions#4160) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(runner): add ubuntu 24.04 support (actions#3598) * Fix image pull secrets list arguments in the chart (actions#4164) * Remove workflow actions version comments since upgrades are done via dependabot (actions#4161) * Updates: runner to v2.326.0 (actions#4176) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Update example GitHub URLs in values.yaml to include an example for enterprise account-level runners (actions#4181) * Add Missing Languages to CodeQL Advanced Configuration (actions#4179) * Updates: runner to v2.327.0 (actions#4185) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Remove deprecated preserveUnknownFields from CRDs (actions#4135) * Updates: runner to v2.327.1 (actions#4188) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Remove JIT config from ephemeral runner status field (actions#4191) * Fix usage of underscore in Runner Scale Set name (actions#3545) Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> * Bump docker/login-action from 3.4.0 to 3.5.0 in the actions group (actions#4196) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump actions/checkout from 4 to 5 in the actions group (actions#4205) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Updates: runner to v2.328.0 (actions#4209) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Requeue if create pod returns already exists error (actions#4201) * docs: fix repo path typo (actions#4229) * Update CODEOWNERS (actions#4251) * Update CODEOWNERS to include new maintainer (actions#4253) * Remove ephemeral runner when exit code != 0 and is patched with the job (actions#4239) * Add workflow name and target labels (actions#4240) * Bump the actions group across 1 directory with 5 updates (actions#4262) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Introduce new kubernetes-novolume mode (actions#4250) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Ensure ephemeral runner is deleted from the service on exit != 0 (actions#4260) * docs: fix broken Grafana dashboard JSON path (actions#4270) * Potential fix for code scanning alert no. 3: Workflow does not contain permissions (actions#4273) Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for code scanning alert no. 1: Workflow does not contain permissions (actions#4274) Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: jiaren-wu <190862939+jiaren-wu@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Bump all dependencies (actions#4266) * Bump the gomod group across 1 directory with 4 updates (actions#4277) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Prepare 0.13.0 release (actions#4280) * Revert "gha: customize client-go rate limiter params (#4)" This reverts commit 8728190. Keep several instrumentations * Revert "gha: make MaxConcurrentReconciles for each reconciler configurable (#1)" This reverts commit 057a1e7. * chore(chart): bump version to 0.13.0-rc.1 for testing --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Jesús Fernández <7312236+fernandezcuesta@users.noreply.github.com> Signed-off-by: karamaru-alpha <mrnk3078@gmail.com> Signed-off-by: rskmm0chang <rskmm0chang@hatena.ne.jp> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bassem Dghaidi <568794+Link-@users.noreply.github.com> Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ken Muse <kenmuse@users.noreply.github.com> Co-authored-by: Taketoshi Fujiwara <t-b-fujiwara@mercari.com> Co-authored-by: Rob Herley <robherley@github.com> Co-authored-by: Nikola Jokic <jokicnikola07@gmail.com> Co-authored-by: Han-Wen Nienhuys <hanwen@engflow.com> Co-authored-by: Han-Wen Nienhuys <hanwenn@gmail.com> Co-authored-by: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Co-authored-by: James Ward <james@notjam.es> Co-authored-by: John Wesley Walker III <81404201+jww3@users.noreply.github.com> Co-authored-by: &es <84567633+and-es@users.noreply.github.com> Co-authored-by: Chris Johnston <christophermichaeljohnston@gmail.com> Co-authored-by: thinkbiggerltd <46003245+thinkbiggerltd@users.noreply.github.com> Co-authored-by: Cees-Jan Kiewiet <ceesjank@gmail.com> Co-authored-by: Mikey Smet <26899585+Mikey032@users.noreply.github.com> Co-authored-by: Patrick Vickery <167798809+pvickery-ParamountCommerce@users.noreply.github.com> Co-authored-by: Salman Chishti <salmanmkc@GitHub.com> Co-authored-by: J. Fernández <7312236+fernandezcuesta@users.noreply.github.com> Co-authored-by: kahirokunn <okinakahiro@gmail.com> Co-authored-by: David Maxwell <davidmaxwell77@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ryosei Karaki <38310693+karamaru-alpha@users.noreply.github.com> Co-authored-by: Borislav Velkov <velkov.borislav@gmail.com> Co-authored-by: Mosè Giordano <765740+giordano@users.noreply.github.com> Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com> Co-authored-by: scodef <52595068+scodef@users.noreply.github.com> Co-authored-by: Ryo Sakamoto <4330349+rskmm0chang@users.noreply.github.com> Co-authored-by: Tingluo Huang <tingluohuang@github.com> Co-authored-by: Nash Luffman <nashluffman@gmail.com> Co-authored-by: mluffman <mluffman@thoughtmachine.net> Co-authored-by: Wim Fournier <github@fournier.nl> Co-authored-by: Jeev B <jeevb@users.noreply.github.com> Co-authored-by: Mark Huijgen <48476507+mhuijgen@users.noreply.github.com> Co-authored-by: calx <108835591+null-calx@users.noreply.github.com> Co-authored-by: adjn <104127038+adjn@users.noreply.github.com> Co-authored-by: Ho Kim <ho.kim@ulagbulag.io> Co-authored-by: Cory Calahan <corycalahan@github.com> Co-authored-by: Kylie Stradley <4666485+KyFaSt@users.noreply.github.com> Co-authored-by: Alex Hatzenbuhler <hatz@hey.com> Co-authored-by: clechevalli <56869366+clechevalli@users.noreply.github.com> Co-authored-by: zkpepe <shim.growers_6u@icloud.com> Co-authored-by: Dennis Stone <densto88@github.com> Co-authored-by: Berat Postalcioglu <bpstlcgl@gmail.com> Co-authored-by: Jiaren Wu <jiaren-wu@github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: jiaren-wu <190862939+jiaren-wu@users.noreply.github.com> Co-authored-by: Junya Okabe <junya@mercari.com>

…failed ephemeral runners (actions#4059)

nikola-jokic added 2 commits April 25, 2025 18:41

Create backoff mechanism for failed runners and allow re-creation of …

3c62dba

…failed ephemeral runners

fix test

850d112

nikola-jokic marked this pull request as ready for review April 28, 2025 10:15

Copilot AI review requested due to automatic review settings April 28, 2025 10:15

nikola-jokic requested review from a team, mumoshu, rentziass and toast-gear as code owners April 28, 2025 10:15

Copilot AI reviewed Apr 28, 2025

View reviewed changes

nikola-jokic added the gha-runner-scale-set Related to the gha-runner-scale-set mode label Apr 28, 2025

nikola-jokic assigned Link- Apr 28, 2025

Link- reviewed May 14, 2025

View reviewed changes

Link- approved these changes May 14, 2025

View reviewed changes

nikola-jokic merged commit cae7efa into master May 14, 2025
17 checks passed

nikola-jokic deleted the nikola-jokic/retry-ephemeral-runner branch May 14, 2025 13:38

kahirokunn mentioned this pull request May 15, 2025

I want to disable "Pod has failed to start more than 5 times" #2721

Closed

mhuijgen mentioned this pull request Jun 2, 2025

Ephemeral Runner failure limit hardcoded to 5 causing issues #4102

Closed

4 tasks

nikola-jokic added this to the gha-runner-scale-set-0.12.0 milestone Jun 4, 2025

rentziass mentioned this pull request Jun 10, 2025

EphemeralRunner stuck in failed state if the job it was allocated to is cancelled #4091

Closed

4 tasks

amd-justchen mentioned this pull request Jun 13, 2025

CI - Waiting for a runner to pick up this job... (unusually high occurences for ARC CPU Builders and GPU baremetal test runners) ROCm/TheRock#787

Closed

nikola-jokic mentioned this pull request Jun 13, 2025

Prepare 0.12.0 release #4122

Merged

kyrylomiro mentioned this pull request Jun 19, 2025

EphemeralRunners are stuck in failed state after the job succeeds #4136

Closed

4 tasks

badstreff mentioned this pull request Oct 12, 2025

retry ephemeral runner even if pod creation fails #4272

Closed

unpollito pushed a commit to DistruApp/actions-runner-controller that referenced this pull request Jan 21, 2026

Create backoff mechanism for failed runners and allow re-creation of …

faed3ce

…failed ephemeral runners (actions#4059)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners #4059

Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners #4059

Uh oh!

nikola-jokic commented Apr 25, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

adamcharnock commented May 2, 2025

Uh oh!

rr-krupesh-savaliya commented May 13, 2025 •

edited

Loading

Uh oh!

nikola-jokic commented May 13, 2025

Uh oh!

Link- May 14, 2025

Uh oh!

nikola-jokic May 14, 2025

Uh oh!

Link- left a comment

Uh oh!

Uh oh!

jjoseph456 commented May 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners #4059

Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners #4059

Uh oh!

Conversation

nikola-jokic commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

adamcharnock commented May 2, 2025

Uh oh!

rr-krupesh-savaliya commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikola-jokic commented May 13, 2025

Uh oh!

Link- May 14, 2025

Choose a reason for hiding this comment

Uh oh!

nikola-jokic May 14, 2025

Choose a reason for hiding this comment

Uh oh!

Link- left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jjoseph456 commented May 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

nikola-jokic commented Apr 25, 2025 •

edited

Loading

rr-krupesh-savaliya commented May 13, 2025 •

edited

Loading