feat(runner): add ubuntu 24.04 support

[HoKim98]

This PR supports Ubuntu 24.04 runners.

Ubuntu 24.04 is officially released. We need the latest Ubuntu distribution binaries to build our systems. Especially, when using buildah, due to differences in buildah's binary version, there are some features that are difficult to use during the build process (making not to use the distribution binary). like: redhat-actions/buildah-build#116.

The Ubuntu 24.04 image is based on the 22.04 image. I was able to build and utilize the runner image without any additional manipulation. However, more rigorous testing may be needed.

Given that the Ubuntu 24.04 image is based on the 22.04 image, it seems possible to write code to recycle the 22.04 Dockerfile.

[christianherweg0807]

Plz merge :-)

[giokara-work]

+1

[mikkoc]

Who can review and approve this please? @rentziass

[christianherweg0807]

This is stale since 13.Jun... close or merge it plz :-)

Has anyone of the project recognized this? For example @nikola-jokic or @rentziass as active committers of this project?

Thank you for any comment
christian

[kimxogus]

Github hosted runner's ubuntu 24.04 is GA and being migrated into ubuntu-latest, and I think we need to follow this.
https://github.blog/changelog/2024-09-25-actions-new-images-and-ubuntu-latest-changes/

Can anyone review and merge this PR?
@mumoshu @toast-gear @rentziass @nikola-jokic

[SB-MFJ]

+1

[Phil-T1]

Please can we get this merged?

[joerg]

I know that the summerwind runner is the "old" version and community supported, but if that means that the github team is blocking cummunity updates then something is seriously wrong here. @mumoshu @toast-gear @rentziass

[deniscsz]

+1

[tcsullens]

+1 — Is there any plan/expectation to get this done? GitHub has now migrated ubuntu-latest runners to ubuntu-24.04, and it'd be nice to have a ubuntu-24.04 image we can use for self-hosted as well.

[marcispauls]

any plans to merge this?

[ervinpopescu]

@mumoshu
@toast-gear
@nikola-jokic
@rentziass

What is the current blocker for this? We are now more than 9 months since release and ~4 months since GA in hosted runners.

[edulix]

Any update? BTW there's a discussion on this, please anyone interested also upvote it here https://github.com/actions/actions-runner-controller/discussions/3994

[nankigrewal]

any updates? 👀 What are the blockers? @rentziass Do you know?

[jan-navratil-IOC]

Any updates on this one ? It's been stale for a long time

[FearlessHyena]

Can someone from GitHub please respond and get this merged? If possible can we add more reviewers, since it's been almost a year since this PR was submitted and there's been no movement at all

[SB-MFJ]

I know that the summerwind runner is the "old" version and community supported

There is a 'new' version? My understanding was that this chart should technically support any docker image with the Runner binaries loaded to it?
Could I just switch to using an alpine base?

[andreaswachs]

Joining the ping fest as it's been almost a year 😢
@mumoshu
@rentziass
@toast-gear
@nikola-jokic
<3 thank you

[SB-MFJ]

What is the difference between the runner controller and the other 2 helm charts in the repo, the scale-set?
Are they a rewrite of the action-runner-controller?
Just wondering if this is lower priority than getting the other charts up to par and eventually replace the need for this PR?
Wish we got some kind of update here.

[aservedio-swi]

@HoKim98 - I am not sure how the maintainers are queuing in their work, but it appears pinging them from a PR is not working. I do see them triaging and closing/merging new PR's.... So, I am wondering if closing this PR and opening a new one might put it into their queue to review. As we are coming up on a year with no input, it may be worth a try. I am not sure what else we can do at this point.

[valerii-slice]

Also joining the ping fest as it's been almost a year...

@mumoshu
@rentziass
@toast-gear
@nikola-jokic

Thank you

[mumoshu]

@HoKim98 @christianherweg0807 @giokara-work @mikkoc @kimxogus @SB-MFJ @Phil-T1 @joerg @deniscsz @tcsullens @marcispauls @ervinpopescu @edulix @nankigrewal @jan-navratil-IOC @FearlessHyena @andreaswachs @aservedio-swi @valerii-slice Hi everyone commented on this issue!

Honestly speaking, I'm quite surprised there are so many people who have not yet migrated to the new autoscalingrunnerset-based ARC.
Before merging this, can I confirm if ALL of you are still using the legacy version of ARC? To be clear, this image does not work for the newer version of ARC maintained by the GitHub Actions team.

[mumoshu]

By the way, to be clear, we prefer to make the runner images minimal.

See #2050 by @toast-gear and also my comment on #3148 (comment).

The new runner image included in this PR looks almost minimal- awesome!
However, I see a few packages like iproute2, iptables, uidmap, and fuse-overlayfs that are not present in its predecessor, the Ubuntu 22.04 image. Are those required for buildah, by any chance @HoKim98? If so, it would be nicer to remove those for now, so that you can build a custom runner image based on the new minimal 24.04 image.

EDIT: Ok sorry I was comparing different images. All three images contained in this PR look minimal. Great job @HoKim98!

[SB-MFJ]

Honestly speaking, I'm quite surprised there are so many people who have not yet migrated to the new autoscalingrunnerset-based ARC. Before merging this, can I confirm if ALL of you are still using the legacy version of ARC? To be clear, this image does not work for the newer version of ARC maintained by the GitHub Actions team.

Still using them here, though this issue made us realize that there has been a rework in the first place.
The helm notes didn't mention deprecation, so we kept using it.

There doesn't seem to be an equivalent webhook setup in the new scaling set charts.
The docs seem to indicate that we have to create a webhook separately without much guidance as to how to point the scale set listeners to use said webhook. Unless proxying is the intended way of achieving this configuration?
For a little more context, we are running in EKS and have the webhook exposed through an ALB.
We have multiple runnersets, some are scaling from 0, all are listening for workflow_events from github to scale up compute as needed.

As for images, we build ours from Ubuntu base which isn't far from your scale set images with the added dot net binaries. If we add those, I don't imagine the images stop working, and even then reworking them shouldn't be too big a deal.

I'd be more than happy to upgrade to the new way of running our actions if I had examples on how to achieve an equivalent setup.

[valerii-slice]

We made a setup based on ARC official documentation, judging by the versions we are using the latest versions (gha-runner-scale-set-0.11.0 and gha-runner-scale-set-controller-0.11.0). But in the documentation I did not find any mention of how to use or make your own custom image, in this particular case - Ubuntu 24.04.
Now I am completely confused - what does legacy mean? Some links to the new non-legacy ARC lead to the same repo, which contains examples from the old one. So it would be great to understand whether it is possible to build your own image (or I missed it in docs) and not wait another year for this PR to merge, because the ARC project/solution itself is simply excellent! Thanks!

[SB-MFJ]

@valerii-slice Yep, that was confusing to figure out as well.
But yes, there is a 'new' set of helm charts for running ARC's
https://github.com/actions/actions-runner-controller/tree/master/charts

• actions-runner-controller is the 'legacy' version with summerwind's runners which this PR targets.
• gha-runner-scale-set-controller and gha-runner-scale-set are the two new charts that you are using.

And yes, ARC is fantastic.

[aservedio-swi]

Honestly speaking, I'm quite surprised there are so many people who have not yet migrated to the new autoscalingrunnerset-based ARC.
Before merging this, can I confirm if ALL of you are still using the legacy version of ARC? To be clear, this image does not work for the newer version of ARC maintained by the GitHub Actions team.

@mumoshu - since you asked, https://github.com/actions/actions-runner-controller/discussions/2921#discussioncomment-7080396 was one of the reasons we did not migrate. Perhaps there is another way to accomplish the same thing in ARC, but we have two groups of runners (one set is "small" and another "large"). So that the large ones don't sit around unused, we have the same label on the small and large - but an extra label on the large when we really do need that one. Per the discussion, we understand that there is no current interest in supporting labels and - so far as we understood it - our use case and so we did not upgrade. Other than this delay in moving to Ubuntu 24.04, it has been working fine for us and we respect your need for a different architecture to meet other requirements - so not asking for a "change of mind." ARC just does not appear to fit our use case. If there has been an update to the documentation that we missed that shows it can support what we are doing, we could certainly look again.

One other - smaller - part of this: we had followed your initial documentation to add self-hosted for all of our workflows. Meaning, dropping support for that - as also discussed in the link above - will cost us several days of work (perhaps up to a week or three) to clear it out from all of our workflows...

[SB-MFJ]

Oh wow, I completely missed the label part. I'm in the same boat here. Scaling labeled pods with tainted ASG'd nodes have been a really nice cost savings tool. Thanks for pointing that out @aservedio-swi

[ervinpopescu]

+1 to @SB-MFJ @aservedio-swi... I'm not happy with the new charts.

[mumoshu]

Sorry for hijacking the thread with a different topic, everyone- this could be a good chance to move things forward.

Hi @SB-MFJ! First of all, thank you very much for your response!
From the perspective of the maintainer of the legacy ARC and a user of the new ARC, you can migrate to the new one.

we are running in EKS and have the webhook exposed through an ALB.

I used to do the same!

And I didn't love having an ALB and exposing the webhook server to the internet to make runners ephemeral while keeping jobs start quickly.

That's why I seconded the new ARC's design (although I'm not the one who designed it, to be clear). It kept things mostly "scale-to-zero", mostly real-time, without needing LBs and the webhook servers.

We have multiple runnersets, some are scaling from 0, all are listening for workflow_events from github to scale up compute as needed.

The new ARC does not need an ALB or webhook server to receive and process workflow job webhook events. Instead, the "listener" per AutoscalingRunnerSet establishes a connection to GitHub that acts like a tunnel(this is not strictly correct, but you may get it), so that you can receive events from GitHub Actions in near real-time without exposing the webhook server. Runners remain ephemeral as they used to be.

Just so you know, several folks raised concerns about the inability to have multiple runner labels per new RunnerSet, which increases cost(correct me if I'm wrong). Although I understand concerns, I would like to know if we are comparing apples and oranges with some bias toward the legacy ARC.

With the new ARC, you get to have a "listener" pod per RunnerSet, where you had a githubwebhookserver pod per installation of the legacy ARC.

The listeners should be more scalable and maintainable. Previously, you had to create multiple ARC installations to scale beyond a single installation, because the githubwebhookserver was not shard-aware- I assume adding more webhook server replicas did not scale well due to high update contention of HRAs.

As the listener is created per RunnerSet, it's naturally shard-ed by RunnerSets(or labels), enabling a new ARC installation scale beyond dozens or hundreds of RunnerSets architecture-wise. More scalability without having to manage multiple installations anymore.

Scaling labeled pods with tainted ASG'd nodes has been a really nice cost savings tool.

You can still save the same degree of cost with the new ARC.
Could you clarify why the restriction on the number of runner labels per RunnerSet affects cost in your specific case?

Put another way, having two or more labels per runner group might not be shard- or auto-scale-friendly. It might make sense only when you have many non-ephemeral runners. I think the runner label restriction makes sense as the new ARC focuses on ephemeral runners 🤔

To be clear, there seems to be another use case mentioned in https://github.com/actions/actions-runner-controller/discussions/2921 where you want "catch all" labels like "something-latest". You shouldn't do it in the first place, where runners are ephemeral. Specify the label that denotes the size and/or the kind of runner your job exactly wants- runners still scale to zero when there are no jobs(although the listener kept running).

To be clear once again, I'm not the one who designed the new ARC and spoke from the perspective of the maintainer of the legacy ARC. I helped folks move to the latest ARC, believing it makes their lives easier. I'd like to know if anything is missing!!

[SB-MFJ]

I was referring to pod labels that trigger cluster autoscale. Not the runner label as it appears in workflows. Though yea swapping all of those out will be annoying, and we would need at least two types of labels for Dependabot to grab our runners as well.

put another way, there is an exerpt terragrunt config that we are using to give you an idea.

# the github tag to use would be
# self-hosted-<map name> as defined below
# so `self-hosted-ubuntu-latest` would be the tag
inputs = {
  namespace = dependency.namespace.outputs.name

  runner_configs = {
    "ubuntu-latest" = {
      image = "docker.mfj.io/mfj/actions-runner:dind-ubuntu-22.04"
      namespace = dependency.namespace.outputs.name
      maxReplicas = 15
      minReplicas = 5
      scaleDownDelay = 600
      customLabels = ["dependabot"] # in addition to `self-hosted-ubuntu-latest`
      request = {
        "cpu" = "300m"
        "memory" = "2Gi"
      }
      priority = dependency.priorityClass.outputs.name
    }
    "ubuntu-latest-compute-c5-medium" = {
      image = "docker.mfj.io/mfj/actions-runner:dind-ubuntu-22.04"
      namespace = dependency.namespace.outputs.name
      # this should be proportinal to the maximum number of c5.4xlarge instances
      # configured in /aws/k8/k8-workers/gha-b
      maxReplicas = 2
      minReplicas = 0
      scaleDownDelay = 300
      cleanupDelay = "1h"
      scopeTags = {"dedicated" = "gha-medium"} # cluster taint tolerance
      limit = {
        "cpu" = "15"
        "memory" = "28Gi"
      }
      request = {
        "cpu" = "15"
        "memory" = "28Gi"
      }
      priority = dependency.priorityClass.outputs.name
    }
    "ubuntu-latest-compute-c5" = {
      image = "docker.mfj.io/mfj/actions-runner:dind-ubuntu-22.04"
      namespace = dependency.namespace.outputs.name
      # this should be proportinal to the maximum number of c5.12xlarge instances
      # configured in /aws/k8/k8-workers/gha-a
      maxReplicas = 2
      minReplicas = 0
      scaleDownDelay = 300
      cleanupDelay = "24h"
      scopeTags = {"dedicated" = "gha-large"}
      limit = {
        "cpu" = "47"
        "memory" = "89Gi"
      }
      request = {
        "cpu" = "47"
        "memory" = "89Gi"
      }
      priority = dependency.priorityClass.outputs.name
    }

I'll try to set aside some time to do a test run of those charts and see if we can come to roughly the same config.

Thanks for the deep dive!

[matshch]

Is anything prevents merging this PR for the benefit of those who is not yet ready to migrate to the new mode?

[mumoshu]

LGTM. Let me merge this to get the ball rolling!
Thank you to everyone who participated in this pull request, directly and indirectly!

P.S. I've left the newer version of my sentiment on the new vs the legacy ARC discussion https://github.com/orgs/community/discussions/160682#discussioncomment-13626906

[matshch]

Hi! It looks like no Docker images were released for Ubuntu 24.04 yet. Is there anything else that needs to be done to support this?

[neumann-nico]

The action to publish the images does not support Ubuntu 24.04 yet:
https://github.com/actions-runner-controller/releases/actions/runs/16271307773

I created a PR to add support: actions-runner-controller/releases#12

[neumann-nico]

@mumoshu Would you mind to check the PR above to get the images for Ubuntu 24.04 published?
Thanks a lot!

[younsl]

Are there any plans to host an Ubuntu 24.04 actions-runner container image on Docker Hub? Been waiting for this.

[edulix]

Any updates?

[younsl]

@edulix Ubuntu 24.04 images are now available on Docker Hub starting with summerwind/actions-runner v2.329.0. See issue #4249 for details.

[oded-eid-sequentech]

@edulix Ubuntu 24.04 images are now available on Docker Hub starting with summerwind/actions-runner v2.329.0. See issue #4249 for details.

@younsl - I think the images you pointed are from repo 'summerwind'. Are these compatible with ARC?

[younsl]

@oded-eid-sequentech My mistake. ARC's scaleset officially requires using the ghcr.io/actions/actions-runner image. Currently, only the legacy actions-runner-controller supports Ubuntu 24.04.