[CI][Crossbow] Use apache/arrow instead of separated repository (e.g. ursacomputing/crossbow)

[kou]

Describe the enhancement requested

We didn't have enough GitHub Actions capacity when we develop Crossbow. So using separated repository (and separated CI services such as Azure Pipelines and Travis CI) was reasonable.

But we have enough GitHub Actions capacity recently. Can we reconsider the Crossbow approach?

We may be able to use on.workflow_dispatch and gh wokrflow run --ref as an alternative of the current Crossbow approach.

Some inconvenient points for the current Crossbow approach:

• CI results use a GitHub comment (example) instead of GitHub status checks.
  • It may cause a long history.
• No direct link to artifacts.
  • Crossbow creates a new tag for each job and uses GitHub Release associated with the tag for artifacts. There is no direct link for the associated GitHub Release.
• We can't use Dependabot for dev/tasks/*/*.yml
• Difficult to automate job submission.
  • For example, we want to submit binary artifacts build jobs after source archive build is completed automatically in release process.
  • FYI: We already have secrets.GITHUB_CROSSBOW_TOKEN in apache/arrow GitHub Actions. So we will be able to implement it with the current Crossbow approach.
• ...

Component(s)

Continuous Integration, Developer Tools

[amoeba]

In case it helps, Datafusion just implemented something like this, see https://github.com/apache/datafusion/blob/main/.github/workflows/pr_comment_commands.yml for how they did it.

[kou]

Wow! I didn't know it!

[kszucs]

I have been thinking of the same recently, we could take advantage of additional GHA features though I have some questions:

• What if the apache/arrow GHA capacity decreases? I'm not sure what is our current concurrency limit.
• Are there crossbow jobs running on other CI systems which couldn't be ported?
• What if we need to support non-github hosted architectures? We would solve it using self-hosted runners?

In an ideal setup we could easily create a release by:

1. pressing a button, triggering jobs, uploading artifacts using https://github.com/actions/upload-artifact?
2. notifying the release manager or dev@ that the artifacts are available
3. release manager downloads, signs and uploads the artifacts under the release tag?
4. trigger the post-release tasks (wherever additional credentials are not required or machine tokens could be used)

[kou]

What if the apache/arrow GHA capacity decreases? I'm not sure what is our current concurrency limit.

https://infra.apache.org/github-actions-policy.html has the following:

• The average number of minutes a project uses per calendar week MUST NOT exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 hours).
• The average number of minutes a project uses in any consecutive five-day period MUST NOT exceed the equivalent of 30 full-time runners (216,000 minutes, or 3,600 hours).

2025-04-01/2025-04-08 statistics: https://github.com/apache/arrow/actions/metrics/usage?dateRangeType=DATE_RANGE_TYPE_CUSTOM&range=1743465600000-1744070400000

Total minutes: 77,900

Are there crossbow jobs running on other CI systems which couldn't be ported?

Currently, we only use GitHub Actions and Azure Pipelines. So we'll be able to port all jobs to GitHub Actions.

What if we need to support non-github hosted architectures? We would solve it using self-hosted runners?

We'll try self-hosted runners or we mark these architectures as tier 2 or tier 3 platforms. See also: #46002

In an ideal setup we could easily create a release by:

1. pressing a button, triggering jobs, uploading artifacts using https://github.com/actions/upload-artifact?
2. notifying the release manager or dev@ that the artifacts are available
3. release manager downloads, signs and uploads the artifacts under the release tag?
4. trigger the post-release tasks (wherever additional credentials are not required or machine tokens could be used)

I hope that git tag && git push can trigger all of them (build source archive, sign it, upload it, build binary artifacts from it, sign them, upload them, verify them, start a vote, ...) with Automated Release Signing. :-)

[assignUser]

Yes, I have discussed this last year with @raulcd too and while it will be a bunch of work to port everything over the benefits outweigh that effort.

There is an infra page that shows the metrics in a format matching the policy.

We could potentially even port the benchmark jobs but that comes with additional considerations in regards to noisiness of the results.

I hope that git tag && git push can trigger all of them (build source archive, sign it, upload it, build binary artifacts from it, sign them, upload them, verify them, start a vote, ...) with Automated Release Signing. :-)

I don't mind triggering it via the actions tab but either way automatic signing will be great!

[assignUser]

In case it helps, Datafusion just implemented something like this, see https://github.com/apache/datafusion/blob/main/.github/workflows/pr_comment_commands.yml for how they did it.

As a caveat: runs triggered with 'workflow_dispatch' have full permissions and secret access by default, so the jobs have to be carefully checked for minimal permissions, no secret use and they can never be run using the pr version of the workflow.

I think I'd prefer that we don't trigger these via comment but rather have the reviewer trigger them via a manual workflow dispatch in the actions tab.

[assignUser]

Hm, I think we could solve this within pull_request as a trigger by gating 'extended' jobs on the commit title. So you could just push an empty commit with [run extended] or something to kick them off. That way all the potential security issues go away.

[kou]

How about using CI: Extra label to run extra CI jobs?
#46501 is an implementation.

[assignUser]

Yes, that seems a bit nicer and still doesn't require a new privileged job! Great!