Skip to content

Add PR Buildkite detective workflow#246

Merged
strawgate merged 9 commits intomainfrom
main-a60a44c3a9aa858b
Feb 21, 2026
Merged

Add PR Buildkite detective workflow#246
strawgate merged 9 commits intomainfrom
main-a60a44c3a9aa858b

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Feb 21, 2026

Summary

  • Add a new PR Buildkite Detective workflow package under gh-agent-workflows/pr-buildkite-detective/ with a README and trigger example.
  • Add .github/workflows/gh-aw-pr-buildkite-detective.md to configure the Buildkite MCP integration (mcp.buildkite.com) and required BUILDKITE_API_TOKEN secret.
  • Update docs/workflows/gh-agent-workflows.md to list the new workflow and document the additional provider-specific secret requirement.

Behavior

  • The example trigger runs on failed workflow_run events associated with a pull request and calls gh-aw-pr-buildkite-detective.lock.yml@v0.
  • The workflow inspects Buildkite builds/jobs/logs/annotations and posts a read-only diagnostic comment with root cause and remediation guidance.

Validation

  • make compile
  • make lint-workflows
  • make lint

Related to #239


What is this? | From workflow: PR Body Update Agent

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Summary by CodeRabbit

  • New Features

    • Added a PR Buildkite Detective workflow that detects failed PR Buildkite checks, analyzes failures, summarizes root causes, and posts actionable fix recommendations as PR comments.
  • Documentation

    • Added comprehensive docs, README, example workflow, and quick-start instructions covering triggers, inputs, required secrets, install steps, and expected outputs for configuring and running the detective workflow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@coderabbitai
Copy link

coderabbitai bot commented Feb 21, 2026

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Adds a new PR Buildkite Detective reusable GitHub Actions workflow, lock file, example trigger, and documentation that gather Buildkite/MCP failure artifacts, run Copilot analysis, and post diagnostic PR comments with remediation suggestions.

Changes

Cohort / File(s) Summary
Reusable workflow & lock
​.github/workflows/gh-aw-pr-buildkite-detective.md, ​.github/workflows/gh-aw-pr-buildkite-detective.lock.yml
Adds a new copilot-driven reusable workflow (source + generated lock) with inputs, required secrets (COPILOT_GITHUB_TOKEN, BUILDKITE_API_TOKEN), multi-stage activation/agent/conclusion jobs, safe-outputs, MCP/tooling setup, artifact redaction, and detailed reporting steps.
Workflow trigger example
gh-agent-workflows/pr-buildkite-detective/example.yml
New example workflow that conditionally invokes the reusable Buildkite Detective after an upstream CI workflow fails and associated PR(s) exist; forwards inputs and required secrets.
Docs & README
docs/workflows/gh-agent-workflows.md, gh-agent-workflows/pr-buildkite-detective/README.md
Adds documentation entry and a README describing purpose, install, trigger conditions, inputs, required secrets, outputs, and usage examples for the PR Buildkite Detective.
Support files
gh-agent-workflows/pr-buildkite-detective/*
New directory with README and example artifacts to support installation and consumption of the reusable workflow.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Upstream as Upstream CI Workflow
  participant GH as GitHub Actions (caller)
  participant Reusable as PR Buildkite Detective (reusable WF)
  participant Copilot as Copilot Agent
  participant MCP as Buildkite / MCP API
  participant PR as GitHub Pull Request

  Upstream->>GH: workflow_run (failed, includes PR)
  GH->>Reusable: call reusable workflow (inputs, secrets)
  Reusable->>Copilot: start analysis job (setup, prompt, artifacts)
  Copilot->>MCP: fetch pipeline runs & failure artifacts (BUILDKITE_API_TOKEN)
  MCP-->>Copilot: return logs/artifacts
  Copilot->>Copilot: analyze logs, determine root cause & fixes
  Copilot->>PR: post diagnostic comment (COPILOT_GITHUB_TOKEN)
  PR-->>GH: comment posted (results)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Poem

🐰 I hop through failing jobs and sniff each log so bright,
I nudge a patch, tuck hints, then scurry off at night.
A carrot-coded suggestion, a tiny debugging cheer,
“Try this tweak,” I say, and watch the pipeline clear! 🥕

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add PR Buildkite detective workflow' directly and specifically describes the main change: introducing a new workflow for Buildkite-based PR failure diagnosis.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch main-a60a44c3a9aa858b

Comment @coderabbitai help to get the list of available commands and usage tips.

@strawgate strawgate marked this pull request as ready for review February 21, 2026 00:10
Copy link
Contributor Author

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE (non-blocking).


What is this? | From workflow: PR Review Agent

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #246

@github-actions
Copy link
Contributor Author

Fixed and pushed a follow-up commit to address the failed check in Verify compiled workflows.

Root cause: workflow compilation rejected two unauthorized expressions in .github/workflows/gh-aw-pr-buildkite-detective.md:

  • github.event.repository.name
  • github.event.workflow_run.head_branch

Changes applied:

  • Updated those two prompt lines to avoid disallowed expressions while preserving intent in .github/workflows/gh-aw-pr-buildkite-detective.md.
  • Added generated lock file .github/workflows/gh-aw-pr-buildkite-detective.lock.yml so lock validation can pass on the next run.

Validation run locally:

  • make compile ✅ (0 errors)
  • make lint-workflows

Commit pushed to this PR branch:

  • 2ee1df4

What is this? | From workflow: PR Checks Fixer

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@github-actions
Copy link
Contributor Author

The CI failure is caused by disallowed expressions in the new workflow prompt file, not by Buildkite connectivity.

make compile fails with:

  • Validation failed for field 'expressions'
  • unauthorized expressions: github.event.repository.name, github.event.workflow_run.head_branch

Evidence in this PR:

  • .github/workflows/gh-aw-pr-buildkite-detective.md:113 uses $\{\{ github.event.repository.name }}
  • .github/workflows/gh-aw-pr-buildkite-detective.md:116 uses $\{\{ github.event.workflow_run.head_branch }}

Recommended fix (minimal):

- Otherwise call Buildkite MCP `list_pipelines` ... matching `$\{\{ github.event.repository.name }}`
+ Otherwise call Buildkite MCP `list_pipelines` ... matching `$\{\{ github.repository }}`

- If no SHA match is found, try branch `$\{\{ github.event.workflow_run.head_branch }}` ...
+ If no SHA match is found, select the latest failed build for the pipeline.

Optional correctness cleanup in the same file:

- `$\{\{ inputs.buildkite-org }}`
+ `$\{\{ inputs['buildkite-org'] }}`

- `inputs.buildkite-pipeline`
+ `inputs['buildkite-pipeline']`

Commands run:

  • make compile (reproduced the same failure locally)
  • inspected failed Actions job Verify compiled workflows and step Compile workflows (run 22246265890)

What is this? | From workflow: PR Checks Fixer

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@strawgate
Copy link
Collaborator

/ai can you address feedback? also see claude-workflows/build-failure-buildkite/rwx/action.yml i think we can detect the pipeline, etc which should make this a lot simpler

@github-actions
Copy link
Contributor Author

Addressed the feedback and pushed a follow-up commit to main-a60a44c3a9aa858b.

What I changed

  • Updated .github/workflows/gh-aw-pr-buildkite-detective.md to remove disallowed/fragile prompt expressions and simplify build selection logic:
    • pipeline auto-discovery now matches $\{\{ github.repository }} (instead of github.event.repository.name)
    • fallback build selection now uses the latest failed build for the pipeline (no head_branch expression)
  • Recompiled workflows and added the generated lock file:
    • .github/workflows/gh-aw-pr-buildkite-detective.lock.yml

Validation

  • make compile
  • make lint-workflows
  • make lint

What is this? | From workflow: PR Assistant

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #246

@github-actions
Copy link
Contributor Author

Failed check root cause is confirmed: workflow compilation rejects two unauthorized expressions in .github/workflows/gh-aw-pr-buildkite-detective.md.

  • Failing job: Verify compiled workflows (step: Compile workflows)
  • Compiler error: Validation failed for field 'expressions'
  • Unauthorized expressions found:
    • .github/workflows/gh-aw-pr-buildkite-detective.md:113 uses $\{\{ github.event.repository.name }}
    • .github/workflows/gh-aw-pr-buildkite-detective.md:116 uses $\{\{ github.event.workflow_run.head_branch }}

Recommended fix (minimal):

  • Replace $\{\{ github.event.repository.name }} with an allowed expression, e.g. $\{\{ github.repository }}.
  • Remove branch fallback using $\{\{ github.event.workflow_run.head_branch }}; keep SHA-first lookup with $\{\{ github.event.workflow_run.head_sha }} and/or use another allowed expression from the compiler allowlist.

Validation run:

  • make compile (local, on main-a60a44c3a9aa858b) → fails with the same two expression-security violations above.

Follow-up:

  • After updating those two lines, rerun make compile and CI should proceed past workflow compilation.

What is this? | From workflow: PR Checks Fixer

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

strawgate and others added 5 commits February 20, 2026 18:48
- Replace github.event.repository.name with github.repository
  (agent extracts repo name from owner/name format)
- Remove github.event.workflow_run.head_branch fallback; use
  SHA matching with latest-failed-build fallback instead
- Remove workflow-edit-guardrails import (read-only workflow)
- Recompile lock file

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant