[aw-failures] Fix P0: Gemini API key (GOOGLE_API_KEY) is invalid — all Smoke Gemini runs failing since 2026-04-29

[github-actions]

Problem Statement

The GOOGLE_API_KEY secret used by the Smoke Gemini workflow is invalid or expired. All Smoke Gemini runs since at least 2026-04-29 00:21 UTC are failing immediately on first LLM call with:

HTTP 400 INVALID_ARGUMENT
reason: API_KEY_INVALID
service: generativelanguage.googleapis.com
message: "API key not valid. Please pass a valid API key."

The agent makes zero LLM calls, zero tool calls, and produces no output before terminating.

Affected Workflows and Runs

Run	Branch	Time (UTC)	Status
§25084493525	copilot/align-docs-with-canonical-apm	2026-04-29 00:21	failure — API_KEY_INVALID
§25085302973	copilot/align-docs-with-canonical-apm	2026-04-29 00:49	failure — API_KEY_INVALID

No successful Smoke Gemini run found in the last 24h. Prior successful runs date to 2026-04-14 or earlier.

Root Cause

The GOOGLE_API_KEY repository secret is invalid. Evidence from Gemini CLI error dump files (gemini-client-error-*.json) on run 25085302973 confirms the identical error at two call sites:

• Models.generateContent → NumericalClassifierStrategy.route
• Models.generateContentStream → Turn.run

The key is either expired, deleted, or replaced with an incorrect value.

Proposed Remediation

1. Go to Settings → Secrets and variables → Actions for github/gh-aw
2. Locate GOOGLE_API_KEY
3. Rotate or replace it with a valid Gemini API key (project must have generativelanguage.googleapis.com enabled)
4. Re-run a Smoke Gemini job to verify recovery

Success Criteria

• Smoke Gemini run completes with conclusion: success
• Firewall logs show generativelanguage.googleapis.com:443 receiving successful responses (no blocked/error domains)
• Agent produces at least 1 LLM turn and a noop or valid safe output

References

• Parent group issue: [aw-failures] [aw] Failure Investigator (6h) - Issue Group #28948
• Existing failure issue: [aw] Smoke Gemini failed #29014 (Smoke Gemini failed — contains raw error traceback)
• Audit analysis: Investigator run §25086158628

References:

• §25084493525
• §25085302973
• §25086158628

Note

🔒 Integrity filter blocked 6 items

The following items were blocked because they don't meet the GitHub integrity level.

• #25944 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #23655 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Add <img> to safe-outputs HTML tag allowlist #29009 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Flaky awf-api-proxy unhealthy failure in downstream Copilot workflow #28898 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• MCP gateway fails on ARC self-hosted runners with dind sidecar — "Invalid container ID format" + "Docker socket not found" #28888 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Support self-hosted runners whose service account/home is not /home/runner #27260 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by [aw] Failure Investigator (6h) · ● 668.8K · ◷

• expires on May 6, 2026, 1:35 AM UTC

[github-actions]

🍪 *Issue Monster selected this for Copilotecho BEGIN___COMMAND_OUTPUT_MARKER ; PS1= ; PS2= ; unset HISTFILE ; EC=0 ; echo ___BEGIN___COMMAND_DONE_MARKER___0 ; }

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

Caution

Security scanning requires review for Issue Monster

Details

Potential security threats were detected in the agent output. The workflow output should be reviewed before merging.

Review the workflow run logs for details.

🍪 Om nom nom by Issue Monster · ● 21K · ◷

[lpcox]

🔗 AWF tracking issue: https://github.com/github/gh-aw-firewall/issues/github/gh-aw-firewall#2288

Generated by Firewall Issue Dispatcher · ● 986.3K · ◷

[lpcox]

🔗 AWF tracking issue: https://github.com/github/gh-aw-firewall/issues/github/gh-aw-firewall#2296

Generated by Firewall Issue Dispatcher · ● 436.3K · ◷

[lpcox]

This is an operational credential issue — the GOOGLE_API_KEY repository secret needs to be rotated/replaced with a valid key.

Code-side mitigations already in place in gh-aw-firewall:

• PR #2200 — startup API key validation
• PR #2182 — Gemini auth param stripping

Action needed: Rotate the GOOGLE_API_KEY secret in github/gh-aw repo settings.