[smoke-detector] 🔍 Smoke Test Investigation - GenAIScript OPENAI_API_KEY Missing

[github-actions]

🔍 Smoke Test Investigation - Run #18722204018

Summary

The GenAIScript smoke test failed due to a missing OPENAI_API_KEY environment variable. The workflow is configured to use the openai:gpt-4.1 model, but the required API key is not provided in the workflow configuration, causing immediate failure during model resolution.

Failure Details

• Run: #18722204018
• Commit: 5d9b9b9 - "Add changeset for reporting instructions feature"
• Trigger: workflow_dispatch
• Duration: 1.8 minutes
• Workflow: Smoke GenAIScript

Root Cause Analysis

Primary Failure: Missing API Key

The GenAIScript engine attempted to resolve the model openai:gpt-4.1 but failed because the OPENAI_API_KEY environment variable is not configured:

Error: OPENAI_API_KEY missing
    at parseTokenFromEnv ((redacted))

This error occurred during the model resolution phase, before any actual AI work could begin.

Secondary Failure: Missing Output File

The create_issue job failed as a cascading failure because it expected an output file from the agent job:

Error: ENOENT: no such file or directory, open '/tmp/gh-aw/safe-outputs/agent_output.json'

Since the agent job failed early in execution, it never created the expected output file.

Tertiary Error: Unhandled Exception in GenAIScript

Additionally, GenAIScript itself threw an unhandled exception when trying to set outputs:

TypeError: Cannot read properties of undefined (reading 'text')
    at githubActionSetOutputs ((redacted))

This indicates that GenAIScript's error handling doesn't gracefully handle failed runs.

Failed Jobs and Errors

Job: agent

• Status: Failed (exit code 255)
• Duration: 1.1 minutes
• Primary Error: OPENAI_API_KEY missing
• Location: env.js:216:19 in genaiscript core

Job: create_issue

• Status: Failed
• Duration: 5 seconds
• Primary Error: ENOENT: no such file or directory
• Root Cause: Dependency on failed agent job

Investigation Findings

Configuration Issue

The GenAIScript smoke test workflow has a model configured in the prompt file:

script({
  model: 'openai:gpt-4.1',
  system: [],
  'system-safety': false,
})

However, the workflow does not provide the necessary OPENAI_API_KEY environment variable that this model requires.

Error Handling Gap

GenAIScript's GitHub Action integration has insufficient error handling - when a run fails, it attempts to set outputs but encounters undefined values, causing a secondary error that masks the primary issue.

Recommended Actions

Immediate Fixes

• Option A: Add OPENAI_API_KEY to GitHub repository secrets and configure the Smoke GenAIScript workflow to use it
• Option B: Change the model configuration to use a properly configured provider (e.g., GitHub Models with github:gpt-4o, or Azure OpenAI if configured)
• Option C: Add conditional execution to skip GenAIScript smoke test if OPENAI_API_KEY is not available

Code Improvements

• Fix GenAIScript's githubActionSetOutputs function to handle failed runs gracefully (add null checks)
• Improve error messages in GenAIScript to suggest configuration steps when API keys are missing
• Add a validation step at workflow start to check for required secrets before running tests

Documentation

• Document required environment variables for each smoke test workflow
• Add troubleshooting guide for common configuration errors
• Update GenAIScript smoke test README with setup instructions

Prevention Strategies

1. Pre-flight Validation: Add a validation step in workflows to verify required environment variables are set before executing agent tasks
2. Better Error Messages: Enhance GenAIScript error handling to provide actionable guidance when configuration is missing
3. Workflow Guards: Use GitHub Actions conditionals to skip tests when required secrets are unavailable
4. Smoke Test Matrix: Consider testing multiple configurations (with/without various API keys) to ensure graceful degradation

Historical Context

This is the first recorded instance of this specific error pattern in the investigation database. The error signature OPENAI_API_KEY_MISSING_GENAISCRIPT has been documented for future pattern matching.

Pattern Severity: High
Is Flaky: No
External Dependency: Yes (OpenAI API)

---

Investigation completed by Smoke Detector 🔍
Pattern stored: /tmp/gh-aw/cache-memory/patterns/openai_api_key_missing.json
Investigation record: /tmp/gh-aw/cache-memory/investigations/2025-10-22-18722204018.json

AI generated by Smoke Detector - Smoke Test Failure Investigator