ci: add agentic documentation preview workflow with Playwright

[Copilot]

Adds an agentic PR preview workflow for documentation that runs for every PR, uses Playwright to browse the rendered docs, and posts a preview comment.

Changes

• New workflow .github/workflows/docs-preview.yml:

  • Triggers on all PRs to main branch
  • Builds Astro docs site and uploads as artifact (7-day retention)
  • Creates GitHub Deployment with transient_environment: true for tracking
  • Uses Playwright to browse the documentation preview and capture a screenshot
  • Uploads screenshot as a separate artifact
  • Posts/updates PR comment with page title, screenshot link, and viewing instructions

• Playwright integration:

  • Serves the built documentation locally
  • Navigates to the documentation homepage
  • Captures the page title and takes a screenshot
  • Screenshot available as downloadable artifact

• Automatic cleanup:

  • pull_request_target with types: [closed] triggers cleanup job
  • Marks all PR-specific deployments as inactive when PR closes/merges

PR Comment Preview

## 📚 Documentation Preview

A preview of the documentation has been built and reviewed using Playwright.

### 📸 Preview Screenshot

The documentation homepage screenshot has been captured and is available as an artifact.

**Page Title:** Agentic Workflow Firewall

> 💡 Download the `docs-preview-screenshot-pr-123` artifact to view the screenshot.

### Resources

| Resource | Link |
|----------|------|
| 📷 **Screenshot** | [Download Screenshot](link) |
| 📦 **Full Preview** | [Download docs-preview-pr-123](link) |
| 🔧 **Workflow Run** | [View Build Logs](link) |

<details>
<summary>How to view the full preview locally</summary>

1. Download the artifact
2. Extract the zip file
3. `npx serve .`
4. Open http://localhost:3000/gh-aw-firewall/ in your browser

</details>

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Medium Priority] Add documentation preview environment for PRs</issue_title>
<issue_description>## Context
From CI/CD Pipeline Gap Assessment (Discussion #227)

Description

Add PR preview environment for documentation changes to allow reviewers to see rendered docs before merge.

Acceptance Criteria

• Set up documentation preview deployment for PRs
• Provide link to preview in PR comments
• Ensure previews are automatically cleaned up after PR close/merge
• Consider using GitHub Pages, Netlify, or Vercel for hosting

Estimated Effort

To be determined (depends on chosen solution)

Related

• Discussion CI/CD Pipeline and Integration Tests - Gap Assessment #227</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes [Medium Priority] Add documentation preview environment for PRs #235

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[github-actions]

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	77.19%	77.19%	➡️ +0.00%
Statements	77.27%	77.27%	➡️ +0.00%
Functions	77.17%	77.17%	➡️ +0.00%
Branches	69.76%	69.76%	➡️ +0.00%

---

Coverage comparison generated by scripts/ci/compare-coverage.ts

[github-actions]

📚 Documentation Preview

A preview of your documentation changes has been built and is available for download.

Resource	Link
📦 Preview Artifact	Download docs-preview-pr-246
🔧 Workflow Run	View Build Logs

How to view the preview

1. Download the artifact from the link above
2. Extract the zip file
3. Open index.html in your browser, or serve locally with:

   npx serve .

---

Preview generated from commit b1cca33

[github-actions]

🚨 Security Review: Critical Vulnerabilities Found

This PR introduces critical security vulnerabilities that could allow attackers to compromise the repository and CI/CD pipeline.

❌ Issue 1: Unsafe use of pull_request_target with code checkout (CRITICAL)

File: .github/workflows/docs-preview.yml
Lines: 10-11, 26, 30-31

Problem:

on:
  pull_request_target:    # Runs with GITHUB_TOKEN write permissions
    types: [closed]

jobs:
  build-and-preview:
    if: github.event_name == 'pull_request' && github.event.action != 'closed'
    steps:
      - name: Checkout PR source code
        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
        # ⚠️ No ref specified - checks out untrusted PR code

Why this is dangerous:

• pull_request_target provides write access to GITHUB_TOKEN (permissions on lines 13-16)
• The workflow checks out code from the PR without specifying a safe ref
• This allows attackers from forks to execute arbitrary code with repository write access

Attack scenario:

1. Attacker creates a fork and malicious PR that modifies docs-site/package.json scripts
2. Workflow runs with pull_request_target (write permissions)
3. Line 43: npm ci installs malicious dependencies
4. Line 48: npm run build executes attacker's code with GITHUB_TOKEN write access
5. Attacker can push code, create releases, modify workflows, steal secrets

❌ Issue 2: Running untrusted code with write permissions

File: .github/workflows/docs-preview.yml
Lines: 40-51

Problem:

- name: Install dependencies
  run: |
    cd docs-site
    npm ci          # Executes install scripts from PR's package.json

- name: Build documentation for preview
  run: |
    cd docs-site
    npm run build   # Executes build scripts from PR's package.json

Why this is dangerous:

• npm lifecycle scripts (install, preinstall, postinstall, build, prebuild, postbuild) run arbitrary code
• These scripts are defined in the PR's package.json which an attacker controls
• Combined with pull_request_target, this gives full write access to the repository

❌ Issue 3: Missing path restrictions on cleanup job

File: .github/workflows/docs-preview.yml
Lines: 10-11 (removed in latest commit but still a concern)

The pull_request_target cleanup job originally had path restrictions, but these were removed. This means the cleanup job will run for any PR closure, not just documentation changes.

---

✅ Recommended Fixes

Option 1: Use pull_request instead (Safest)

Replace pull_request_target with pull_request and remove write permissions:

on:
  pull_request:
    branches: [main]
    paths:
      - 'docs-site/**'
      - 'docs/**'

permissions:
  contents: read      # Read only
  pull-requests: none # Remove write access

Trade-off: Comments and deployments won't work from forks (acceptable for preview workflow)

Option 2: Separate into two workflows (Secure but complex)

1. Build workflow (pull_request trigger, no write access):

   • Checks out PR code safely
   • Runs npm ci and npm run build
   • Uploads artifact

2. Comment workflow (workflow_run trigger, write access):

   • Triggered after build completes
   • Only comments on PR (no code execution)
   • Has write access but doesn't run untrusted code

Option 3: Explicit safe checkout (Minimum fix)

If you must use pull_request_target, explicitly check out base branch for workflow files:

steps:
  # Check out base branch (safe workflow files)
  - name: Checkout base
    uses: actions/checkout@v4
    with:
      ref: ${{ github.event.pull_request.base.sha }}
  
  # Then check out PR code in separate directory
  - name: Checkout PR code
    uses: actions/checkout@v4
    with:
      ref: ${{ github.event.pull_request.head.sha }}
      path: pr-code

However, this still allows arbitrary code execution - Option 1 or 2 is strongly recommended.

---

📚 References

• Keeping your GitHub Actions secure
• Security hardening for GitHub Actions: pull_request_target

---

Verdict: ⛔ DO NOT MERGE until security issues are resolved.

This workflow allows arbitrary code execution with repository write access from untrusted forks.

Ahoy! This treasure was crafted by 🏴‍☠️ Security Guard fer issue #246 🗺️

[Mossaka]

@copilot modify this to an agentic workflow that runs for every PR, and uses playwright tool to browsethe documentation preview and output a comment for the preview.

[Copilot]

@copilot modify this to an agentic workflow that runs for every PR, and uses playwright tool to browsethe documentation preview and output a comment for the preview.

Modified in commit eef90e1. The workflow now:

• Runs for every PR (removed paths filter)
• Uses Playwright to browse and screenshot the documentation homepage
• Uploads the screenshot as an artifact (docs-preview-screenshot-pr-{number})
• Posts a PR comment with the page title and links to download both the screenshot and full preview

[github-actions]

🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨

[github-actions]

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

[github-actions]

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

[github-actions]

Smoke Test Results

Last 2 merged PRs:

• chore: delete existing firewall tests and migrate smoke tests #229: chore: delete existing firewall tests and migrate smoke tests
• feat(ci): add coverage regression detection #244: feat(ci): add coverage regression detection

Tests:

• ✅ GitHub MCP
• ✅ Playwright (title contains "GitHub")
• ✅ File Writing
• ✅ Bash Tool

Status: PASS

AI generated by Smoke Claude

[github-actions]

🧪 Smoke Test Results (Run #21092102920)

Last 2 Merged PRs:

• chore: delete existing firewall tests and migrate smoke tests #229: chore: delete existing firewall tests and migrate smoke tests
• feat(ci): add coverage regression detection #244: feat(ci): add coverage regression detection

Test Results:

• ✅ GitHub MCP: Retrieved PR data successfully
• ❌ Playwright: Browser installation timeout
• ✅ File Writing: Created /tmp/gh-aw/agent/smoke-test-copilot-21092102920.txt
• ✅ Bash Tool: Verified file content

Overall Status: FAIL

cc: @Mossaka (author), @copilot (assignee)

AI generated by Smoke Copilot