📚 Documentation Noob Test Report - March 22, 2026

[github-actions[bot]]

Test Summary

• Date: 2026-03-22
• Tester: Automated noob simulation (Copilot coding agent acting as first-time user)
• Pages visited: 12 key pages (homepage, Quick Start, CLI Commands, Creating Workflows, How They Work, About Workflows, Introduction/Architecture, Glossary, Frontmatter, Engines, FAQ, Common Issues, Triage & Analysis examples, Scheduled examples)
• Server: Built and served Astro dev server (npm install && npm run build then astro dev)
• Overall impression: The docs are comprehensive with a clear getting-started path, but a newcomer will hit some terminology walls and a few navigation surprises.

⚠️ Note: Playwright browser automation was not accessible from the Astro server due to network topology (container bridge firewall blocks host→container connections). All analysis was performed via direct file inspection and curl. Screenshots could not be taken.

---

🔴 Critical Issues Found

1. Typo: warp instead of wrap in a code fence — How They Work page

File: docs/src/content/docs/introduction/how-they-work.mdx, line 14

```aw warp    ← should be "wrap"
---
on: ...
```

This breaks word-wrapping in the rendered code block. Not catastrophic but looks sloppy for a page that's the second thing a newcomer reads.

2. /gh-aw/examples/ URL returns 404

The "Examples" link in the top navigation bar points to \$\{base}index.html#gallery (the homepage gallery section), but if a user navigates to /gh-aw/examples/ directly (e.g., from an external link or bookmark), they get a 404. Similarly, /gh-aw/patterns/ returns a 404. There is no index page for either section. Users who find their way to individual example pages cannot easily navigate up to a section overview.

3. Search is unavailable in dev mode

While this only affects developers who run the dev server locally, the search bar shows: "Search is only available in production builds. Try building and previewing the site to test it out locally." This is a significant friction point since the npm run dev documentation lifecycle only mentioned npm run dev, not npm run preview. For anyone following the Quick Start without building first, search won't work.

---

🟡 Confusing Areas

4. "Safe Outputs" term introduced without definition on first use

Page: Homepage (/gh-aw/) — Key Concepts section

"Workflows run with read-only permissions by default. Write operations require explicit approval through sanitized safe outputs..."

The link is there, but a first-time reader scanning the homepage will see "safe outputs" and have no idea what it means. The term is not explained inline. As a noob, I would wonder: "Does my workflow output something? What is 'sanitized' output?"

Recommendation: Add a one-sentence parenthetical like "(pre-approved GitHub write operations such as creating issues or comments)" on first use on the homepage.

5. "Docs" header nav link leads to "About Workflows" — label mismatch

File: docs/src/components/CustomHeader.astro, line 13

<a href="\$\{base}introduction/overview/" class="header-link">Docs</a>

The nav link says Docs but leads to a page titled About Workflows. As a beginner clicking "Docs" expecting a full documentation index, landing on a page called "About Workflows" is disorienting. The sidebar on that page shows the full docs tree, so it works eventually, but the label is misleading.

Recommendation: Either rename the nav link to "Overview" or "Introduction", or change it to point to a true docs landing page.

6. "Examples" nav link anchors to homepage gallery — confusing behavior

The header nav Examples link points to \$\{base}index.html#gallery (the homepage gallery). Clicking it on the homepage scrolls down. Clicking it from any other page navigates away from the sidebar context. Meanwhile, actual workflow examples live at /gh-aw/examples/scheduled/, /gh-aw/examples/issue-pr-events/ etc. but these pages are not linked from the header.

A beginner clicking "Examples" expects a list of example workflows to browse — not to be sent back to the homepage gallery which just shows category headings without concrete examples.

7. Prerequisites don't mention Git knowledge

Page: Quick Start, Prerequisites section

- AI Account (Copilot/Claude/Codex)
- GitHub Repository
- GitHub Actions enabled
- GitHub CLI (gh) v2.0.0+
- Operating System: Linux, macOS, or Windows with WSL

Git knowledge and comfort with a terminal are assumed. A user who has GitHub but has never used the CLI or committed files could get stuck at Step 4 ("Commit and push to your repository") without knowing what git add, git commit -m, and git push are.

Recommendation: Add "Basic Git knowledge (add, commit, push)" to prerequisites, or link to a GitHub tutorial.

8. "Frontmatter" jargon used before being defined

On the Quick Start page Step 4:

"If you have changed the frontmatter, regenerate the workflow YAML..."

First-time users may not know what "frontmatter" means. The page earlier shows a code example with a --- block but never labels it as "frontmatter". The glossary defines it, but there's no inline link at this point in the Quick Start.

Recommendation: Add a link or parenthetical: frontmatter (the configuration block between --- markers).

9. Quick Start Step 2 mentions "lock file" without explanation

"3. Add the workflow — Adds the workflow and lock file to .github/workflows/."

The term "lock file" is new jargon. While it is explained elsewhere, a beginner following Step 2 won't know what a lock file is or why they need it. The add-wizard command creates it automatically, so it's fine, but the label "lock file" is confusing if you're used to npm/yarn lock files (which have a different meaning).

10. Design Patterns section has opaque names (CentralRepoOps, TrialOps, etc.)

In the sidebar there are 15 design patterns, all named "XxxOps". From a beginner's perspective:

• "DailyOps" could mean anything
• "DispatchOps" sounds technical
• "SpecOps" sounds like special operations (military?)

None of these names suggest what the pattern actually does. The individual pages have good descriptions, but navigating the sidebar for the first time is confusing.

Recommendation: Add a "Design Patterns overview" landing page that lists all patterns with one-line descriptions, so users can scan and choose.

11. Quick Start mentions gh aw run without explaining what "workflow run" means

Step 2 mentions triggering an initial run: "5. Optionally trigger an initial run — Starts the workflow immediately." But it doesn't show the command until Step 4. A beginner might wonder how to trigger it.

Also, in Step 4: gh aw run daily-repo-status — no explanation of what happens next (the workflow starts on GitHub Actions, you can watch it via Actions tab, it takes 2-3 minutes per Step 3).

12. "The --push flag" section in CLI docs is oddly placed under "Global Options"

The --push flag is documented under the "Global Options" heading but it only applies to the run command. This is misleading — a user looking at global options might try gh aw compile --push and get an error.

---

🟢 What Worked Well

1. Quick Start flow is logical — 4 steps, clear timeline estimate (10 minutes), focused on a real working example (daily status report)

2. Prerequisite list is thorough — Lists AI accounts, GitHub CLI, Actions requirement. The tip about alternative installation (curl -sL ... | bash) for auth issues is very helpful.

3. The "add-wizard" command is brilliant — Single interactive command handles prerequisites check, engine selection, secret setup, workflow addition, and optional first run. Perfect for beginners.

4. Good use of tips and notes — [!TIP] and [!NOTE] callouts on Quick Start page provide helpful context without cluttering the main flow.

5. Multiple creation paths in "Creating Workflows" — The tabbed examples for Issue Triage, Activity Report, Documentation Updater, and AGENTS.md Maintainer are excellent for copy-pasting prompts. Having GitHub web, VSCode/CLI, and manual editing paths covers all skill levels.

6. Excellent glossary — Terms like "frontmatter", "lock file", "agentic", "safe outputs", "MCP" are all clearly defined. Great reference once users know to look there.

7. FAQ is comprehensive and reassuring — Questions about determinism vs CI/CD, privacy, and costs address exactly the concerns a skeptical engineer would have.

8. Auth page has a magic link — The pre-filled PAT creation link for COPILOT_GITHUB_TOKEN is a small detail that shows care for user experience: https://github.com/settings/personal-access-tokens/new?name=COPILOT_GITHUB_TOKEN&...

9. Estimated time on guides — Quick Start ("10 minutes"), Creating Workflows ("5-15 minutes") — these time estimates set appropriate expectations.

10. Troubleshooting page — The Common Issues page is well organized with specific error messages, causes, and solutions.

---

Recommendations (Prioritized)

High Priority (Quick Wins)

1. Fix warp typo in how-they-work.mdx line 14 — 1 minute fix
2. Add inline definition for "frontmatter" in Quick Start Step 4 — adds clarity for beginners
3. Add inline definition for "safe outputs" on homepage first use — reduces jargon shock
4. Add "lock file" brief explanation in Quick Start Step 2's wizard description

Medium Priority

5. Rename "Docs" nav link to "Introduction" or "Overview" to match the destination page title
6. Fix "Examples" nav link — either point to a real examples index, or label it "Gallery"
7. Add Git prerequisite to Quick Start requirements, with link to GitHub tutorials
8. Add Design Patterns overview page with one-liner descriptions for each XxxOps pattern

Lower Priority (Longer Term)

9. Add npm run preview to local testing docs — so search works during local development
10. Create /gh-aw/examples/ index page to prevent 404 and allow section navigation
11. Create /gh-aw/patterns/ index page for the same reason
12. Move --push flag docs from "Global Options" to run command section

---

Pages Visited

URL	Status	Notes
/gh-aw/	✅ 200	Home/landing page
/gh-aw/setup/quick-start/	✅ 200	Main getting started guide
/gh-aw/setup/creating-workflows/	✅ 200	Multiple creation methods
/gh-aw/setup/cli/	✅ 200	CLI command reference
/gh-aw/introduction/overview/	✅ 200	"About Workflows" page
/gh-aw/introduction/how-they-work/	✅ 200	Architecture/concepts
/gh-aw/introduction/architecture/	✅ 200	Security architecture
/gh-aw/reference/engines/	✅ 200	AI engines reference
/gh-aw/reference/faq/	✅ 200	FAQ
/gh-aw/reference/glossary/	✅ 200	Glossary
/gh-aw/troubleshooting/common-issues/	✅ 200	Common issues
/gh-aw/examples/scheduled/	✅ 200	Scheduled examples
/gh-aw/examples/issue-pr-events/	✅ 200	Event-triggered examples
/gh-aw/examples/	❌ 404	No index page
/gh-aw/patterns/	❌ 404	No index page

---

References:

• §23398559751

AI generated by Documentation Noob Tester · history

• expires on Mar 23, 2026, 8:05 AM UTC

[pelikhan]

/q investigate the connections issues with playwright . Fix prompt as needed

[github-actions[bot]]

🔧 Pay attention, 007! Q is preparing your gadgets for this discussion comment...

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #22395.