[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-21

[github-actions[bot]]

Summary

• Date of test: 2026-04-21
• Pages visited:
  • /gh-aw/ (Home)
  • /gh-aw/setup/quick-start/
  • /gh-aw/setup/cli/
• Overall impression: The homepage does a solid job explaining what this tool does and why it matters, and the Quick Start guide has a clear structure with numbered steps. However, a few jargon terms and unexplained concepts would trip up a true beginner before they can complete step 2.

⚠️ Note: Playwright browser screenshots were unavailable due to a network isolation constraint in this workflow environment. All analysis was performed via curl and HTML parsing of the live Astro dev server.

---

🔴 Critical Issues Found

1. add-wizard path format is unexplained (Quick Start, Step 2)

gh aw add-wizard githubnext/agentics/daily-repo-status

A beginner has no context for githubnext/agentics/daily-repo-status. What is githubnext/agentics? Is this a GitHub org? A marketplace? A registry? The command just appears without explanation. I'd expect something like:

"This uses the githubnext/agentics workflow registry — a curated collection of pre-built workflows maintained by GitHub Next."

Without that sentence, a beginner either cargo-cults the command or gets stuck wondering if they can swap in their own repo name.

Page: /gh-aw/setup/quick-start/, Step 2
Fix: Add 1–2 sentences explaining what githubnext/agentics is before the command.

---

2. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN — the distinction is easy to miss (Quick Start, Step 2)

The prerequisite says "GitHub Copilot" but most beginners assume that the existing GITHUB_TOKEN that Actions already injects is enough. The fact that you need a separate, manually created personal access token (PAT) with Copilot scope is only mentioned inline with a link to the auth page, buried inside the add-wizard sub-list.

A new user will likely skip this, hit an authentication error, and not know why.

Page: /gh-aw/setup/quick-start/, Prerequisites section
Fix: Add a callout box or ⚠️ note in the Prerequisites section: "GitHub Copilot requires a separate personal access token (COPILOT_GITHUB_TOKEN) — the default GITHUB_TOKEN is not sufficient." with a link.

---

🟡 Confusing Areas

3. "Frontmatter" used without in-line definition (Quick Start, Step 4)

Step 4 says:

"If you have changed the frontmatter (the YAML configuration block between --- markers at the top of the file), regenerate the workflow YAML by running: gh aw compile"

The parenthetical helps, but "frontmatter" is used as if it's a known concept. A true beginner who has no MDX/Hugo/Jekyll experience won't recognize it. The word is hyperlinked to a reference page, but the link is on the word itself — easy to overlook when following step-by-step instructions.

Fix: Expand the inline explanation slightly: "the YAML front matter block (the configuration at the very top of the .md file, between the two --- lines)".

---

4. GitHub Enterprise Server section appears before basic commands (CLI page)

On the CLI Commands page, the section order is:

1. Most Common Commands ✅
2. Installation ✅
3. Pinning to a Specific Version ⚠️ (advanced)
4. Alternative: Standalone Installer ⚠️ (advanced)
5. GitHub Actions Setup Action ⚠️ (advanced)
6. GitHub Enterprise Server Support 🚫 (very advanced — before any command docs!)

A beginner just wants to know how to install, run, and compile. The GHES section (which talks about GITHUB_SERVER_URL, configure_gh_for_ghe.sh, and GHE data residency) appearing this early creates a wall of advanced content between the new user and the command reference they actually need.

Fix: Move all enterprise/GHES content to a collapsible <details> block or a dedicated sub-page. Place the command reference immediately after the installation section.

---

5. The "early development / use at your own risk" disclaimer is prominent but may feel alarming

The homepage has a prominent info box:

"i Note: GitHub Agentic Workflows is in early development and may change significantly... Use it with caution, and at your own risk."

While accurate, this is positioned before users see the examples and success stories. For a new user who just arrived at the site, this could create hesitation before they've even seen what the tool can do.

Fix: Consider moving this disclaimer after the feature highlights or softening it with a link to a stability/roadmap page so users know what "early development" means in practice.

---

6. "Peli's Agent Factory" in the nav is mysterious

The main navigation shows: Quick Start | Create | Examples | Docs | FAQ | Blog | Peli's Agent Factory

"Peli's Agent Factory" as a nav item reads like a personal project, not a documentation section. A new user doesn't know who "Peli" is or what the "Agent Factory" is. Clicking it reveals a blog about curated workflows — which is useful, but the name doesn't communicate that.

Fix: Rename to "Workflow Gallery" or "Community Workflows", or add a short tooltip/subtitle under the nav item.

---

✅ What Worked Well

1. Homepage value proposition is immediately clear — "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions." Excellent single-sentence summary.

2. Two clear CTAs on the hero — "Quick Start with CLI" and "Creating Workflows" give beginners a choice without overwhelming them.

3. 10-minute estimate on Quick Start — Setting expectations upfront builds trust.

4. Video tutorial embedded in Quick Start — A walkthrough video for visual learners is a great addition.

5. Security guardrails explained on the homepage — The five-layer security model with a flow diagram is genuinely impressive and reassuring. Most tools bury security docs; surfacing them on the landing page builds credibility.

6. Most Common Commands table on the CLI page — The table at the top is a perfect quick-reference for returning users.

7. Tip boxes in Quick Start — "Having trouble? Check your repository secrets..." gives beginners an escape hatch before they get frustrated.

8. All internal links functional — Checked ~10 key links; no 404s on the core getting-started path.

---

Recommendations

Quick wins (high impact, low effort):

1. Add 1 sentence explaining githubnext/agentics in Quick Start Step 2. This is the single highest-friction point for a new user.
2. Add a ⚠️ callout in Prerequisites about COPILOT_GITHUB_TOKEN being a separate PAT, not the default GITHUB_TOKEN.
3. Move or collapse the GHES section on the CLI page — hide it in <details> or move it to a dedicated page.

Medium-term improvements:

4. Reorder the homepage disclaimer — move the "early development" note after the feature section so users see value before caveats.
5. Rename "Peli's Agent Factory" in the nav to something self-explanatory like "Workflow Gallery".
6. Add a "What happens when something goes wrong?" section to the Quick Start — beginners will inevitably hit auth errors or failed runs; a brief failure-mode guide would reduce frustration.

Longer-term:

7. Add a "Beginner's Glossary" callout at the top of the Quick Start that defines: frontmatter, lock file, safe-outputs, and engine. These terms appear early and are unfamiliar to non-advanced users.
8. Consider a "5-minute concept video" on the homepage that shows a real workflow being created and run end-to-end.

---

References:

• Workflow run: §24724403967

Generated by Documentation Noob Tester · ● 2.9M · ◷

• expires on Apr 22, 2026, 1:32 PM UTC

[github-actions[bot]]

💥 WHOOSH! ⚡ KRAKOOM! 🦸

SMOKE TEST AGENT WAS HERE!

[Scene: A masked figure in a cape bursts through the wall, clutching a terminal]

"Fear not, citizens of gh-aw! I, the Claude Smoke Test Agent, have swooped in from Run §24725580945 to verify all systems are... NOMINAL!"

POW! BAM! ZAP! ✅✅✅

📋 All core tests passed. The agentic workflows live to fight another day!

[The agent disappears in a cloud of YAML smoke]

🔥 Until next time — same agent channel, same agent time! 🔥

Note

🔒 Integrity filter blocked 1 item

The following item was blocked because it doesn't meet the GitHub integrity level.

• Isolate Codex runtime home from MCP config staging to prevent same-file copy failures #27515 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

💥 [THE END] — Illustrated by Smoke Claude · ● 322.4K · ◷

[github-actions[bot]]

🤖✨ The smoke test agent dropped by and wanted to say hello! All 13 tests passed with flying colors. I built the binary, poked your APIs, took playwright screenshots of GitHub.com, and even found 8 Go symbols with Serena. Your repo is in tip-top shape! 🎊🚀

Note

🔒 Integrity filter blocked 1 item

The following item was blocked because it doesn't meet the GitHub integrity level.

• Isolate Codex runtime home from MCP config staging to prevent same-file copy failures #27515 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

📰 BREAKING: Report filed by Smoke Copilot · ● 838.5K · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #27834.