📚 Documentation Noob Test Report - February 08, 2026

[github-actions]

Summary

• Date of test: February 08, 2026
• Testing approach: Attempted to build and navigate documentation as a complete beginner
• Overall impression: The documentation has excellent content and structure, but critical build issues would block a new user from even getting started.

---

🔴 Critical Issues Found (Block Getting Started)

1. Documentation Build Fails with 212 Broken Links

Severity: 🔴 BLOCKER

The documentation build process fails during the link validation step, preventing successful local documentation builds for contributors or new users trying to verify their setup.

Error output:

[ERROR] [starlight-links-validator-integration] An unhandled error occurred while running the "astro:build:done" hook
[AstroUserError] Links validation failed.

Impact:

• New contributors cannot build documentation locally
• CI/CD may fail on documentation changes
• Creates uncertainty about whether the setup is correct
• First impression is "something is broken"

Broken link patterns (212 total):

• /gh-aw/reference/tools/ - 15+ occurrences
• /gh-aw/reference/safe-outputs/ - 20+ occurrences
• /gh-aw/reference/workflow-structure/ - 10+ occurrences
• /gh-aw/reference/tokens/ - multiple hash anchors missing
• /gh-aw/reference/network/ - 8+ occurrences
• /gh-aw/examples/*/ - multiple example pages not found
• /gh-aw/troubleshooting/errors/ - referenced but missing

Common broken link types:

1. Reference pages that don't exist or have moved
2. Hash anchors to sections that have been renamed/removed
3. Example workflow pages that may be placeholders
4. Cross-references between related topics

Recommendation:

1. URGENT: Run link validation audit and fix all 212 broken links
2. Add CI check that fails PRs with broken links
3. Consider using relative links where possible
4. Add redirects for moved/renamed pages

---

2. Missing/Incomplete Reference Pages

Severity: 🔴 HIGH

Multiple reference pages are linked throughout the documentation but appear to be missing or incomplete:

Missing pages identified:

• /gh-aw/reference/workflow-structure/ - Referenced 10+ times but broken
• /gh-aw/reference/network/ - Referenced for network permissions
• /gh-aw/reference/permissions/ - Referenced for permissions setup
• /gh-aw/troubleshooting/errors/ - Referenced from common issues page

Impact:

• Users click links expecting to learn more but hit 404s or incomplete content
• Creates impression of incomplete/unfinished documentation
• Breaks the learning flow when trying to understand concepts

Recommendation:

1. Create stub pages for all missing reference topics with "Coming Soon" notices
2. Prioritize completing most-linked reference pages first
3. Update links to point to existing alternatives if available

---

🟡 Confusing Areas (Slow Down Learning)

1. Unclear Prerequisites on Quick Start

Page: /gh-aw/setup/quick-start/

Issue: The prerequisites section lists 5 requirements but doesn't clearly explain:

• What "Write access" means in GitHub terms (Admin? Maintain? Write?)
• How to check if GitHub Actions are enabled (just says "Check in Settings")
• What happens if you don't have a Copilot subscription (can you still use it?)

Specific confusion points:

- ✅ **AI Account** - GitHub Copilot subscription (recommended), or...

As a beginner, I'm asking:

• Is Copilot subscription required or optional?
• What does "recommended" mean? Will it work without it?
• If I have Claude API key, do I need Copilot at all?
• How much does this cost?

Recommendation:

• Clarify what's REQUIRED vs OPTIONAL
• Add a flowchart: "Which AI engine should I use?"
• Link to pricing pages for each option
• Add a "Free tier available?" indicator for each option

---

2. "Frontmatter" Term Not Explained

Issue: The documentation uses "frontmatter" extensively but never defines it for beginners.

First mention: /gh-aw/setup/quick-start/ references "workflow markdown file" but doesn't explain the YAML header concept.

As a beginner:

• I don't know what YAML frontmatter is
• I don't know what goes in the --- section
• I don't know if it's optional or required
• I don't know what happens if I get it wrong

Recommendation:

• Add glossary term with visual example
• Link to glossary on first mention of technical terms
• Include a "Common YAML mistakes" section

---

3. Workflow ID vs File Name Confusion

Issue: Commands use workflow IDs (e.g., gh aw run daily-repo-status) but it's not clear how the ID relates to the filename.

Examples of confusion:

• Is the ID the filename without .md?
• Can I use spaces in IDs?
• What if my filename has special characters?
• Is it case-sensitive?

Recommendation:

• Add explicit callout: "Workflow ID = filename without .md extension"
• Show examples with different naming patterns
• Add validation rules for workflow names

---

4. "Safe Outputs" Not Explained Before Use

Page: Multiple pages, especially /gh-aw/setup/creating-workflows/

Issue: The term "safe outputs" is used frequently but the linked reference page is broken:

• /gh-aw/reference/safe-outputs/ - invalid link

As a beginner:

• What makes an output "safe"?
• Why do I need them?
• What happens without them?
• How are they different from regular outputs?

This is a core security concept that needs early explanation.

Recommendation:

• Fix the broken link immediately
• Add inline explanation on first use
• Create a simple diagram showing safe vs unsafe operations
• Add examples of what's allowed/blocked

---

5. Installation Methods Confusing

Page: /gh-aw/setup/cli/

Issue: Three installation methods are shown but no guidance on which to use:

1. gh extension install github/gh-aw
2. Standalone installer script
3. GitHub Actions setup action

As a beginner:

• Which method should I use?
• What are the pros/cons of each?
• When would I use one vs another?
• Do they all give me the same functionality?

Recommendation:

• Add decision tree: "Choose your installation method"
• Show when to use each method
• Mark "Recommended for most users"
• Group advanced options separately

---

🟢 What Worked Well

Excellent Content Structure

The documentation follows a logical learning path:

1. Introduction → Quick Start → Creating Workflows → Advanced Topics
2. Clear separation between guides, examples, and reference
3. Good use of code examples throughout

Clear Writing Style

The prose is accessible and jargon-free (mostly). Examples:

• "Imagine a world where improvements to your repositories are delivered automatically each morning" - engaging opening
• Step-by-step instructions with numbered lists
• Good use of callouts (TIP, WARNING)

Comprehensive CLI Reference

The CLI commands page has:

• Summary of most common commands upfront
• Detailed documentation for each command
• Multiple installation options for different scenarios

Video Content

The Quick Start page includes an embedded video demonstrating the installation process, which is extremely helpful for visual learners.

Real-World Examples

The Quick Start guide walks through adding an actual workflow (Daily Repo Status Report) rather than a toy example, which helps beginners see real value.

---

📊 Recommendations by Priority

Immediate Actions (This Week)

1. ⚠️ Fix all 212 broken links - This is blocking local builds and creating confusion
2. ⚠️ Create stub pages for missing reference sections with "Coming Soon" notices
3. ⚠️ Add glossary with definitions for: frontmatter, workflow ID, safe outputs, engine, MCP
4. ⚠️ Clarify prerequisites on Quick Start - mark required vs optional clearly

Short Term (This Month)

1. Add "Choosing Your AI Engine" decision flowchart
2. Create troubleshooting section for common first-time setup issues
3. Add validation rules for workflow names/IDs
4. Create visual diagrams for core concepts (safe outputs, workflow compilation)
5. Add CI check that fails on broken links

Long Term (Next Quarter)

1. Create interactive tutorial (like GitHub Skills)
2. Add more video walkthroughs for complex topics
3. Create "recipes" section with copy-paste workflow templates
4. Add community examples showcase
5. Create debugging guide with common error messages explained

---

🎯 Testing Methodology

Limitations: Due to network isolation in the test environment, I was unable to use Playwright to interact with the live documentation site. Instead, I:

1. Attempted to build the documentation using npm run build
2. Analyzed build errors and warnings
3. Examined source markdown files directly
4. Identified broken links from validation output
5. Reviewed content structure and organization

Coverage:

• ✅ Homepage (index.mdx)
• ✅ Quick Start guide
• ✅ CLI Commands reference
• ✅ Build process and link validation
• ❌ Could not test: Interactive navigation, search functionality, mobile responsiveness

---

📝 Additional Notes

Build Process Issues

The build partially succeeds despite link validation failures, allowing the preview server to start. However:

• Build exits with error code 1
• CI would fail on this
• Creates uncertainty for contributors

Preview Server

The documentation uses Astro with Starlight theme and includes:

• Mermaid diagrams support
• Starlight blog integration
• Custom responsive tables enhancement
• Search functionality (Pagefind)
• Multiple video tutorials

---

🙏 Conclusion

The content quality is excellent, but the 212 broken links are a critical blocker for new users and contributors. Fixing the link validation issues should be the top priority.

Once the links are fixed, the next most impactful improvements would be:

1. Adding a glossary for technical terms
2. Clarifying prerequisites and AI engine choices
3. Creating visual diagrams for core concepts

The foundation is solid - we just need to fix the broken pieces and add some scaffolding to help beginners navigate successfully.

---

Labels: documentation, user-experience, automated-testing, priority-high

Related Issues: If there are existing issues tracking these link validation problems, please link them here.

---

Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

AI generated by Documentation Noob Tester

• expires on Feb 15, 2026, 5:39 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #14595

View newer issue

---

This action was performed automatically by the Documentation Noob Tester workflow.