Skip to content

feat(readiness): Org-level and GitHub settings assessment pillar #128

@digitarald

Description

@digitarald

Problem

AgentRC's readiness assessment currently operates at the repository level — it checks for instruction files, lockfiles, CI configs, and other repo-scoped signals. But during customer engagements, field teams also need to assess organization-level and GitHub-settings-level readiness that determines whether AI tooling can be effective across an org's repos.

These signals are queryable via the GitHub API but aren't captured by any AgentRC pillar today. Without them, a repo can score well on readiness while the org-level configuration actually prevents effective Copilot usage.

Signals to assess

GitHub org settings (via API)

  • Copilot enablement: Is GitHub Copilot enabled for the org? Which policy (all members, selected teams, disabled)?
  • Copilot Chat in IDE: Enabled or restricted?
  • Code referencing: Public code filter on or off?
  • Content exclusions: Are there Copilot content exclusion patterns that might affect the assessed repos?

Repository-level GitHub settings

  • Branch protection rules: Are default branch protections configured? (Required reviews, status checks, signed commits)
  • Code scanning / CodeQL: Enabled? Which languages covered?
  • Dependabot: Enabled for security alerts? Version updates configured?
  • Secret scanning: Enabled with push protection?
  • Actions permissions: Which actions are allowed? Are reusable workflows enforced?

Cross-repo patterns

  • Template repos: Does the org use template repos that include AI instruction files?
  • .github repo: Is there a community health .github repo with default instruction files?
  • Inner source config: Are there org-level .github/copilot-instructions.md or instruction packages?

Proposal

New readiness pillar: org-settings

Add an optional pillar that queries org/repo settings when a GitHub token is available:

agentrc readiness --scope=org --org=contoso

Or assess org-level settings for the current repo's org:

agentrc readiness --include-org

Criteria examples

Criterion ID What it checks Impact
copilot-enabled Copilot is enabled for the org High
branch-protection Default branch has protection rules Medium
code-scanning CodeQL or equivalent is configured Medium
dependabot-security Dependabot security alerts enabled Medium
secret-scanning Secret scanning with push protection Medium
org-instructions Org-level .github repo has instruction files High

Auth requirements

This pillar requires a GitHub token with read:org and admin:org scopes. When the token is missing or insufficient, the pillar should gracefully skip with a clear message rather than failing the entire assessment.

Context

  • Discussed in "AgentRC: Quick Chat on Futures" meeting
  • Distinction made between what's assessable (repo configs, GitHub settings) vs. what's not (cultural practices, team workflows)
  • Barton highlighted that org-level signals are critical for customer assessments
  • This complements the maturity report card concept — org settings feed into the overall maturity tier

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions