feat(policy): add custom_checks for arbitrary subprocess validation in apm-policy.yml

[danielmeppiel]

Why

APM's policy system (apm-policy.yml) enforces governance over dependencies, MCP servers, compilation targets, and manifest fields. But enterprise teams need to enforce rules beyond what APM natively checks — readiness thresholds, quality evaluation, license compliance, custom org validators.

Today there is no extension point. The check list is hardcoded in run_policy_checks(). Adding a custom_checks section lets governance teams plug in any tool via subprocess commands, keeping APM tool-agnostic (like npm never hardcodes ESLint).

What

A new custom_checks field in apm-policy.yml that defines arbitrary subprocess commands. Each runs during apm audit --ci --policy, returns pass/fail via exit code, and integrates into the existing JSON/SARIF audit output.

Policy YAML format

name: contoso-org-policy
version: "1.0.0"
enforcement: block

custom_checks:
  - name: agentrc-readiness
    description: "Repo must meet AI readiness level 3"
    command: "agentrc readiness --json --fail-level 3"
    timeout: 60
    on_failure: warn

  - name: license-check
    description: "All dependencies must have OSS licenses"
    command: "./scripts/check-licenses.sh"
    timeout: 30
    on_failure: block

Execution model

During apm audit --ci --policy org:

1. Baseline checks run (lockfile, integrity, Unicode) — unchanged
2. Policy checks run (allowlist, denylist, required) — unchanged
3. Custom checks run — NEW: for each entry, run command as subprocess with timeout. Exit 0 = pass, non-zero = fail. Results appear alongside built-in checks in JSON/SARIF.

Result format

Uses existing CheckResult model — no new output format:

{
  "name": "custom:agentrc-readiness",
  "passed": false,
  "message": "Repo must meet AI readiness level 3",
  "details": ["exit code 1: achieved level 2, required 3"]
}

How

~150 lines across 4 files:

• schema.py — CustomCheck dataclass + field on ApmPolicy
• parser.py — parse custom_checks from YAML
• policy_checks.py — _run_custom_checks() with subprocess + CheckResult
• Tests

Design principles

• APM stays tool-agnostic — any command works (agentrc, scripts, scanners)
• No library coupling — subprocess execution across language boundaries
• Inherits enforcement — on_failure defaults to top-level enforcement
• Safe — timeout prevents hangs; missing commands handled gracefully

Use cases

Check	Command	Value
AI readiness	agentrc readiness --json --fail-level 3	Gate on maturity
License compliance	./scripts/check-licenses.sh	Legal requirements
Secret scanning	gitleaks detect --no-banner -q	Security hygiene
Doc coverage	test -f README.md	Documentation standards

[github-actions]

Triage decision

needs-design

Proposed labels

theme/governance
area/audit-policy
area/enterprise
area/docs-site
type/feature
status/needs-design

Milestone

null

Suggested next action

Draft a design doc specifying: subprocess execution environment (PATH, env vars, working directory, timeout), token-isolation guarantees, allowed command formats (no shell expansion), CI/CD output format, and integration with the existing policy_engine/ci_checks pipeline.

Suggested issue comment

Extending APM's policy system with custom subprocess validators is a compelling enterprise feature -- it lets teams enforce readiness thresholds, license checks, and org-specific rules without forking APM.

The direction is accepted. Before code lands, a design doc must settle:

1. **Subprocess execution environment**: What PATH, working directory, and environment variables are visible to the command? Critically, APM's auth token env vars must NOT be inherited by custom check subprocesses.
2. **Allowed command formats**: Shell expansion (`$()`, backticks, pipes) must be prohibited. Only a command + argument list is safe.
3. **Timeout and resource limits**: Unbounded subprocess execution blocks the install pipeline. A configurable timeout (default: 30s) and output size cap are required.
4. **CI/CD output format**: When a custom check fails, the diagnostic output must integrate with APM's `DiagnosticCollector` -- not raw subprocess stderr.
5. **Policy file trust boundary**: `apm-policy.yml` is itself a file that APM installs from packages. Custom checks in a policy file from an untrusted package are a post-install code execution vector. The design must address this trust boundary explicitly.

Once those five questions have written answers linked here, the implementation path is clear.

Per-lens notes (collapsed)

DevX UX Expert -- User-Need Reviewer

The user task is clear: enforce org-specific rules in the APM policy gate without modifying APM itself. The pass_if.exit_code and on_fail: block/warn API mirrors APM's existing policy model -- ergonomically consistent. Key gap in the proposal: what does the user see when a custom check fails? The error message must name the check, the exit code, and a configurable message field (not raw subprocess stderr). If the proposal does not include a message field in the schema, add one.

Supply Chain Security Expert -- Risk-Surface Reviewer

HIGH concern: custom_checks with command + args is post-install code execution. Specific risks: (1) APM's auth token env vars must not be inherited by subprocesses -- a malicious apm-policy.yml installed from a package could exfiltrate them. (2) Shell expansion in args values (backticks, $(), pipes) must be blocked -- use subprocess.run(list, shell=False) exclusively. (3) apm-policy.yml is itself installed from packages -- custom checks in a policy file from an untrusted source are a supply-chain attack vector. (4) Arbitrary command values could invoke destructive system tools. The design must address all four. theme/governance and area/audit-policy are required.

OSS Growth Hacker -- Contributor-Tone Reviewer

Not activated -- author is the maintainer filing a governance extensibility feature; no tone-tuning needed.

Python Architect -- Architecture Reviewer

New components required: (1) schema extension in the policy model for custom_checks, (2) a subprocess executor in ci_checks.py or a new custom_check_runner.py, (3) integration with DiagnosticCollector for output capture, (4) timeout handling via subprocess.run(timeout=...). Cross-cutting impact: policy model changes affect compilation, CI gate, and the apm audit command. The subprocess execution environment design is the critical safety boundary. status/needs-design is correct -- the security design must be written before any code.

Doc Writer -- Documentation Reviewer

New policy schema requires docs: policy-reference.md needs a custom_checks section with the schema, security guidance, and examples. area/docs-site secondary label is warranted. The security guidance (do not use shell expansion, token isolation) must appear in the reference docs, not only in code comments.

APM CEO -- Triage Arbiter

Powerful enterprise extensibility feature with a legitimate security concern that must be resolved in design, not in code review. The Supply Chain Security Expert's five-point concern is valid and non-negotiable: token isolation and shell-expansion prohibition are safety invariants. Setting status/needs-design with no milestone is correct. The design doc must explicitly address the policy-file trust boundary (custom checks in an installed policy file = post-install code execution).

{
  "decision": "needs-design",
  "decision_detail": "Requires design doc covering: subprocess env isolation (token vars must not leak), allowed command formats (no shell expansion), timeout/resource limits, CI output format via DiagnosticCollector, and policy-file trust boundary for custom checks installed from packages.",
  "theme": "theme/governance",
  "areas": ["area/audit-policy", "area/enterprise", "area/docs-site"],
  "type": "type/feature",
  "status": "status/needs-design",
  "priority": null,
  "preserved_labels": [],
  "milestone": null,
  "next_action": "Draft a design doc specifying subprocess execution environment isolation, allowed command formats, timeout handling, CI output format, and the policy-file trust boundary for installed custom checks.",
  "comment_markdown": "Extending APM's policy system with custom subprocess validators is a compelling enterprise feature -- it lets teams enforce readiness thresholds, license checks, and org-specific rules without forking APM.\n\nThe direction is accepted. Before code lands, a design doc must settle:\n\n1. **Subprocess execution environment**: What PATH, working directory, and environment variables are visible to the command? Critically, APM's auth token env vars must NOT be inherited by custom check subprocesses.\n2. **Allowed command formats**: Shell expansion (`$()`, backticks, pipes) must be prohibited. Only a command + argument list is safe.\n3. **Timeout and resource limits**: Unbounded subprocess execution blocks the install pipeline. A configurable timeout (default: 30s) and output size cap are required.\n4. **CI/CD output format**: When a custom check fails, the diagnostic output must integrate with APM's `DiagnosticCollector` -- not raw subprocess stderr.\n5. **Policy file trust boundary**: `apm-policy.yml` is itself a file that APM installs from packages. Custom checks in a policy file from an untrusted package are a post-install code execution vector. The design must address this trust boundary explicitly.\n\nOnce those five questions have written answers linked here, the implementation path is clear."
}

---

Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:

• Override any label or milestone above by editing it directly --
  human edits are authoritative and will not be reverted on
  subsequent runs.
• Re-trigger triage by applying the status/needs-triage label, or
  by removing status/triaged to enroll the issue in the next
  daily sweep.

Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.

Generated by Triage Panel · ● 1.9M · ◷