From 56a9f82aadffe5a2f75512d32e69d6f97197bf65 Mon Sep 17 00:00:00 2001 From: Leynos Date: Thu, 23 Apr 2026 15:03:08 +0000 Subject: [PATCH] docs(ci-guidance): add draft ExecPlan for CI-focused guidance Add a comprehensive draft ExecPlan document detailing the plan to provide clear, cohesive, and tested Continuous Integration (CI) guidance for Netsuke automation users. The document outlines the purpose, constraints, tolerances, risks, progress, discoveries, decision log, skill requirements, documentation references, contextual information, and a staged plan of work for auditing, documentation, behavioural and integration testing, and final validation. It serves as the foundation for integrating user-facing CI guidance examples, design decisions, and test coverage before marking roadmap item 3.13.3 as complete. This commit introduces only the draft documentation in docs/execplans/3-13-3-provide-ci-focused-guidance.md awaiting further approval and implementation. Co-authored-by: devboxerhub[bot] --- .../3-13-3-provide-ci-focused-guidance.md | 462 ++++++++++++++++++ 1 file changed, 462 insertions(+) create mode 100644 docs/execplans/3-13-3-provide-ci-focused-guidance.md diff --git a/docs/execplans/3-13-3-provide-ci-focused-guidance.md b/docs/execplans/3-13-3-provide-ci-focused-guidance.md new file mode 100644 index 00000000..12bc5580 --- /dev/null +++ b/docs/execplans/3-13-3-provide-ci-focused-guidance.md @@ -0,0 +1,462 @@ +# 3.13.3. Provide CI-focused guidance + +This ExecPlan (execution plan) is a living document. The sections +`Constraints`, `Tolerances`, `Risks`, `Progress`, `Surprises & Discoveries`, +`Decision Log`, and `Outcomes & Retrospective` must be kept up to date as work +proceeds. + +Status: DRAFT (awaiting approval) + +## Purpose / big picture + +Roadmap item `3.13.3` asks for CI-focused guidance under the "Friendly" +polish phase. Netsuke already ships the core runtime behaviours that automation +needs: + +- machine-readable diagnostics via `--diag-json` and + `output_format = "json"`, +- layered configuration through OrthoConfig, +- low-noise execution by disabling progress output, and +- verbose timing summaries for troubleshooting. + +What is missing is a cohesive, tested, user-facing guide that explains how to +combine those behaviours in Continuous Integration (CI) pipelines without +guesswork. + +After this work is complete: + +1. `docs/users-guide.md` will include a dedicated CI-and-automation subsection + that explains: + - how to consume Netsuke's JSON diagnostics safely, + - how to configure a quiet baseline for automation using the shipped + OrthoConfig surfaces, and + - how to enable verbose mode for debugging while preserving the documented + JSON-mode guarantees. +2. The CI guidance will explicitly show OrthoConfig layering through CLI + flags, `NETSUKE_...` environment variables, and configuration-file keys, + using the current localized help and orthographic naming conventions. +3. `rstest` coverage will prove the documented examples are accurate for happy + paths, unhappy paths, and edge cases, including JSON parsing and + quiet/verbose combinations. +4. `rstest-bdd` v0.5.0 scenarios will provide executable documentation for the + user-visible CI workflows. +5. `docs/netsuke-design.md` will record any design decision needed to resolve + terminology or behavioural ambiguity. +6. `docs/roadmap.md` will mark `3.13.3` done only after the implementation, + documentation, and full validation gates all pass. + +Observable success means a user can copy the documented CI examples, run them +against the real binary, and see the same stream separation and configuration +effects that the docs describe. + +## Constraints + +- Treat this roadmap item as documentation-backed validation of existing + automation features unless the audit proves a genuine runtime gap. Do not + invent a new CLI/config surface just because the roadmap uses the word + "quiet". +- Reuse the existing OrthoConfig merge root and field vocabulary in + `src/cli/mod.rs` and `src/cli/config.rs`. The plan must document the shipped + knobs (`verbose`, `progress`, `spinner_mode`, `diag_json`, + `output_format`) instead of bypassing them. +- Keep the CLI help/localization path authoritative. Any new user-visible + wording about automation must remain consistent with the localized help + surface described in `docs/ortho-config-users-guide.md` and implemented via + Fluent. +- Add `rstest`-based tests for happy, unhappy, and edge paths. +- Add `rstest-bdd` v0.5.0 behavioural scenarios for the same user-facing + workflows. +- Reuse existing BDD fixtures and steps where practical. Add a new step module + only if the current library cannot express the required assertions cleanly. +- Keep files below the 400-line cap in `AGENTS.md`. If a feature file, step + module, or Rust test file grows too large, split it. +- Use the environment-isolation guidance from `docs/developers-guide.md` and + `docs/reliable-testing-in-rust-via-dependency-injection.md`. Tests must not + mutate process-global environment state unsafely. +- Update `docs/users-guide.md` with any user-visible clarification. +- Record design decisions in `docs/netsuke-design.md`. +- Mark roadmap item `3.13.3` done in `docs/roadmap.md` only after all quality + gates pass. +- Final validation must include logged runs of: + - `make fmt` + - `make check-fmt` + - `make lint` + - `make test` + - `make markdownlint` + - `make nixie` + +## Tolerances (exception triggers) + +- Scope: if implementation grows beyond 12 changed files or roughly 700 net + new lines, stop and reassess before proceeding. +- Behaviour gap: if the audit shows the current binary cannot actually express + the roadmap's intended automation story without a new `--quiet`-style + surface, stop and escalate with options rather than silently expanding + scope. +- Documentation drift: if the current runtime behaviour materially disagrees + with `docs/users-guide.md` and `docs/netsuke-design.md`, stop and decide + which source is authoritative before pinning tests. +- Test brittleness: if BDD or integration assertions depend on terminal-width + wrapping, animated progress rendering, or external tools such as `jq`, stop + and reduce the assertions to stable stream/content checks. +- Environment isolation: if safe CI-style env injection cannot be achieved + through the existing helpers after two focused attempts, stop and document + the blocker. +- `.feature` recompilation: if updated feature files do not rebuild, touch + `tests/bdd_tests.rs` before the final `make test` run. + +## Risks + +- Risk: roadmap wording says "quiet/verbose modes", but the current runtime + does not expose a dedicated `--quiet` flag. Severity: high. Likelihood: + high. Mitigation: start by auditing the shipped behaviour and document + "quiet for automation" in terms of `spinner_mode = "disabled"` and + `--progress false` unless the audit proves otherwise. Record that decision in + `docs/netsuke-design.md`. + +- Risk: documentation examples that use `jq` or shell snippets can drift from + the actual JSON contract. Severity: medium. Likelihood: medium. Mitigation: + docs may show `jq` for readability, but automated tests should parse the + JSON with `serde_json` and assert only the stable documented fields. + +- Risk: behaviour-driven tests that mutate `NETSUKE_...` variables can become + flaky in parallel execution. Severity: medium. Likelihood: medium. + Mitigation: reuse `TestWorld` environment tracking and the helpers described + in `docs/developers-guide.md`. + +- Risk: the guidance could duplicate or contradict the existing advanced usage + chapter. Severity: low. Likelihood: high. Mitigation: place the new material + as a focused CI subsection and cross-link rather than re-explaining every + option in isolation. + +- Risk: JSON mode plus verbose mode is already special-cased by the runtime. + Documentation that explains verbose mode without mentioning that suppression + would mislead users. Severity: medium. Likelihood: high. Mitigation: make + the interaction a first-class documented rule and cover it in both BDD and + `rstest` assertions. + +## Progress + +- [x] 2026-04-23: Reviewed roadmap item `3.13.3`, neighbouring ExecPlans, and + the current user/design/configuration/testing documentation. +- [x] 2026-04-23: Audited the shipped automation-related surfaces in + `docs/users-guide.md`, `docs/netsuke-design.md`, `src/cli/config.rs`, + `src/runner/mod.rs`, `tests/features/json_diagnostics.feature`, and + `tests/advanced_usage_tests.rs`. +- [x] 2026-04-23: Drafted this ExecPlan. +- [ ] Stage A: confirm the exact automation contract and final documentation + placement. +- [ ] Stage B: write the CI-focused user-guide guidance. +- [ ] Stage C: add `rstest-bdd` scenarios for documented CI workflows. +- [ ] Stage D: add `rstest` coverage for JSON consumption and quiet/verbose + automation combinations. +- [ ] Stage E: update design docs, roadmap, and run full validation. + +## Surprises & Discoveries + +- The repository already documents most of the raw ingredients for this task: + `docs/users-guide.md` covers JSON diagnostics, progress suppression, + verbose timing summaries, and OrthoConfig layering, but not as one cohesive + CI story. +- The current runtime does **not** expose a dedicated `--quiet` flag. + Automation-friendly low-noise output currently comes from disabling progress + with `--progress false` or `spinner_mode = "disabled"`. +- `docs/users-guide.md` already includes one CI-oriented JSON example in + Section 12.5, so roadmap `3.13.3` should refine and expand that material + rather than duplicating it blindly. +- Existing tests already cover key automation primitives: + `tests/features/json_diagnostics.feature`, + `tests/features/progress_output.feature`, and + `tests/advanced_usage_tests.rs` can all be extended or reused. +- The current runner behaviour already encodes the crucial automation rule: + verbose timing output is suppressed when JSON diagnostics are active + (`src/runner/mod.rs`). + +## Decision Log + +- Decision: unless Stage A uncovers a real runtime gap, interpret roadmap + "quiet mode" for CI as the existing progress-suppression controls + (`spinner_mode` / `progress`) rather than adding a new `--quiet` flag. + Rationale: this roadmap item is guidance-focused, and the shipped binary + already provides a low-noise automation path. Date/Author: 2026-04-23 / + planning agent. + +- Decision: document CI guidance in `docs/users-guide.md` as a dedicated + subsection inside the advanced-usage chapter rather than scattering small + notes across unrelated sections. Rationale: the user asked specifically for + CI-focused guidance, and Section 12 already houses the automation-adjacent + features. Date/Author: 2026-04-23 / planning agent. + +- Decision: validate the documentation with both `rstest` and `rstest-bdd`. + Rationale: `rstest` is the best fit for programmatically parsing JSON and + asserting precedence combinations, while BDD scenarios keep the documented + CI workflows readable and executable. Date/Author: 2026-04-23 / + planning agent. + +## Skills and references + +This plan should signpost the documents and skills an implementation agent +needs before starting work. + +### Skills + +- `execplans`: requested for this task, but not installed in the current skill + registry. Follow the established ExecPlan conventions in `docs/execplans/` + and keep this document updated as implementation proceeds. +- `skill-installer`: relevant only if a future session needs to install the + missing `execplans` skill before continuing. + +### Primary documentation + +- `docs/roadmap.md`: source of truth for roadmap item `3.13.3`. +- `docs/users-guide.md`: current user-facing documentation for configuration, + progress, verbose timing, and JSON diagnostics. +- `docs/netsuke-design.md`: design-level contract for JSON diagnostics, + progress reporting, and OrthoConfig-backed preferences. +- `docs/ortho-config-users-guide.md`: layered configuration, orthographic + naming, merge helpers, and localized help support. +- `docs/rust-testing-with-rstest-fixtures.md`: parameterized `rstest` + patterns for focused, readable automation tests. +- `docs/rstest-bdd-users-guide.md`: step/scenario authoring rules for + behaviour-driven coverage. +- `docs/reliable-testing-in-rust-via-dependency-injection.md`: safe + dependency-injection and isolation guidance, especially for env-driven tests. +- `docs/rust-doctest-dry-guide.md`: guidance for keeping documentation + examples aligned with executable behaviour. +- `docs/developers-guide.md`: commit gates, BDD harness layout, and safe + environment mutation utilities. + +## Context and orientation + +The following repository areas matter for this task. + +### Runtime/configuration surfaces + +- `src/cli/config.rs` + - Defines the user-facing OrthoConfig-backed preferences: + `verbose`, `progress`, `spinner_mode`, `diag_json`, and `output_format`. + - `CliConfig::resolved_diag_json()` and `resolved_progress()` are the key + helpers the documentation must describe accurately. +- `src/runner/mod.rs` + - Applies the effective runtime contract: + `progress_enabled = cli.resolved_progress() && !cli.resolved_diag_json()` + and `verbose: cli.verbose && !cli.resolved_diag_json()`. + - This is the authoritative source for the quiet/verbose/JSON interaction. +- `src/cli/mod.rs` and `src/cli_l10n.rs` + - Define the CLI surface and localized help mapping that should be cited in + the docs, not reinterpreted manually. + +### Existing documentation to refine + +- `docs/users-guide.md` + - Section 8 already documents progress suppression, timing summaries, and + output-format layering. + - Section 12.4 covers configuration layering. + - Section 12.5 covers JSON diagnostics and already includes one CI example. +- `docs/netsuke-design.md` + - Section 7.3 records the JSON diagnostics contract. + - Section 8.4 records the OrthoConfig-backed CLI/configuration design and + the current progress/verbose semantics. +- `docs/netsuke-cli-design-document.md` + - Historical reference only. It still mentions a conceptual `quiet` field + and should be treated as background, not as the acceptance contract. + +### Existing tests to reuse or extend + +- `tests/features/json_diagnostics.feature` + - Already covers JSON-mode error/success paths and verbose suppression. +- `tests/features/progress_output.feature` + - Already covers progress enable/disable behaviour and verbose timing. +- `tests/advanced_usage_tests.rs` + - Already contains integration coverage for verbose precedence and JSON mode. +- `tests/bdd/steps/json_diagnostics.rs` + - Reusable assertions for JSON validity and stream cleanliness. +- `tests/bdd/steps/manifest_command.rs` + - Reusable workspace setup, command execution, and stdout/stderr assertions. + +## Plan of work + +### Stage A: audit the contract and settle wording + +Before editing user-facing docs, confirm the exact behaviour that automation +guidance will promise. + +1. Audit the current user guide, design doc, and runtime code for: + - JSON diagnostics activation (`--diag-json`, `NETSUKE_DIAG_JSON`, + `output_format = "json"`), + - quiet automation behaviour (`--progress false`, + `spinner_mode = "disabled"`), + - verbose timing behaviour (`--verbose`, `NETSUKE_VERBOSE=true`), and + - the JSON-plus-verbose interaction. +2. Decide where the new guidance belongs. The current expectation is a new + subsection under Section 12, likely "CI and automation", with cross-links + back to Sections 8, 12.4, and 12.5. +3. If the audit confirms that roadmap "quiet" maps to progress suppression + rather than a dedicated flag, capture that design decision in + `docs/netsuke-design.md`. + +Stage A exit criteria: + +- One clear acceptance contract exists for "quiet", "verbose", and JSON mode. +- The planned documentation placement is fixed. +- Any terminology mismatch is resolved in the design document before tests are + written. + +### Stage B: write the CI-focused user guidance + +Add a dedicated CI subsection to `docs/users-guide.md` that turns the existing +behaviour into a coherent workflow guide. + +The new content should include: + +1. A quiet-baseline automation example: + - `NETSUKE_SPINNER_MODE=disabled` or `NETSUKE_PROGRESS=false` + - explanation that this suppresses progress noise while keeping real errors + visible. +2. A verbose troubleshooting example: + - `NETSUKE_VERBOSE=true` + - explanation that successful runs gain timing summaries on `stderr`. +3. A structured-diagnostics example: + - `NETSUKE_OUTPUT_FORMAT=json` or `netsuke --diag-json ...` + - explicit stream contract: artefacts on `stdout`, diagnostics on `stderr`. +4. At least one example of consuming JSON diagnostics in automation. + - The docs may use `jq` for readability. + - The example should only depend on stable fields already documented: + `schema_version`, `generator`, `diagnostics[*].code`, + `diagnostics[*].message`. +5. A note on combining modes: + - quiet + verbose => no progress noise, timing summary on success, + - JSON + verbose => JSON contract wins; verbose timing/tracing is + suppressed. +6. A short configuration-file example showing how OrthoConfig makes these + preferences persistent in CI-oriented `.netsuke.toml` files. + +Stage B exit criteria: + +- The users guide contains a single, cohesive CI subsection. +- The examples align with the current runtime contract and OrthoConfig naming. +- The wording uses en-GB-oxendict spelling and matches the docs style guide. + +### Stage C: add behavioural coverage with `rstest-bdd` + +Add a focused behavioural feature that acts as executable documentation for the +CI workflows. Prefer a new `tests/features/ci_guidance.feature` file unless +extending an existing file is clearly simpler and keeps line counts sane. + +Planned scenarios: + +1. Environment-driven JSON diagnostics in CI: + - set `NETSUKE_OUTPUT_FORMAT=json`, + - run a failing command, + - assert empty `stdout` and valid diagnostics JSON on `stderr`. +2. Quiet automation mode suppresses progress noise: + - set `NETSUKE_SPINNER_MODE=disabled` or equivalent, + - run a successful build with fake Ninja progress available, + - assert progress markers are absent. +3. Verbose automation mode emits timing summaries: + - set `NETSUKE_VERBOSE=true`, + - run a successful command, + - assert timing summary fragments appear on `stderr`. +4. JSON mode suppresses verbose output even when verbose is configured: + - combine JSON mode with verbose, + - assert `stderr` is valid JSON (or empty on success), with no timing noise. + +Prefer reusing existing steps from: + +- `tests/bdd/steps/manifest_command.rs` +- `tests/bdd/steps/json_diagnostics.rs` +- `tests/bdd/steps/progress_output.rs` + +Only add a small new step module if the current library lacks: + +- env-var setup steps with scenario-safe cleanup, or +- a concise assertion for "stderr omits progress/timing fragments". + +Stage C exit criteria: + +- Behavioural scenarios cover happy, unhappy, and edge paths. +- The feature text reads like user-facing CI documentation. +- No scenario relies on external tools such as `jq`. + +### Stage D: add focused `rstest` coverage + +Add or extend `rstest` integration coverage for the lower-level automation +claims that are awkward in BDD. + +Recommended home: a new `tests/ci_guidance_tests.rs` module, unless the +existing `tests/advanced_usage_tests.rs` remains comfortably under the file +limit and the new cases fit naturally there. + +Tests should cover: + +1. JSON diagnostics example validation: + - run Netsuke in JSON mode, + - parse `stderr` with `serde_json`, + - assert the fields used in the docs example exist and have sensible values. +2. OrthoConfig layering for automation: + - config file enables quiet baseline, + - environment variable overrides config, + - CLI flag overrides environment variable. +3. Quiet plus verbose interaction: + - progress remains suppressed, + - timing summary still appears on successful human-readable runs. +4. Invalid automation configuration: + - malformed `spinner_mode` or `output_format`, + - assert failure is actionable and matches the documented shape. + +Use `rstest` parameterized cases to keep the precedence permutations readable. +Use the existing fake-Ninja helpers and env-forwarding helpers rather than +mutating process-global state directly. + +Stage D exit criteria: + +- `rstest` tests cover the documentation-backed JSON fields and precedence + claims. +- Happy, unhappy, and edge paths are present. +- The tests remain deterministic and do not rely on shell parsing tools. + +### Stage E: design doc, roadmap, and validation + +Finish the work by aligning the design record, updating the roadmap, and +running the full gate suite. + +1. Update `docs/netsuke-design.md` with the final design decision for CI + guidance, especially if "quiet" is formally documented as + progress suppression rather than a dedicated flag. +2. Update `docs/roadmap.md` to mark `3.13.3` done only after all tests and + documentation changes land successfully. +3. If any `.feature` file changed, `touch tests/bdd_tests.rs` before the final + `make test`. +4. Run all required gates with `tee` and `set -o pipefail`: + - `make fmt` + - `make check-fmt` + - `make lint` + - `make test` + - `make markdownlint` + - `make nixie` + +Stage E exit criteria: + +- User guide, design doc, and roadmap all reflect the shipped behaviour. +- All required validation gates pass. +- The roadmap entry is updated only after success. + +## Validation and acceptance + +The feature is complete when all of the following are true: + +1. `docs/users-guide.md` includes a dedicated CI-focused subsection with: + - JSON-diagnostics consumption examples, + - quiet/verbose automation guidance, + - OrthoConfig layering examples for CLI, env, and config-file use. +2. `docs/netsuke-design.md` records the final terminology/behaviour decision. +3. `rstest-bdd` scenarios prove the documented CI workflows end to end. +4. `rstest` coverage proves the JSON fields and precedence combinations used in + the docs. +5. `make fmt`, `make check-fmt`, `make lint`, `make test`, + `make markdownlint`, and `make nixie` all pass. +6. `docs/roadmap.md` marks `3.13.3` done. + +## Outcomes & Retrospective + +Pending implementation approval.