diff --git a/Makefile b/Makefile index bdc7ea1b..704a9daa 100644 --- a/Makefile +++ b/Makefile @@ -12,7 +12,7 @@ release: target/release/$(APP) ## Build release binary all: release ## Default target builds release binary -clean: ## Remove build artifacts +clean: ## Remove build artefacts $(CARGO) clean test: ## Run tests with warnings treated as errors diff --git a/docs/behavioural-testing-in-rust-with-cucumber.md b/docs/behavioural-testing-in-rust-with-cucumber.md index c49f15bb..85965f8b 100644 --- a/docs/behavioural-testing-in-rust-with-cucumber.md +++ b/docs/behavioural-testing-in-rust-with-cucumber.md @@ -979,7 +979,7 @@ The process involves two main steps: 2. **Publish reports:** Many CI platforms can parse and display test results in a structured format. The `cucumber` crate supports generating JUnit XML reports via the `output-junit` feature flag.[^16] These XML files can then - be published as test artifacts for platforms like GitHub Actions, GitLab + be published as test artefacts for platforms like GitHub Actions, GitLab CI,[^34] or Jenkins to consume.[^33] This CI integration closes the BDD loop. The `.feature` files, once checked diff --git a/docs/netsuke-design.md b/docs/netsuke-design.md index 381d6bca..7b64eb23 100644 --- a/docs/netsuke-design.md +++ b/docs/netsuke-design.md @@ -31,11 +31,11 @@ understand and execute. This separation of concerns—Netsuke managing build logic and Ninja managing execution—is the foundational principle of the entire architecture. -### 1.2 The Five Stages of a Netsuke Build +### 1.2 The Six Stages of a Netsuke Build The process of transforming a user's `Netsukefile` manifest into a completed -build artifact follows a distinct, five-stage pipeline. This multi-stage data -flow ensures that dynamic configurations are fully resolved into a static plan +build artefact now follows a six-stage pipeline. This data flow validates the +manifest as YAML first, then resolves all dynamic logic into a static plan before execution, a critical requirement for compatibility with Ninja. 1. Stage 1: Manifest Ingestion @@ -43,23 +43,30 @@ before execution, a critical requirement for compatibility with Ninja. The process begins by locating and reading the user's project manifest file (e.g., Netsukefile) from the filesystem into memory as a raw string. -2. Stage 2: Jinja Evaluation +2. Stage 2: Initial YAML Parsing - The raw manifest string is treated as a Jinja template. Netsuke's templating - engine processes this string, evaluating all expressions, executing control - structures (loops, conditionals), and applying filters. This stage resolves - all dynamic aspects of the build, producing a static, pure YAML string as - its output. + The raw string is parsed into an untyped `serde_yml::Value`. This step + ensures the manifest is valid YAML before any templating takes place. -3. Stage 3: YAML Parsing & Deserialization +3. Stage 3: Template Expansion - The static YAML string generated in the previous stage is passed to a YAML - parser. This parser validates the YAML syntax and deserializes the content - into a set of strongly typed Rust data structures. This collection of - structs, which directly mirrors the YAML schema, can be considered an - "unprocessed" Abstract Syntax Tree (AST) of the build plan. + Netsuke walks the YAML `Value`, evaluating Jinja macros, variables, and the + `foreach` and `when` keys. Each mapping containing these keys is expanded + with an iteration context providing `item` and optional `index`. Variable + lookups respect the precedence `globals` < `target.vars` < per-iteration + locals, and this context is preserved for later rendering. At this stage + Jinja must not modify the YAML structure directly; control constructs live + only within these explicit keys. Structural Jinja blocks (`{% ... %}`) are + not permitted to reshape mappings or sequences. -4. Stage 4: IR Generation & Validation +4. Stage 4: Deserialisation & Final Rendering + + The expanded `Value` is deserialised into strongly typed Rust structs. Jinja + expressions are then rendered, but only within string fields. Structural + templating using `{% %}` blocks is forbidden; all control flow must appear + in YAML values. + +5. Stage 5: IR Generation & Validation The AST is traversed to construct a canonical, fully resolved Intermediate Representation (IR) of the build. This IR represents the build as a static @@ -70,7 +77,7 @@ before execution, a critical requirement for compatibility with Ninja. exactly one of `rule`, `command`, or `script`. Circular dependencies and missing inputs are also detected at this stage. -5. Stage 5: Ninja Synthesis & Execution +6. Stage 6: Ninja Synthesis & Execution The final, validated IR is traversed by a code generator. This generator synthesizes the content of a `build.ninja` file, translating the IR's nodes @@ -84,18 +91,12 @@ before execution, a critical requirement for compatibility with Ninja. output suitable for caching or source control. ```mermaid -sequenceDiagram - participant User - participant Netsuke - participant IR_Generator - participant Ninja - - User->>Netsuke: Provide Netsukefile and environment - Netsuke->>IR_Generator: Parse and validate manifest - IR_Generator->>IR_Generator: Generate deterministic BuildGraph (IR) - IR_Generator->>Netsuke: Return BuildGraph (byte-for-byte deterministic) - Netsuke->>Ninja: Synthesize build.ninja and invoke Ninja - Ninja-->>User: Execute build and report results +flowchart TD + A[Stage 1:\nManifest Ingestion] --> B[Stage 2:\nInitial YAML Parsing] + B --> C[Stage 3:\nTemplate Expansion] + C --> D[Stage 4:\nDeserialisation & Final Rendering] + D --> E[Stage 5:\nIR Generation & Validation] + E --> F[Stage 6:\nNinja Synthesis & Execution] ``` ### 1.3 The Static Graph Mandate @@ -120,8 +121,8 @@ Representation (IR) generated in Stage 4. The IR serves as a static snapshot of the build plan after all Jinja logic has been resolved. It is the "object code" that the Netsuke "compiler" produces, which can then be handed off to the Ninja "assembler" for execution. This mandate for a pre-computed static graph -dictates the entire five-stage pipeline and establishes a clean boundary -between the user-facing logic layer and the machine-facing execution layer. +dictates the entire six-stage pipeline and establishes a clean boundary between +the user-facing logic layer and the machine-facing execution layer. ## Section 2: The Netsuke Manifest: A User-Centric YAML Schema @@ -336,19 +337,28 @@ are specified. Large sets of similar outputs can clutter a manifest when written individually. Netsuke supports a `foreach` entry within `targets` to generate multiple -outputs succinctly. The expression assigned to `foreach` is evaluated during -the Jinja render phase, and each value becomes `item` in the target context. +outputs succinctly. The `foreach` and optional `when` keys accept bare Jinja +expressions evaluated after the initial YAML pass. Each resulting value becomes +`item` in the target context, and the per-iteration environment is carried +forward to later rendering. ```yaml -- foreach: "{{ glob('assets/svg/*.svg') }}" +- foreach: glob('assets/svg/*.svg') + when: item | basename != 'logo.svg' name: "{{ outdir }}/{{ item | basename | replace('.svg', '.png') }}" rule: rasterise sources: "{{ item }}" ``` -Each element in the sequence produces a separate target. The resulting build -graph is still fully static and behaves the same as if every target were -declared explicitly. +Each element in the sequence produces a separate target. The iteration context: + +- `item`: current element +- `index`: 0-based index (optional) +- Variables resolve with precedence `globals` < `target.vars` < iteration locals + +Jinja control structures cannot shape the YAML; all templating must occur +within the string values. The resulting build graph is still fully static and +behaves the same as if every target were declared explicitly. ### 2.6 Table: Netsuke Manifest vs. Makefile @@ -1426,13 +1436,13 @@ treated as the default subcommand if none is provided, allowing for the common* The behaviour of each subcommand is clearly defined: - `Netsuke build [--emit FILE] [targets...]`: This is the primary and default - command. It executes the full five-stage pipeline: ingestion, Jinja - rendering, YAML parsing, IR generation, and Ninja synthesis. By default the - generated Ninja file is written to a securely created temporary location and - removed after the build completes. Supplying `--emit FILE` writes the Ninja - file to `FILE` and retains it. If no targets are provided on the command - line, the targets listed in the `defaults` section of the manifest are - built. +command. It executes the full six-stage pipeline: Manifest Ingestion, Initial +YAML Parsing, Template Expansion, Deserialisation & Final Rendering, IR +Generation & Validation, and Ninja Synthesis & Execution. By default the +generated Ninja file is written to a securely created temporary location and +removed after the build completes. Supplying `--emit FILE` writes the Ninja +file to `FILE` and retains it. If no targets are provided on the command line, +the targets listed in the `defaults` section of the manifest are built. - `Netsuke clean`: This command provides a convenient way to clean the build directory. It will invoke the Ninja backend with the appropriate flags, such @@ -1495,7 +1505,7 @@ goal. - **Success Criterion:** Netsuke can successfully take a `Netsukefile` file *without any Jinja syntax* and compile it to a `build.ninja` file, then - execute it to produce the correct artifacts. This phase validates the + execute it to produce the correct artefacts. This phase validates the entire static compilation pipeline. - **Phase 2: The Dynamic Engine** @@ -1565,7 +1575,7 @@ powerful build tool. The use of a decoupled IR, in particular, opens up many possibilities for future enhancements beyond the initial scope. - **Advanced Caching:** While Ninja provides excellent file-based incremental - build caching, Netsuke could implement a higher-level artifact caching layer. + build caching, Netsuke could implement a higher-level artefact caching layer. This could involve caching build outputs in a shared network location (e.g., S3) or a local content-addressed store, allowing for cache hits across different machines or clean checkouts. diff --git a/docs/roadmap.md b/docs/roadmap.md index d1354ad2..eb6ecf52 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -79,19 +79,24 @@ configurations with variables, control flow, and custom functions. - [x] Integrate the `minijinja` crate into the build pipeline. - - [x] Implement the two-pass parsing mechanism: the first pass renders the - manifest as a Jinja template, and the second pass parses the resulting pure - YAML string with serde_yml. + - [x] Implement data-first parsing: parse the manifest into a + `serde_yml::Value` (Stage 2: Initial YAML Parsing), expand `foreach` and + `when` entries with a Jinja environment (Stage 3: Template Expansion), then + deserialise the expanded tree into the typed AST and render remaining + string fields (Stage 4: Deserialisation & Final Rendering). - [x] Create a minijinja::Environment and populate its initial context with the global vars defined in the manifest. - [ ] **Dynamic Features and Custom Functions:** - - [x] Implement support for basic Jinja control structures (`{% if %}` and - `{% for %}`) + - [x] Evaluate Jinja expressions only within string values, forbidding + structural tags such as `{% if %}` and `{% for %}`. - - [ ] Implement the foreach key for target generation. + - [ ] Implement the `foreach` and `when` keys for target generation, + exposing `item` and optional `index` variables and layering + per-iteration locals over `target.vars` and manifest globals for + subsequent rendering phases. - [ ] Implement the essential custom Jinja function env(var_name) to read system environment variables. @@ -105,8 +110,9 @@ configurations with variables, control flow, and custom functions. - **Success Criterion:** - [ ] Netsuke can successfully build a manifest that uses variables, - conditional logic, the foreach loop, custom macros, and the glob() function - to discover and operate on source files. + conditional logic within string values, the `foreach` and `when` keys, + custom macros, and the `glob()` function to discover and operate on source + files. ## Phase 3: The "Friendly" Polish 🛡️ diff --git a/scripts/assert-file-absent.sh b/scripts/assert-file-absent.sh index eb3cb6c3..8fabbdd3 100755 --- a/scripts/assert-file-absent.sh +++ b/scripts/assert-file-absent.sh @@ -6,7 +6,7 @@ set -euo pipefail if [[ $# -ne 1 ]]; then echo "Usage: $(basename "$0") " >&2 - exit 2 # usage error + exit 64 # EX_USAGE fi file="$1" diff --git a/src/cli.rs b/src/cli.rs index 664e7c87..2812dd9c 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -97,7 +97,7 @@ pub enum Commands { /// Build specified targets (or default targets if none are given) `default`. Build(BuildArgs), - /// Remove build artifacts and intermediate files. + /// Remove build artefacts and intermediate files. Clean, /// Display the build dependency graph in DOT format for visualization. diff --git a/tests/runner_tests.rs b/tests/runner_tests.rs index 76c8ddfc..39ff6248 100644 --- a/tests/runner_tests.rs +++ b/tests/runner_tests.rs @@ -119,7 +119,7 @@ fn run_executes_ninja_without_persisting_file() { // Ensure no ninja file remains in project directory assert!(!temp.path().join("build.ninja").exists()); - // Drop the fake ninja artifacts. PATH is restored by guard drop. + // Drop the fake ninja artefacts. PATH is restored by guard drop. drop(ninja_path); } @@ -150,7 +150,7 @@ fn run_build_with_emit_keeps_file() { assert!(emitted.contains("build ")); assert!(!temp.path().join("build.ninja").exists()); - // Drop the fake ninja artifacts. PATH is restored by guard drop. + // Drop the fake ninja artefacts. PATH is restored by guard drop. drop(ninja_path); } @@ -179,7 +179,7 @@ fn run_build_with_emit_creates_parent_dirs() { assert!(emit_path.exists()); assert!(nested_dir.exists()); - // Drop the fake ninja artifacts. PATH is restored by guard drop. + // Drop the fake ninja artefacts. PATH is restored by guard drop. drop(ninja_path); }