Skip to content

Refactor read_pipe and workspace-scoped tempfile plumbing#232

Merged
leynos merged 3 commits intocodex/enforce-command-output-size-limitsfrom
terragon/refactor-read-pipe-function-33sbru
Nov 6, 2025
Merged

Refactor read_pipe and workspace-scoped tempfile plumbing#232
leynos merged 3 commits intocodex/enforce-command-output-size-limitsfrom
terragon/refactor-read-pipe-function-33sbru

Conversation

@leynos
Copy link
Copy Markdown
Owner

@leynos leynos commented Nov 6, 2025

Summary

  • Refactors read_pipe to use dedicated helper functions for Capture and Tempfile modes
  • read_pipe delegates to two helpers based on mode:
    • read_pipe_capture(reader, limit)
    • read_pipe_tempfile(reader, limit)
  • read_pipe signature updated to take reader by value (no longer mut) while still working with Read
  • read_pipe_capture collects all bytes into a Vec and returns PipeOutcome::Bytes
  • read_pipe_tempfile streams data into a NamedTempFile and returns PipeOutcome::Tempfile with a UTF-8 path
  • Preserved limit handling and error mapping (Io and StreamPathNotUtf8)
  • Introduces internal plumbing for tempfile management (CommandConfig, CommandTempDir, CommandTempFile) to centralize workspace-backed temp file handling
  • StdlibConfig updated to carry an optional workspace_root_path so tempfile paths can be created inside the workspace
  • Manifest wiring updated to propagate workspace_root_path into StdlibConfig
  • Tests updated to use a single-entry context helper for templates (render_with_single_context)
  • Tests and feature tests updated to work with the new workspace-root-aware tempfile flow
  • Docs updated to reflect per-mode read_pipe helpers and their behavior

Changes

Core

  • In src/stdlib/command.rs:
    • Read pipe now delegates to two helpers based on mode:
      • read_pipe_capture(reader, limit)
      • read_pipe_tempfile(reader, limit)
    • read_pipe signature updated to take reader by value (no longer mut) while still working with Read
    • read_pipe_capture collects all bytes into a Vec and returns PipeOutcome::Bytes
    • read_pipe_tempfile streams data into a NamedTempFile and returns PipeOutcome::Tempfile with a UTF-8 path
    • Preserved limit handling and error mapping (Io and StreamPathNotUtf8)
    • Added constants and internal structures to support workspace-scoped tempfile creation

Tests

  • In tests/steps/stdlib_steps/rendering.rs:

    • Added render_with_single_context(world, template, key, value) helper to render a template with a single key-value pair
    • Updated render_stdlib_template_with_url, render_stdlib_template_with_command, and render_stdlib_template_with_text to use render_with_single_context with a single-entry map (url, cmd, text)
  • Additional unit tests for read_pipe_CAPTURE and read_pipe_TEMPFILE behavior were added to validate new helpers (see test additions in updated diff)

Other

  • Updated docs to reflect per-mode read_pipe helpers and their behavior
  • StdlibConfig updated to carry an optional workspace_root_path so tempfile paths can be created inside the workspace
  • Tests and feature tests updated to work with the new workspace-root-aware tempfile flow
  • Manifest wiring updated to pass workspace root path into StdlibConfig

Why

  • Isolates and simplifies per-mode logic for pipe reading, making future maintenance easier
  • Reduces duplication by extracting common reading/streaming behavior into helper functions
  • Improves test reliability and readability by consolidating context creation for single-key templates
  • Enables tempfile outputs to live inside a configured workspace directory, improving testability and determinism

Validation

  • Build and run test suite locally
  • Rendering tests updated to use the single-context helper pass
  • No external API changes; behavior remains equivalent to prior implementation

📎 Task: https://www.terragonlabs.com/task/c81b565b-2354-4ea0-a22b-7581d2b62469

…functions

Refactored the read_pipe function in src/stdlib/command.rs by splitting its logic based on OutputMode into two distinct functions, read_pipe_capture and read_pipe_tempfile. This improves code clarity and separation of concerns while maintaining existing behavior.

Additionally, refactored template rendering test helpers in tests/steps/stdlib_steps/rendering.rs to consolidate context construction into a single helper function render_with_single_context, reducing code duplication in tests.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
@sourcery-ai
Copy link
Copy Markdown
Contributor

sourcery-ai Bot commented Nov 6, 2025

Reviewer's Guide

Refactors read_pipe by extracting capture and tempfile modes into read_pipe_capture and read_pipe_tempfile helpers with preserved limit and error handling, and streamlines template rendering tests by introducing a single-key context helper.

Class diagram for refactored pipe reading helpers in command module

classDiagram
    class PipeSpec {
        +into_limit()
        +mode()
    }
    class PipeLimit {
        +record(read: usize)
    }
    class PipeOutcome {
        Bytes
        Tempfile
    }
    class CommandFailure {
        Io
        StreamPathNotUtf8
    }
    class NamedTempFile {
        +new()
        +write_all(slice)
        +flush()
        +into_temp_path()
    }
    class Utf8PathBuf {
        +from_path_buf(path)
    }
    class read_pipe {
        +read_pipe<R>(reader: R, spec: PipeSpec) Result<PipeOutcome, CommandFailure>
    }
    class read_pipe_capture {
        +read_pipe_capture<R>(reader: R, limit: PipeLimit) Result<PipeOutcome, CommandFailure>
    }
    class read_pipe_tempfile {
        +read_pipe_tempfile<R>(reader: R, limit: PipeLimit) Result<PipeOutcome, CommandFailure>
    }
    read_pipe <|-- read_pipe_capture
    read_pipe <|-- read_pipe_tempfile
    PipeOutcome <.. read_pipe
    PipeOutcome <.. read_pipe_capture
    PipeOutcome <.. read_pipe_tempfile
    CommandFailure <.. read_pipe
    CommandFailure <.. read_pipe_capture
    CommandFailure <.. read_pipe_tempfile
    NamedTempFile <.. read_pipe_tempfile
    Utf8PathBuf <.. read_pipe_tempfile
Loading

Class diagram for new single-context rendering helper in tests

classDiagram
    class World {}
    class render_with_single_context {
        +render_with_single_context(world, template, key, value)
    }
    class render_stdlib_template_with_url {
        +render_stdlib_template_with_url(world, template, url)
    }
    class render_stdlib_template_with_command {
        +render_stdlib_template_with_command(world, template, cmd)
    }
    class render_stdlib_template_with_text {
        +render_stdlib_template_with_text(world, template, text)
    }
    render_stdlib_template_with_url --> render_with_single_context
    render_stdlib_template_with_command --> render_with_single_context
    render_stdlib_template_with_text --> render_with_single_context
Loading

File-Level Changes

Change Details Files
Delegate read_pipe logic to helper functions
  • Removed inline capture and tempfile loops from read_pipe
  • Updated read_pipe signature to take reader by value
  • Delegated modes to read_pipe_capture and read_pipe_tempfile
src/stdlib/command.rs
Implement read_pipe_capture helper for Capture mode
  • Read data into Vec in 8KB chunks
  • Enforce limit and map IO errors
  • Return PipeOutcome::Bytes
src/stdlib/command.rs
Implement read_pipe_tempfile helper for Tempfile mode
  • Stream data into NamedTempFile with limit enforcement
  • Flush file, keep temp path, and handle UTF-8 conversion errors
  • Return PipeOutcome::Tempfile with path
src/stdlib/command.rs
Introduce single-context rendering helper in tests
  • Added render_with_single_context building a single-entry BTreeMap
  • Updated render_stdlib_template_with_url/command/text to use the new helper
tests/steps/stdlib_steps/rendering.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Nov 6, 2025

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch terragon/refactor-read-pipe-function-33sbru

Comment @coderabbitai help to get the list of available commands and usage tips.

@leynos leynos marked this pull request as ready for review November 6, 2025 13:07
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there - I've reviewed your changes and found some issues that need to be addressed.

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `src/stdlib/command.rs:915` </location>
<code_context>
     match spec.mode() {
-        OutputMode::Capture => {
-            let mut buf = Vec::new();
-            let mut chunk = [0_u8; 8192];
-            loop {
-                let read = reader.read(&mut chunk).map_err(CommandFailure::Io)?;
</code_context>

<issue_to_address>
**suggestion:** Chunk size is hardcoded; consider making it configurable.

Since both functions use the same chunk size, exposing it as a constant or config parameter would simplify future performance adjustments.

Suggested implementation:

```rust
const PIPE_CHUNK_SIZE: usize = 8192;

fn read_pipe<R>(reader: R, spec: PipeSpec) -> Result<PipeOutcome, CommandFailure>

```

```rust
            let mut buf = Vec::new();
            let mut chunk = [0_u8; PIPE_CHUNK_SIZE];
            loop {
                let read = reader.read(&mut chunk).map_err(CommandFailure::Io)?;

```

You should also update any other function in this file (such as `read_pipe_tempfile`) that uses `[0_u8; 8192]` to use `[0_u8; PIPE_CHUNK_SIZE]` for consistency.
</issue_to_address>

### Comment 2
<location> `src/stdlib/command.rs:918` </location>
<code_context>
+    }
+}
+
+fn read_pipe_capture<R>(mut reader: R, mut limit: PipeLimit) -> Result<PipeOutcome, CommandFailure>
+where
+    R: Read,
</code_context>

<issue_to_address>
**issue (review_instructions):** Add behavioural and unit tests for the new read_pipe_capture function.

You have introduced the read_pipe_capture function, but there are no corresponding behavioural or unit tests verifying its correctness. Ensure that both types of tests are present to validate this new functionality.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*`

**Instructions:**
For any new feature or change to an existing feature, both behavioural *and* unit tests are required.

</details>
</issue_to_address>

### Comment 3
<location> `src/stdlib/command.rs:935` </location>
<code_context>
+    Ok(PipeOutcome::Bytes(buf))
+}
+
+fn read_pipe_tempfile<R>(mut reader: R, mut limit: PipeLimit) -> Result<PipeOutcome, CommandFailure>
+where
+    R: Read,
</code_context>

<issue_to_address>
**issue (review_instructions):** Add behavioural and unit tests for the new read_pipe_tempfile function.

The new read_pipe_tempfile function lacks both behavioural and unit tests. Please provide tests to ensure its correct operation and to comply with the review instructions.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*`

**Instructions:**
For any new feature or change to an existing feature, both behavioural *and* unit tests are required.

</details>
</issue_to_address>

### Comment 4
<location> `src/stdlib/command.rs:907` </location>
<code_context>
 }

-fn read_pipe<R>(mut reader: R, spec: PipeSpec) -> Result<PipeOutcome, CommandFailure>
+fn read_pipe<R>(reader: R, spec: PipeSpec) -> Result<PipeOutcome, CommandFailure>
 where
     R: Read,
</code_context>

<issue_to_address>
**issue (review_instructions):** The function 'read_pipe' is missing a doc comment, which is required before any attributes.

Please add a doc comment above the 'read_pipe' function to comply with documentation standards.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.rs`

**Instructions:**
Place function attributes **after** doc comments.

</details>
</issue_to_address>

### Comment 5
<location> `src/stdlib/command.rs:918` </location>
<code_context>
+    }
+}
+
+fn read_pipe_capture<R>(mut reader: R, mut limit: PipeLimit) -> Result<PipeOutcome, CommandFailure>
+where
+    R: Read,
</code_context>

<issue_to_address>
**issue (review_instructions):** The function 'read_pipe_capture' is missing a doc comment, which is required before any attributes.

Please add a doc comment above the 'read_pipe_capture' function to comply with documentation standards.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.rs`

**Instructions:**
Place function attributes **after** doc comments.

</details>
</issue_to_address>

### Comment 6
<location> `src/stdlib/command.rs:935` </location>
<code_context>
+    Ok(PipeOutcome::Bytes(buf))
+}
+
+fn read_pipe_tempfile<R>(mut reader: R, mut limit: PipeLimit) -> Result<PipeOutcome, CommandFailure>
+where
+    R: Read,
</code_context>

<issue_to_address>
**issue (review_instructions):** The function 'read_pipe_tempfile' is missing a doc comment, which is required before any attributes.

Please add a doc comment above the 'read_pipe_tempfile' function to comply with documentation standards.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.rs`

**Instructions:**
Place function attributes **after** doc comments.

</details>
</issue_to_address>

### Comment 7
<location> `src/stdlib/command.rs:907` </location>
<code_context>
 }

-fn read_pipe<R>(mut reader: R, spec: PipeSpec) -> Result<PipeOutcome, CommandFailure>
+fn read_pipe<R>(reader: R, spec: PipeSpec) -> Result<PipeOutcome, CommandFailure>
 where
     R: Read,
</code_context>

<issue_to_address>
**issue (review_instructions):** The module 'command.rs' does not begin with a `//!` comment as required.

Please add a `//!` module-level comment at the top of 'command.rs' to describe its purpose.

<details>
<summary>Review instructions:</summary>

**Path patterns:** `**/*.rs`

**Instructions:**
Every module **must** begin with a `//!` comment.

</details>
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread src/stdlib/command.rs
…mpfiles for command output

- Introduce distinct configurable byte limits for stdout capture and streaming in CommandConfig.
- Implement workspace-rooted temporary file creation for streaming command outputs, replacing system temp directory usage.
- Modify child process pipe readers to enforce configured limits and produce bytes or workspace-based tempfiles accordingly.
- Propagate workspace root path into stdlib configuration, enabling temp file placement inside project directories.
- Add tests validating capturing and streaming output within limits and error handling on limit exceedance.
- Enhance documentation with a Mermaid class diagram illustrating pipe reading abstractions.
- Update stdlib configuration defaults to include workspace root path for consistent temp file handling.
- Adjust tests and feature files to verify new output limit enforcement and streaming behavior.

This improves resource control and workspace hygiene for command output handling in Netsuke standard library.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
@leynos leynos changed the title Refactor read_pipe with helper functions Refactor read_pipe to use capture/tempfile helpers Nov 6, 2025
codescene-delta-analysis[bot]

This comment was marked as outdated.

…cloning

Changed functions to accept `&CommandContext` instead of consuming `CommandContext` by value to improve efficiency and avoid unnecessary cloning. Made `CommandContext::new` a `const fn`. Updated related pipe reader functions to accept config by reference rather than Arc cloning. Fixed a slice indexing panic by validating bounds before write.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
@leynos leynos changed the title Refactor read_pipe to use capture/tempfile helpers Refactor read_pipe and workspace-scoped tempfile plumbing Nov 6, 2025
Copy link
Copy Markdown

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gates Failed
Enforce critical code health rules (1 file with Low Cohesion)

Gates Passed
4 Quality Gates Passed

See analysis details in CodeScene

Reason for failure
Enforce critical code health rules Violations Code Health Impact
command.rs 1 critical rule 8.68 → 8.55 Suppress

Quality Gate Profile: Pay Down Tech Debt
Want more control? Customize Code Health rules or catch issues early with our IDE extension and CLI tool.

@leynos leynos merged commit 66b3f80 into codex/enforce-command-output-size-limits Nov 6, 2025
12 of 13 checks passed
@leynos leynos deleted the terragon/refactor-read-pipe-function-33sbru branch November 6, 2025 19:12
leynos added a commit that referenced this pull request Nov 8, 2025
* Enforce bounded command output and stream large results

* Refactor read_pipe and workspace-scoped tempfile plumbing (#232)

* refactor(stdlib): split read_pipe into separate capture and tempfile functions

Refactored the read_pipe function in src/stdlib/command.rs by splitting its logic based on OutputMode into two distinct functions, read_pipe_capture and read_pipe_tempfile. This improves code clarity and separation of concerns while maintaining existing behavior.

Additionally, refactored template rendering test helpers in tests/steps/stdlib_steps/rendering.rs to consolidate context construction into a single helper function render_with_single_context, reducing code duplication in tests.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* feat(stdlib/command): enforce byte limits and use workspace-rooted tempfiles for command output

- Introduce distinct configurable byte limits for stdout capture and streaming in CommandConfig.
- Implement workspace-rooted temporary file creation for streaming command outputs, replacing system temp directory usage.
- Modify child process pipe readers to enforce configured limits and produce bytes or workspace-based tempfiles accordingly.
- Propagate workspace root path into stdlib configuration, enabling temp file placement inside project directories.
- Add tests validating capturing and streaming output within limits and error handling on limit exceedance.
- Enhance documentation with a Mermaid class diagram illustrating pipe reading abstractions.
- Update stdlib configuration defaults to include workspace root path for consistent temp file handling.
- Adjust tests and feature files to verify new output limit enforcement and streaming behavior.

This improves resource control and workspace hygiene for command output handling in Netsuke standard library.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* refactor(stdlib/command): use references for CommandContext to avoid cloning

Changed functions to accept `&CommandContext` instead of consuming `CommandContext` by value to improve efficiency and avoid unnecessary cloning. Made `CommandContext::new` a `const fn`. Updated related pipe reader functions to accept config by reference rather than Arc cloning. Fixed a slice indexing panic by validating bounds before write.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

---------

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* Refactor rendering tests with macro and absolute workspace roots (#234)

* refactor(stdlib,manifest): improve workspace root handling and render helpers

- Resolve workspace root as absolute path in manifest loading, supporting relative paths by joining with current working directory.
- Enforce absolute workspace root path in StdlibConfig with panic on invalid input.
- Return error if workspace root not configured when creating command temp files.
- Replace unsafe slice indexing with safe slice usage in command file writes.
- Introduce macro in stdlib_steps for DRY rendering with world string fields, simplifying template rendering functions.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* feat(manifest): add workspace root path handling and related tests

- Handle resolution of relative and absolute workspace root paths with UTF-8 validation
- Expose workspace_root_path() getter in StdlibConfig
- Add tests for resolving workspace roots and rejecting non-UTF-8 paths
- Add test ensuring command tempdir requires workspace root path
- Add manifest subcommand test for relative manifest paths

These changes improve manifest workspace handling by enforcing UTF-8 constraints and providing better API visibility and test coverage.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* refactor(manifest): extract absolute workspace root resolution to helper

Refactored workspace root resolution logic in stdlib_config_for_manifest into a
new function resolve_absolute_workspace_root for clarity and reuse. Updated
related tests to combine relative and absolute workspace root checks into a
single parameterized test for better coverage and reduced duplication.

Additionally, introduced assert_output_limit_error helper in stdlib/command tests
to consolidate output limit error assertions in multiple test cases.

This improves code maintainability and test clarity.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

---------

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* Enforce workspace-scoped tempfile creation with UTF-8 paths (#235)

* refactor(command): replace tempfile crate with cap-std for command temp files

Reworked command temporary file management to use cap-std crate instead of tempfile.
Implemented a custom unique tempfile naming strategy with atomic counter.
Improved tempfile lifecycle handling and removal on drop when not persisted.
Standardized sanitation of tempfile labels for safer filenames.
Added robust error handling for tempfile creation attempts.
Smoothed tempfile usage in pipe reading and empty tempfile creation.
Changed workspace root path validation message for clarity.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* refactor(command): replace manual tempfile creation with tempfile crate

Replaced the custom logic for creating temporary files within the command module with the tempfile crate's Builder API. This simplifies tempfile creation, removes counters and manual uniqueness handling, and improves error handling. Dropped the custom CommandTempFile's complex management for a simpler wrapper around NamedTempFile, ensuring proper cleanup and persistence semantics. Added tests for label sanitization and tempfile lifecycle behavior.

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

* style(stdlib/command): simplify flush call by chaining method calls

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

---------

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>

---------

Co-authored-by: terragon-labs[bot] <terragon-labs[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant