Skip to content

Improve process module docs and refactor#111

Merged
leynos merged 5 commits intomainfrom
codex/add-doc-comments-and-refactor-process.rs
Jul 21, 2025
Merged

Improve process module docs and refactor#111
leynos merged 5 commits intomainfrom
codex/add-doc-comments-and-refactor-process.rs

Conversation

@leynos
Copy link
Copy Markdown
Owner

@leynos leynos commented Jul 20, 2025

Summary

  • add detailed Rustdoc to Options and stream processing functions
  • break process_stream_inner into smaller helpers
  • avoid cloning buffers during processing
  • clean up single-line returns

Testing

  • make fmt
  • make lint
  • make test
  • make markdownlint

https://chatgpt.com/codex/tasks/task_e_687ceafcc8a083229ca3990093c97e7d

Summary by Sourcery

Refactor the Markdown stream processing module by breaking down the core loop into reusable helper functions, eliminating buffer clones, and enriching the public API with detailed Rustdoc and usage examples.

Enhancements:

  • Extract flush_buffer, handle_fence_line, and handle_table_line helpers to simplify process_stream_inner
  • Avoid unnecessary buffer cloning by using in-place drains and appends
  • Clean up single-line returns for clearer control flow

Documentation:

  • Add comprehensive Rustdoc comments and examples for Options, process_stream_inner, process_stream, process_stream_no_wrap, and process_stream_opts

@sourcery-ai
Copy link
Copy Markdown
Contributor

sourcery-ai Bot commented Jul 20, 2025

Reviewer's Guide

This PR overhauls the process module by enriching Rust documentation, refactoring the core processing loop into dedicated helpers, eliminating redundant buffer clones, and streamlining control flow.

Class diagram for refactored process module helpers

classDiagram
    class Options {
        +bool wrap
        +bool ellipsis
        +bool fences
        +bool footnotes
    }
    class process_stream_inner {
        +process_stream_inner(lines: &[String], opts: Options) Vec<String>
    }
    class process_stream {
        +process_stream(lines: &[String]) Vec<String>
    }
    class process_stream_no_wrap {
        +process_stream_no_wrap(lines: &[String]) Vec<String>
    }
    class process_stream_opts {
        +process_stream_opts(lines: &[String], opts: Options) Vec<String>
    }
    class flush_buffer {
        +flush_buffer(buf: &mut Vec<String>, in_table: &mut bool, out: &mut Vec<String>)
    }
    class handle_fence_line {
        +handle_fence_line(line: &str, buf: &mut Vec<String>, in_code: &mut bool, in_table: &mut bool, out: &mut Vec<String>) bool
    }
    class handle_table_line {
        +handle_table_line(line: &str, buf: &mut Vec<String>, in_table: &mut bool) bool
    }
    process_stream_inner --> flush_buffer : uses
    process_stream_inner --> handle_fence_line : uses
    process_stream_inner --> handle_table_line : uses
    process_stream --> process_stream_inner : calls
    process_stream_no_wrap --> process_stream_inner : calls
    process_stream_opts --> process_stream_inner : calls
Loading

File-Level Changes

Change Details Files
Enhanced Rustdoc coverage across Options and processing functions
  • Added detailed comments and examples for Options struct
  • Documented helper functions: flush_buffer, handle_fence_line, handle_table_line
  • Provided examples for public APIs: process_stream_inner, process_stream, process_stream_no_wrap, process_stream_opts
src/process.rs
Refactored process_stream_inner into smaller helper functions
  • Extracted fence detection into handle_fence_line
  • Moved table buffering logic to handle_table_line
  • Centralized buffered flush logic in flush_buffer
  • Updated main loop to invoke these helpers
src/process.rs
Avoided unnecessary buffer cloning during processing
  • Replaced buf.clone() calls with in-place draining or append operations
  • Used out.append and out.extend to move data without copies
src/process.rs
Simplified control flow with early returns
  • Added early exit in flush_buffer when buffer is empty
  • Unified final buffer flush through flush_buffer
  • Removed repeated single-line return blocks in favor of helper returns
src/process.rs

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jul 20, 2025

Summary by CodeRabbit

  • Documentation

    • Expanded documentation for configuration options, including usage examples.
    • Added detailed doc comments and examples for new public functions.
  • New Features

    • Introduced three new public functions for processing streams with default and custom options, enhancing usability.
  • Refactor

    • Improved internal modularity for code handling, resulting in clearer and more maintainable logic.

No changes to existing behaviour or output.

Walkthrough

Refactor the src/process.rs module by extracting key logic into helper functions for buffer flushing, code fence handling, and table line processing. Expand documentation for the Options struct and introduce three new public API functions for stream processing, each with usage examples. Preserve the existing processing logic.

Changes

File(s) Change Summary
src/process.rs Refactor main processing loop by extracting flush_buffer, handle_fence_line, and handle_table_line helpers; expand Options documentation with examples; add public API functions process_stream, process_stream_no_wrap, and process_stream_opts with usage examples.

Sequence Diagram(s)

sequenceDiagram
    participant Caller
    participant Process as process_stream_opts
    participant Helpers as Helper Functions
    participant Output

    Caller->>Process: Call with lines and options
    loop For each line
        Process->>Helpers: handle_fence_line(line, ...)
        alt Fence line handled
            Helpers->>Output: Flush buffer, toggle code fence
        else Not a fence
            Process->>Helpers: handle_table_line(line, ...)
            alt Table line handled
                Helpers->>Output: Buffer or flush table lines
            else Not a table
                Process->>Helpers: Buffer for wrapping
            end
        end
    end
    Process->>Helpers: flush_buffer (at end)
    Helpers->>Output: Output remaining lines
    Process-->>Caller: Return processed lines
Loading

Possibly related PRs

Poem

Refactor, extract, and document anew,
Helpers for fences and tables in view.
Public APIs now clearer to see,
With options and examples for you and for me.
Code flows like a stream, neat and precise—
Modular structure, oh isn't that nice!
🎉📜✨


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d26f8ee and 4568644.

📒 Files selected for processing (1)
  • src/process.rs (5 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs

Instructions used from:

Sources:
📄 CodeRabbit Inference Engine

  • AGENTS.md

⚙️ CodeRabbit Configuration File

🔇 Additional comments (10)
src/process.rs (10)

15-31: Excellent documentation with clear examples.

The expanded documentation for the Options struct follows the coding guidelines perfectly. The example demonstrates proper usage without exposing implementation details.


38-45: Clear field documentation improves API usability.

The documentation comments for each field provide clear, concise descriptions that will be valuable in generated documentation.


48-60: Well-structured helper function with proper memory management.

The flush_buffer function correctly handles the early return pattern and uses std::mem::take for efficient memory management, avoiding unnecessary clones.


62-77: Clean fence detection with appropriate side effects.

The function properly encapsulates fence line handling logic with clear boolean return semantics. The side effects are well-contained and documented.


79-116: Robust table line handling with comprehensive boundary detection.

The function correctly handles various table boundary conditions including empty lines, new block markers (headings, lists, blockquotes), and numeric prefixes. The logic addresses the previous concern about non-table lines being incorrectly buffered.


118-140: Comprehensive documentation with clear usage example.

The documentation for process_stream_inner provides excellent context about its purpose and behaviour. The example demonstrates proper usage of the function with explicit options.


158-159: Effective refactoring improves code readability.

The extraction of helper functions significantly simplifies the main processing loop whilst maintaining identical functionality. The consistent use of flush_buffer throughout ensures uniform behaviour.

Also applies to: 167-168, 171-171, 175-175


191-204: Clear convenience function with appropriate documentation.

The process_stream function provides a sensible default configuration for the primary use case. Documentation clearly indicates its role as the main CLI interface function.


216-227: Well-documented no-wrap variant.

The function name and documentation clearly communicate its purpose. The example demonstrates the expected behaviour for table processing without paragraph wrapping.


234-252: Comprehensive options function with clear documentation.

The process_stream_opts function provides the necessary flexibility for advanced use cases. The documentation clearly explains when to use this function and the example demonstrates custom options configuration.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch codex/add-doc-comments-and-refactor-process.rs

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai auto-generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @leynos - I've reviewed your changes and they look great!

Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments

### Comment 1
<location> `src/process.rs:117` </location>
<code_context>
+        buf.push(line.trim_end().to_string());
+        return true;
+    }
+    if *in_table && !line.trim().is_empty() {
+        buf.push(line.trim_end().to_string());
+        return true;
+    }
+    false
</code_context>

<issue_to_address>
Non-table lines within a table block may be incorrectly buffered.

Currently, non-table lines are added to the table buffer if in_table is true, which may cause incorrect table parsing. Please add stricter checks to ensure only valid table lines are buffered or improve table block termination handling.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread src/process.rs
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

♻️ Duplicate comments (2)
src/process.rs (2)

134-137: Address potential issue with non-table line buffering.

The logic here may incorrectly buffer non-table lines when in_table is true, as identified in the past review. Add stricter validation to ensure only valid table content is buffered.

-    if *in_table && (line.contains('|') || crate::table::SEP_RE.is_match(line.trim())) {
+    if *in_table && (line.contains('|') || crate::table::SEP_RE.is_match(line.trim())) {
+        // Ensure this is actually table content, not just any line containing '|'
         buf.push(line.trim_end().to_string());
         return true;
     }

149-150: Clarify why non-table lines are buffered in table context.

This logic buffers any line when in_table is true, which could lead to incorrect table parsing. Add a comment explaining why this is intentional or fix the logic.

Add a comment explaining the rationale:

+        // Buffer continuation lines that are part of table cells
         buf.push(line.trim_end().to_string());
         return true;
📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1b41d7f and 1d2349e.

📒 Files selected for processing (1)
  • src/process.rs (6 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs

Instructions used from:

Sources:
📄 CodeRabbit Inference Engine

  • AGENTS.md

⚙️ CodeRabbit Configuration File

🔇 Additional comments (1)
src/process.rs (1)

12-28: Fix inconsistent American spelling in documentation.

The documentation uses American spelling "behavior" instead of the required en-GB spelling "behaviour".

-/// Processing options controlling the behaviour of [`process_stream_inner`].
+/// Processing options controlling the behaviour of [`process_stream_inner`].

Likely an incorrect or invalid review comment.

Comment thread src/process.rs Outdated
Comment thread src/process.rs Outdated
Comment thread src/process.rs Outdated
Comment thread src/process.rs Outdated
Comment thread src/process.rs
Comment thread src/process.rs Outdated
Comment thread src/process.rs Outdated
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (6)
src/process.rs (6)

60-75: Remove documentation for private helper function.

Same issue as flush_buffer - private helper functions don't need Rustdoc comments according to the guidelines.

-/// Detects fence lines and toggles code mode, flushing buffered content.
 fn handle_fence_line(

77-114: Remove documentation for private helper function.

Private helper functions should not have Rustdoc comments according to the coding guidelines.

-/// Buffers table lines, returning `true` when a line was consumed.
 fn handle_table_line(

45-58: Remove documentation entirely for private helper functions.

Private helper functions don't require Rustdoc comments according to the coding guidelines. The current documentation doesn't add value and creates maintenance overhead.

-/// Flushes buffered lines to `out`, formatting as a table when required.
-#[allow(clippy::extend_with_drain)] // maintain consistency across helpers
 fn flush_buffer(buf: &mut Vec<String>, in_table: &mut bool, out: &mut Vec<String>) {

185-199: Relocate function attribute after documentation.

The #[must_use] attribute appears before the documentation, violating the coding guidelines which specify that function attributes must be placed after doc comments.

-/// Processes a Markdown stream with all default options enabled.
-#[must_use]
-///
+/// Processes a Markdown stream with all default options enabled.
+///
 /// This is the primary convenience function used by the command-line
 /// interface. Paragraphs are wrapped and tables are reflowed.
 ///
@@ -198,6 +197,7 @@
 /// let out = process_stream(&lines);
 /// assert!(out.iter().any(|l| l.contains("| a | b |")));
 /// ```
+#[must_use]
 pub fn process_stream(lines: &[String]) -> Vec<String> {

212-224: Relocate function attribute after documentation.

The #[must_use] attribute appears before the documentation, violating the coding guidelines.

-/// Processes Markdown without wrapping paragraphs.
-#[must_use]
-///
+/// Processes Markdown without wrapping paragraphs.
+///
 /// Useful when only table reflow and code fence normalisation are required.
 ///
@@ -223,6 +222,7 @@
 /// let out = process_stream_no_wrap(&lines);
 /// assert!(out.iter().any(|l| l.contains("| a | b |")));
 /// ```
+#[must_use]
 pub fn process_stream_no_wrap(lines: &[String]) -> Vec<String> {

237-256: Relocate function attribute after documentation.

The #[must_use] attribute appears before the documentation, violating the coding guidelines.

-/// Runs [`process_stream_inner`] with custom [`Options`].
-#[must_use]
-///
+/// Runs [`process_stream_inner`] with custom [`Options`].
+///
 /// This is exposed for advanced use cases where callers want precise
 /// control over the processing pipeline.
 ///
@@ -255,6 +254,7 @@
 /// let out = process_stream_opts(&lines, opts);
 /// assert_eq!(out, vec!["text"]);
 /// ```
+#[must_use]
 pub fn process_stream_opts(lines: &[String], opts: Options) -> Vec<String> {
📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1d2349e and d26f8ee.

📒 Files selected for processing (1)
  • src/process.rs (6 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs

Instructions used from:

Sources:
📄 CodeRabbit Inference Engine

  • AGENTS.md

⚙️ CodeRabbit Configuration File

⏰ Context from checks skipped due to timeout of 240000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: build-test
🔇 Additional comments (1)
src/process.rs (1)

12-28: Documentation example violates coding guidelines.

The example demonstrates usage of a private helper function process_stream_opts that hasn't been defined yet at this point in the file. This creates forward references and doesn't clearly demonstrate the Options struct usage itself.

Replace the example to focus on the struct construction and field usage rather than function calls:

-/// # Examples
-///
-/// ```
-/// use mdtablefix::process::{Options, process_stream_opts};
-///
-/// let lines = vec!["example".to_string()];
-/// let opts = Options {
-///     wrap: false,
-///     ellipsis: false,
-///     fences: false,
-///     footnotes: false,
-/// };
-/// let out = process_stream_opts(&lines, opts);
-/// assert_eq!(out, vec!["example"]);
-/// ```
+/// # Examples
+///
+/// ```
+/// use mdtablefix::process::Options;
+///
+/// let opts = Options {
+///     wrap: true,
+///     ellipsis: false,
+///     fences: true,
+///     footnotes: false,
+/// };
+/// assert!(opts.wrap);
+/// assert!(opts.fences);
+/// ```

Likely an incorrect or invalid review comment.

Comment thread src/process.rs Outdated
@leynos leynos force-pushed the codex/add-doc-comments-and-refactor-process.rs branch from 8cf2630 to 4568644 Compare July 20, 2025 23:53
@leynos leynos merged commit c0121af into main Jul 21, 2025
2 checks passed
@leynos leynos deleted the codex/add-doc-comments-and-refactor-process.rs branch July 21, 2025 00:09
@coderabbitai coderabbitai Bot mentioned this pull request Jul 21, 2025
@coderabbitai coderabbitai Bot mentioned this pull request Aug 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant