Refactor HTML table handling

[leynos]

Summary

• add new html module with table conversion helpers
• update process_stream to use convert_html_tables
• expose regex constants for cross-module use

Testing

• cargo fmt
• cargo clippy -- -D warnings
• cargo test
• markdownlint '**/*.md' (fails: MD013 and other pre-existing issues)
• nixie AGENTS.md README.md docs/*.md

---

https://chatgpt.com/codex/tasks/task_e_684cda2becd88322841e913e61bce93d

Summary by Sourcery

Refactor HTML table handling by extracting conversion logic into a dedicated module, integrating it into the processing pipeline, and exposing regex constants for reuse.

Enhancements:

• Extract HTML table parsing and conversion functions into a new html module
• Preprocess input with html::convert_html_tables in process_stream
• Promote TABLE_START_RE, TABLE_END_RE, and FENCE_RE constants to pub(crate) visibility for cross-module use

Summary by CodeRabbit

• New Features

  • Added automatic conversion of HTML tables embedded in text documents to Markdown table format, including support for nested tables and preservation of indentation.

• Refactor

  • Streamlined processing by moving HTML table detection and conversion logic into a dedicated module, resulting in cleaner and more maintainable code.

[sourcery-ai]

Reviewer's Guide

This PR refactors HTML table handling by extracting the parsing and conversion logic into a dedicated html module, updates process_stream to leverage the new convert_html_tables helper, and exposes relevant regex constants for cross-module use.

Sequence Diagram: HTML Table Conversion in process_stream

sequenceDiagram
    participant caller
    participant ps as "process_stream (lib.rs)"
    participant cht as "html::convert_html_tables (html.rs)"

    caller->>ps: process_stream(lines)
    ps->>cht: convert_html_tables(lines)
    cht-->>ps: pre_processed_lines
    ps->>ps: Further Markdown/Code Block Processing
    ps-->>caller: result

Loading

File-Level Changes

Change	Details	Files
Extract HTML table conversion logic into html module	Create new src/html.rs Implement node_text, collect_text, collect_tables, collect_rows, table_node_to_markdown, html_table_to_markdown Add public convert_html_tables entrypoint	src/html.rs
Refactor process_stream to use new HTML helpers	Invoke html::convert_html_tables at start Remove inline HTML buffering variables and push_html_line Adjust main loop to process preprocessed lines	src/lib.rs
Expose regex constants for cross-module usage	Change visibility of TABLE_START_RE, TABLE_END_RE and FENCE_RE to pub(crate)	src/lib.rs

Possibly linked issues

• #0: PR implements the issue's suggested refactoring to extract HTML table processing into a separate preprocessing step.
• Add HTML table conversion support #13: The PR refactors HTML handling using a parser and case-insensitive regex to fix table detection issues.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Warning

Rate limit exceeded

@leynos has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 3 minutes and 54 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 698a65d and bb0f902.

📒 Files selected for processing (2)

• src/html.rs (1 hunks)
• src/lib.rs (3 hunks)

Error: Could not generate a valid Mermaid diagram after multiple attempts.

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (2)

src/html.rs (1)

7-15: Minor allocation overhead in node_text

Building an intermediate Vec<String> and then two consecutive join → split_whitespace → collect → join passes means the text is copied three times. A single pass that writes into a String while collapsing whitespace would avoid those allocations and reduce CPU time on large tables.

src/lib.rs (1)

116-119: Re-exporting regexes tightens coupling

Making TABLE_START_RE and TABLE_END_RE pub(crate) exposes implementation details and ties callers to the exact regex. Consider providing helper functions in html instead, keeping the regexes private.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 76b9e94 and 698a65d.

📒 Files selected for processing (2)

• src/html.rs (1 hunks)
• src/lib.rs (4 hunks)

🧰 Additional context used

🧬 Code Graph Analysis (1)

src/lib.rs (1)

src/html.rs (2)

• lines (106-109)
• convert_html_tables (129-191)

🔇 Additional comments (3)

src/html.rs (1)

17-41: Unbounded recursion risk in deeply-nested documents

collect_text is recursively called for every descendant. Extremely deep or adversarial HTML could overflow the stack and panic. Consider either an explicit stack/queue or recursion_limit guard.

src/lib.rs (2)

5-6: Module visibility is fine

Introducing mod html; keeps the parser private to the crate, matching existing design. No issues spotted.

---

250-251: Early HTML conversion path looks correct

process_stream now works on the pre-converted vector, significantly simplifying the function. Good refactor.

[sourcery-ai]

Hey @leynos - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}