Avoid collecting chars into Vec<char> in tokenize_markdown function

[coderabbitai]

Description

The tokenize_markdown function in src/lib.rs currently materialises the whole input as a Vec<char>, which duplicates the memory footprint of the string.

Location

• File: src/lib.rs
• Lines: 231-233 (approximately)
• Function: tokenize_markdown

Issue

The current implementation collects all characters into a Vec<char> which:

• Doubles the memory usage of the input string
• Creates unnecessary memory allocations
• Could impact performance for large inputs

Context

This performance issue was flagged in a previous review and was previously tracked in issue #63.

Backlinks

• PR: Add wrap boundary code tests #62
• Comment: Add wrap boundary code tests #62 (comment)

Requested by: @leynos

[leynos]

@coderabbitai looking at the codebase, is this issue still relevant? If the issue has been fixed in the code, does the documentation reflect this?

If it is still relevant, please provide an assessment of the issue criticality (critical, high, medium or low), based on risk of the issue impacting the user and the anticipated blast radius.

Once a decision has been made, apply the appropriate label.

If the issue is documentation ONLY (i.e., no code changes needed), apply the documentation label in addition to the appropriate criticality label.

Please also provide an impact of the required changes.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# First, let's check the current implementation of tokenize_markdown function
echo "=== Checking tokenize_markdown function in src/lib.rs ==="
ast-grep --pattern $'fn tokenize_markdown($$$) {
  $$$
}'

echo -e "\n=== Also searching with rg for context ==="
rg -nA 20 'fn tokenize_markdown' src/lib.rs

echo -e "\n=== Check for Vec<char> usage ==="
rg -n 'Vec<char>' src/lib.rs

echo -e "\n=== Check for .chars().collect() pattern ==="
rg -n '\.chars\(\)\.collect' src/lib.rs

Length of output: 182

---

🏁 Script executed:

#!/bin/bash
# Check if src/lib.rs exists and show its structure
echo "=== Checking if src/lib.rs exists ==="
ls -la src/lib.rs 2>&1 || echo "File not found"

echo -e "\n=== List all Rust files ==="
fd -e rs

echo -e "\n=== Show all function definitions in src/ ==="
ast-grep --pattern 'fn $_($$$) { $$$ }' | head -50

echo -e "\n=== Search for any tokenize functions ==="
rg -n 'fn.*tokenize' 

echo -e "\n=== Search for 'chars' usage in the codebase ==="
rg -n '\.chars\(\)' src/

echo -e "\n=== Check git log for tokenize_markdown ==="
git log --all --oneline --grep="tokenize" | head -20

Length of output: 7801

---

🏁 Script executed:

#!/bin/bash
# Let's examine the tokenize_markdown function and surrounding context
echo "=== Full tokenize_markdown function implementation ==="
rg -nA 30 'pub fn tokenize_markdown' src/wrap/tokenize.rs

echo -e "\n=== Check the context around line 175 where Vec<char> appears ==="
rg -nB 10 -A 10 'let chars: Vec<char> = text.chars\(\).collect\(\);' src/wrap/tokenize.rs

echo -e "\n=== Check git log for issue #68 or #63 ==="
git log --all --oneline | grep -E '#(68|63)' | head -10

echo -e "\n=== Check git log for PR #191 which refactored to iterate lazily ==="
git log --oneline --all | grep -E '#191' | head -5

echo -e "\n=== Show the commit that refactored tokenize_markdown ==="
git log --all --oneline --grep="iterate lazily" -n 1
git show 449a998 --stat 2>/dev/null || echo "Commit not in current branch"

Length of output: 3086

---

🏁 Script executed:

#!/bin/bash
# Check the tokenize_inline function to see if it calls segment_inline
echo "=== tokenize_inline function implementation ==="
rg -nA 50 'fn tokenize_inline' src/wrap/tokenize.rs | head -80

echo -e "\n=== Check where segment_inline is called ==="
rg -n 'segment_inline' src/wrap/tokenize.rs

echo -e "\n=== Check README or documentation for mention of memory usage or performance ==="
fd README -i -e md -X cat

echo -e "\n=== Check for CHANGELOG or similar ==="
fd -i CHANGELOG -e md -X cat 2>/dev/null || echo "No CHANGELOG found"

echo -e "\n=== Check if issue #68 is closed ==="
gh issue view 68 --json state,title,body,labels | jq .

Length of output: 11260

---

🏁 Script executed:

#!/bin/bash
# Check the next_token function to see if it calls segment_inline
echo "=== next_token function implementation ==="
rg -nB 5 -A 40 'fn next_token' src/wrap/tokenize.rs

echo -e "\n=== Check if segment_inline is used anywhere outside of tests ==="
rg -n 'segment_inline\(' src/ --type rust -g '!*test*'

echo -e "\n=== Show the full context of segment_inline usage ==="
rg -n 'segment_inline' --type rust | grep -v '//' | head -20

Length of output: 3578

---

🏁 Script executed:

#!/bin/bash
# Check the function in inline.rs that calls segment_inline
echo "=== Function calling segment_inline in inline.rs ==="
rg -nB 10 -A 20 'segment_inline' src/wrap/inline.rs

echo -e "\n=== Check what calls functions in inline.rs ==="
rg -n 'use.*inline' src/wrap/ --type rust

echo -e "\n=== Check wrap module structure ==="
cat src/wrap.rs | head -50

echo -e "\n=== Check if wrap_text or similar calls inline functions ==="
rg -n 'wrap_text|wrap_line' src/wrap.rs

Length of output: 3046

---

🏁 Script executed:

#!/bin/bash
# Check the full wrap_text implementation
echo "=== wrap_text implementation ==="
rg -nA 100 'pub fn wrap_text' src/wrap.rs | head -120

echo -e "\n=== Check flush_paragraph and handle_prefix_line from paragraph.rs ==="
rg -nA 15 'pub.*fn (flush_paragraph|handle_prefix_line)' src/wrap/paragraph.rs

echo -e "\n=== Show how paragraph.rs uses wrap_preserving_code ==="
rg -nC 5 'wrap_preserving_code' src/wrap/paragraph.rs

Length of output: 6072

---

@leynos, I've analyzed the codebase and here's my assessment:

Issue Status: Partially Fixed, Still Relevant

The issue was partially addressed in PR #191 (Aug 2025). The main tokenize_markdown function now iterates lazily, avoiding the collection of the entire input into memory. However, the issue persists in the segment_inline helper function.

Current State

Fixed:

• tokenize_markdown (src/wrap/tokenize.rs:361) now uses lazy iteration via source.lines().peekable()

Still Problematic:

• segment_inline (src/wrap/tokenize.rs:175) still collects: let chars: Vec<char> = text.chars().collect();

Call Path

When users invoke --wrap:

wrap_text → flush_paragraph/append_wrapped_with_prefix 
→ wrap_preserving_code → segment_inline 
→ Vec<char> allocation (per line)

Criticality: MEDIUM

Risk Assessment

• User Impact: Affects all users of the --wrap flag (a core feature)
• Frequency: Memory allocation occurs for each paragraph, list item, or blockquote line during wrapping
• Blast Radius: Limited to wrapping functionality; table reflowing is unaffected
• Severity: Performance degradation and increased memory usage, but no crashes or data loss

Why Medium (not High or Low)?

• Not High/Critical: Doesn't affect all users; only impacts --wrap users; no correctness issues
• Not Low: Wrapping is a prominently advertised feature with real performance implications
• Medium: Measurable performance impact on a commonly-used feature

Documentation Status

The documentation does not reflect the partial fix:

• README mentions concurrency/memory considerations for multi-file processing only
• No mention of per-line memory allocations in segment_inline
• The issue description itself contains outdated information (wrong file path: src/lib.rs vs actual src/wrap/tokenize.rs)

Impact of Required Changes

Code Changes:

• Refactor segment_inline to use iterators/slices instead of Vec<char>
• Update helper functions (scan_while, collect_range, has_odd_backslash_escape) that rely on indexed access to the char vector
• Follow the pattern established in PR Refactor tokenize_markdown to iterate lazily #191 for lazy iteration

Estimated Effort: Moderate - requires careful refactoring of character-by-character parsing logic

Risk: Low - well-tested code path with existing test coverage (tests/wrap/tokenize_markdown.rs)

I'll now apply the appropriate label.

---

🏁 Script executed:

#!/bin/bash
# Apply the medium criticality label to issue #68
gh issue edit 68 --add-label "medium"

Length of output: 46

---

Perfect! I've successfully applied the medium criticality label to issue #68.

Summary

✅ Label Applied: medium

Issue Status: Partially resolved in PR #191, but segment_inline still requires optimization

Documentation: Does not reflect the partial fix or remaining issue

Code Changes Required: Yes - refactor segment_inline to avoid Vec<char> collection

Recommendation: Consider prioritizing this issue for the next release cycle, as wrapping is a frequently-used feature. The fix would follow the same lazy iteration pattern successfully applied in PR #191.

---

Would you like me to create a more detailed implementation plan or open a separate documentation issue to track updating the README with memory usage guidance?

  (\__/)
  (='.'=)  *noted for future reviews*
  (")_(")