Whitespace tokens discarded during text wrapping causing words to glue together

[coderabbitai]

A bug has been identified in the wrap_preserving_code function in src/lib.rs where whitespace tokens that overflow the current line are discarded, causing words to be glued together after wrapping.

The issue occurs when a whitespace token would exceed the line width limit - the current implementation skips the token entirely instead of preserving it for the next line. This results in words appearing concatenated without proper spacing.

Location: src/lib.rs in the wrap_preserving_code function around lines 314-318

Reported by: @leynos

References:

• Pull Request: Add wrap boundary code tests #62
• Comment: Add wrap boundary code tests #62 (comment)

[leynos]

@coderabbitai looking at the codebase, is this issue still relevant? If the issue has been fixed in the code, does the documentation reflect this?

If it is still relevant, please provide an assessment of the issue criticality (critical, high, medium or low), based on risk of the issue impacting the user and the anticipated blast radius.

Once a decision has been made, apply the appropriate label.

If the issue is documentation ONLY (i.e., no code changes needed), apply the documentation label in addition to the appropriate criticality label.

Please also provide an impact of the required changes.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Read the wrap_preserving_code function from src/lib.rs
cat src/lib.rs

Length of output: 1861

---

🏁 Script executed:

#!/bin/bash
# Search for any tests related to wrap_preserving_code or whitespace handling
rg -n "wrap_preserving_code|whitespace" --type rust

Length of output: 6531

---

🏁 Script executed:

#!/bin/bash
# Check for any recent changes to src/lib.rs that might have fixed this issue
git log --oneline --all -20 -- src/lib.rs

Length of output: 936

---

🏁 Script executed:

#!/bin/bash
# Look for test files
fd -e rs test

Length of output: 53

---

🏁 Script executed:

#!/bin/bash
# Read the wrap_preserving_code function implementation
cat src/wrap/inline.rs

Length of output: 5494

---

🏁 Script executed:

#!/bin/bash
# Read the line_buffer implementation which is used by wrap_preserving_code
cat src/wrap/line_buffer.rs

Length of output: 4902

---

🏁 Script executed:

#!/bin/bash
# Check if there's a test for the specific bug mentioned in the issue
rg -n -A5 -B5 "glue|whitespace.*discard|word.*concat" tests/

Length of output: 0

---

🏁 Script executed:

#!/bin/bash
# Look at the git history for wrap/inline.rs to see if there were fixes
git log --oneline --all -30 -- src/wrap/inline.rs

Length of output: 62

---

🏁 Script executed:

#!/bin/bash
# Check PR #62 to see what the original fix was
gh pr view 62 --json title,body,state,mergedAt,commits

Length of output: 3293

---

🏁 Script executed:

#!/bin/bash
# Look for tests that might cover the whitespace issue
cat tests/wrap/tests.rs

Length of output: 51

---

🏁 Script executed:

#!/bin/bash
# Create a test case to see if the issue still exists
cat > /tmp/test_whitespace_wrap.rs << 'EOF'
// Simulate the wrap_preserving_code function behavior
// Test case: text that should wrap with whitespace preserved

fn main() {
    // We'll check if there are specific tests for whitespace handling
    println!("Checking for whitespace wrapping tests");
}
EOF

# Look for existing wrap tests
rg -n "wrap_preserving_code" tests/ -A10 -B2

Length of output: 0

---

🏁 Script executed:

#!/bin/bash
# Check the specific git commits around PR 62
git log --all --grep="62" --oneline

Length of output: 515

---

🏁 Script executed:

#!/bin/bash
# Look at the full git log for the wrap module
git log --oneline --all -- src/wrap/

Length of output: 823

---

🏁 Script executed:

#!/bin/bash
# Look at the specific commit that refactored wrap_preserving_code
git show 822f964 --stat

Length of output: 1494

---

🏁 Script executed:

#!/bin/bash
# Check all wrap tests to see if there's a test for the whitespace discard issue
fd -e rs . tests/wrap/ -x cat {}

Length of output: 38723

---

🏁 Script executed:

#!/bin/bash
# Create a simple test to verify if the bug exists
cat > /tmp/test_whitespace_bug.sh << 'EOF'
#!/bin/bash
# Create a test file to check if whitespace is preserved during wrapping
cat > /tmp/test_input.md << 'MDFILE'
This is a long line with words that should wrap properly without gluing words together when whitespace overflows.
MDFILE

echo "Testing if whitespace tokens are preserved during wrapping..."
# We can't actually run the binary, but we can check the code logic
EOF

# Instead, let's look at the exact issue - check push_non_whitespace_span usage
rg -n "push_non_whitespace_span" src/wrap/ -B5 -A5

Length of output: 1170

---

🏁 Script executed:

#!/bin/bash
# Look at the original issue discussion in PR #62
gh pr view 62 --json comments --jq '.comments[].body' | head -50

Length of output: 2754

---

🏁 Script executed:

#!/bin/bash
# Look at the specific code path where whitespace is discarded
# Check if there are any related issues or PRs
gh issue list --state all --json number,title,state --jq '.[] | select(.number == 67)'

Length of output: 118

---

🏁 Script executed:

#!/bin/bash
# Create a demonstration of the bug
cat > /tmp/test_whitespace_issue.md << 'EOF'
# Testing whitespace preservation

The scenario where this bug would manifest:
- A line has "word1 " in the buffer
- The next token span is " word2" which would overflow
- Cannot be split (e.g., it's a single token group)
- Not trailing whitespace (there's more content after)

In this case, the code path goes to:
1. buffer.flush_into(&mut lines) - flushes "word1"
2. buffer.push_non_whitespace_span(&tokens, i, group_end) - adds "word2" WITHOUT the leading space

Result: "word1" on one line, "word2" on the next line (missing the space between them)
EOF
cat /tmp/test_whitespace_issue.md

# Check the specific problematic code path
echo "=== Problematic code in src/wrap/inline.rs ==="
sed -n '180,195p' src/wrap/inline.rs

echo ""
echo "=== The push_non_whitespace_span function that discards whitespace ==="
sed -n '58,70p' src/wrap/line_buffer.rs

Length of output: 1456

---

🏁 Script executed:

#!/bin/bash
# Check if there's any discussion about this specific issue in PR or issue comments
gh issue view 67 --json body,comments --jq '.body, .comments[].body'

Length of output: 1331

---

🏁 Script executed:

#!/bin/bash
# Apply appropriate label based on criticality assessment
# First, let's check what labels are available
gh label list --limit 100

Length of output: 644

---

🏁 Script executed:

#!/bin/bash
# Apply the 'high' criticality label to issue #67
gh issue edit 67 --add-label "high"

Length of output: 46

---

@leynos, I've completed the analysis. The issue is still relevant and requires code changes.

Issue Status: CONFIRMED

After examining the codebase, the bug described in issue #67 still exists in the current code. The problematic code path is present in:

• src/wrap/inline.rs lines 187-189
• src/wrap/line_buffer.rs lines 58-66

The Problem

When wrap_preserving_code encounters a token group that:

1. Contains leading whitespace
2. Would overflow the current line width
3. Cannot be split via split_with_span
4. Is not trailing whitespace

The code executes:

buffer.flush_into(&mut lines);
buffer.push_non_whitespace_span(&tokens, i, group_end);

The push_non_whitespace_span method explicitly discards all whitespace tokens:

pub(crate) fn push_non_whitespace_span(&mut self, tokens: &[String], start: usize, end: usize) {
    for tok in &tokens[start..end] {
        if tok.chars().all(char::is_whitespace) {
            continue;  // ← Whitespace tokens are skipped
        }
        self.push_token(tok.as_str());
    }
}

This causes words to be concatenated without spaces when wrapping occurs.

Criticality Assessment: HIGH

I've applied the high label to the issue. Here's the rationale:

Risk factors:

• Data corruption: Produces visibly incorrect output (words glued together)
• Core functionality: Affects the primary text-wrapping feature
• User impact: Results in unreadable Markdown that requires manual correction
• Moderate frequency: Requires specific conditions but not rare edge cases

Not critical because:

• Doesn't affect all wrapping scenarios
• Workarounds exist (manual line breaks, avoiding certain patterns)
• Not a security issue or crash

Impact of Required Changes

Code changes needed:

1. Modify the overflow handling path in wrap_preserving_code to preserve leading whitespace when moving tokens to a new line
2. Either prepend preserved whitespace to the new line buffer or modify push_non_whitespace_span to have context-aware whitespace handling

Testing impact:

• Add regression test case specifically for this scenario
• Verify existing wrap tests still pass

Estimated scope: Small to medium - localised to the wrap/inline.rs and wrap/line_buffer.rs modules, but requires careful handling of whitespace semantics to avoid introducing new issues.