Add bracket detection in tokenize_inline to prevent merging adjacent links

[coderabbitai]

Issue Description

The tokenize_inline function in src/tokenize.rs does not properly handle adjacent links without spaces between them. The current while loop condition on line 105-107 only checks for whitespace and backticks, but should also check for bracket characters to prevent merging adjacent markdown links.

Current Behaviour

Adjacent links like [link1](url1)[link2](url2) may be incorrectly tokenised as a single text token instead of being recognised as separate link tokens.

Proposed Solution

Modify the while loop condition to include bracket detection:

while i < chars.len() 
    && !chars[i].is_whitespace() 
    && chars[i] != '`' 
    && chars[i] != '[' 
    && !(chars[i] == '!' && i + 1 < chars.len() && chars[i + 1] == '[') 
{
    i += 1;
}

Context

• File: src/tokenize.rs
• Lines: 105-107
• Function: tokenize_inline

References

• PR: Refactor wrap prefixes and doc fix #147
• Comment: Refactor wrap prefixes and doc fix #147 (comment)
• Reported by: @leynos

[leynos]

@coderabbitai looking at the codebase, is this issue still relevant? If the issue has been fixed in the code, does the documentation reflect this?

If it is still relevant, please provide an assessment of the issue criticality (critical, high, medium or low), based on risk of the issue impacting the user and the anticipated blast radius.

Once a decision has been made, apply the appropriate label.

If the issue is documentation ONLY (i.e., no code changes needed), apply the documentation label in addition to the appropriate criticality label.

Please also provide an impact of the required changes.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash

# First, check if src/tokenize.rs exists and examine the tokenize_inline function
echo "=== Checking src/tokenize.rs ==="
if [ -f "src/tokenize.rs" ]; then
    cat src/tokenize.rs
else
    echo "File not found"
fi

echo -e "\n=== Searching for tests related to adjacent links or tokenize_inline ==="
rg -n "adjacent.*link|tokenize_inline" --type rust

echo -e "\n=== Searching for test files that might test link tokenization ==="
fd -e rs test

echo -e "\n=== Looking for documentation files ==="
fd -e md . | head -20

Length of output: 757

---

🏁 Script executed:

#!/bin/bash

# Search for any test cases with adjacent links pattern
echo "=== Searching for test cases with link patterns ==="
rg -n "\[.*\]\(.*\)\[.*\]\(.*\)" --type rust -C 2

echo -e "\n=== Checking if there are any issues or TODOs related to bracket handling ==="
rg -n "bracket|adjacent.*link" --type rust -C 2

Length of output: 779

---

🏁 Script executed:

#!/bin/bash

# Read the actual tokenize file
echo "=== Reading src/wrap/tokenize.rs ==="
cat src/wrap/tokenize.rs

echo -e "\n=== Checking the test file for context ==="
cat src/wrap/tests.rs | head -50

Length of output: 16373

---

🏁 Script executed:

#!/bin/bash

# Check git log to see when the adjacent links test was added
echo "=== Git history for the adjacent links test ==="
git log --all --oneline --grep="adjacent" -- src/wrap/tests.rs

echo -e "\n=== Git log for changes to tokenize_inline function ==="
git log --all -p --oneline -S "tokenize_inline" -- src/wrap/tokenize.rs | head -100

echo -e "\n=== Check when the test case for adjacent links was added ==="
git log --all -p --oneline -- src/wrap/tests.rs | grep -A5 -B5 "link.*url.*another.*url2" | head -50

Length of output: 3917

---

🏁 Script executed:

#!/bin/bash

# Let's trace through what segment_inline actually produces for adjacent links
echo "=== Looking for tests that verify segment_inline output ==="
rg -n "segment_inline.*\[.*\]\(" --type rust -A 3

echo -e "\n=== Check the inline module to understand determine_token_span ==="
cat src/wrap/inline.rs | head -100

Length of output: 3936

---

🏁 Script executed:

#!/bin/bash

# Let's check when the bracket detection was added to segment_inline
echo "=== Git history of segment_inline function changes ==="
git log --all --oneline -- src/wrap/tokenize.rs | head -20

echo -e "\n=== Look for when the link parsing logic was added ==="
git log --all -p --oneline -S "parse_link_or_image" -- src/wrap/tokenize.rs | head -200

Length of output: 7281

---

🏁 Script executed:

#!/bin/bash

# Let's check the full git log to understand when the code was refactored
echo "=== Full git log for tokenize.rs ==="
git log --all --oneline -- src/wrap/tokenize.rs src/tokenize.rs

echo -e "\n=== When was PR #147 merged? ==="
gh pr view 147 --json number,title,state,mergedAt,createdAt,url

echo -e "\n=== When was issue #175 created? ==="
gh issue view 175 --json number,title,state,createdAt,url,body

echo -e "\n=== Let's look at the commit that introduced link handling ==="
git log --all --oneline --grep="link" -- src/wrap/tokenize.rs | head -10

Length of output: 2209

---

🏁 Script executed:

#!/bin/bash

# Check the actual implementation more carefully - look at the else branch in segment_inline
echo "=== Extract the else branch logic from segment_inline ==="
rg -A 5 "} else {" src/wrap/tokenize.rs | head -30

echo -e "\n=== Check if there are any open issues about adjacent links ==="
gh issue list --search "adjacent link" --json number,title,state

echo -e "\n=== Search for bracket in the code ==="
rg -n "ch != '\['" src/wrap/tokenize.rs

Length of output: 359

---

🏁 Script executed:

#!/bin/bash

# Let's look at the exact segment_inline implementation more carefully
echo "=== Extract segment_inline function with line numbers ==="
rg -n "pub\(super\) fn segment_inline" src/wrap/tokenize.rs -A 50

echo -e "\n=== Look for the else clause handling ==="
sed -n '220,240p' src/wrap/tokenize.rs

Length of output: 2901

---

🏁 Script executed:

#!/bin/bash

# Check when the link handling was added to segment_inline
echo "=== Check commit that added bracket/link detection ==="
git log --all -p --oneline -S "c == '['" -- src/wrap/tokenize.rs | head -150

echo -e "\n=== Check the PR/commit history around August 2025 ==="
git log --all --oneline --since="2025-07-30" --until="2025-08-05" -- src/wrap/tokenize.rs

Length of output: 5476

---

🏁 Script executed:

#!/bin/bash

# Let's trace through what segment_inline produces for different patterns
echo "=== Looking for integration tests that might test this ==="
rg -n "text\[link\]" --type rust

echo -e "\n=== Check if there are tests with text before links ==="
rg -n "segment_inline.*[a-z]+\[" --type rust -A 2

echo -e "\n=== Let's run a specific test to see behavior ==="
# Check the determine_token_span test more closely
cat src/wrap/tests.rs | head -80

Length of output: 2708

---

🏁 Script executed:

#!/bin/bash

# Let's write a small test to verify the behavior
echo "=== Creating a test script to check actual tokenization behavior ==="
cat > /tmp/test_tokenize.rs << 'EOF'
// Quick test to see what segment_inline produces
fn main() {
    // Simulated test cases
    let test_cases = vec![
        "[link](url)[another](url2)",
        "text[link](url)",
        "word[link](url)[another](url2)",
    ];
    
    println!("Test cases to verify:");
    for case in test_cases {
        println!("  Input: {}", case);
        println!("  Expected: separate tokens for each link");
        println!();
    }
}
EOF

cat /tmp/test_tokenize.rs

echo -e "\n=== Check what the scan_while closure actually does in the else branch ==="
rg -n "ch != '\`'" src/wrap/tokenize.rs -B 2 -A 2

Length of output: 788

---

🏁 Script executed:

#!/bin/bash

# Check if documentation mentions tokenization or link handling
echo "=== Checking documentation for tokenization behavior ==="
fd -e md | xargs rg -l "tokeniz|segment_inline" 2>/dev/null

echo -e "\n=== Check README for any mentions of link handling ==="
cat README.md | grep -i "link\|tokeniz" -A 3 -B 3 || echo "No mentions found"

echo -e "\n=== Check if there's any architectural documentation ==="
cat docs/architecture.md 2>/dev/null || echo "File not found"

Length of output: 13659

---

🏁 Script executed:

#!/bin/bash

# Let's verify the exact behavior by checking the logic flow more carefully
echo "=== Complete segment_inline function ==="
sed -n '173,232p' src/wrap/tokenize.rs

echo -e "\n=== Apply the medium label to the issue ==="
gh issue edit 175 --add-label "medium"

Length of output: 2346

---

@leynos, I've completed my analysis of this issue.

Issue Status: Still Relevant ✓

The issue described in #175 remains valid. The segment_inline function in src/wrap/tokenize.rs (lines 226-230) does not detect brackets in its else branch:

} else {
    let start = i;
    i = scan_while(&chars, i, |ch| !ch.is_whitespace() && ch != '`');
    tokens.push(collect_range(&chars, start, i));
}

This causes text like word[link](url) to be incorrectly tokenised as a single text token "word[link](url)" instead of separate tokens "word" and "[link](url)".

Why Current Tests Pass

The existing test on line 22 of src/wrap/tests.rs:

#[case("[link](url)[another](url2)", "[link](url)[another](url2)")]

passes because the links start at the beginning of the string. The loop checks c == '[' before entering the else branch, so adjacent links starting with brackets are handled correctly. However, no tests exist for links preceded by text without spaces (e.g., text[link](url)).

Documentation Status

The docs/architecture.md file documents link and punctuation handling but does not mention this tokenisation limitation.

Criticality Assessment: Medium

Justification:

• User Impact: Links preceded by text without spaces fail to tokenise correctly, causing them not to be recognised as links during text wrapping
• Frequency: Relatively uncommon pattern—most well-formed markdown includes spaces before links
• Blast Radius: Limited to inline text wrapping behaviour; does not affect tables, code fences, or other core functionality
• Workaround: Users can add a space before the link
• Correctness: This is valid markdown syntax that should be supported

Impact of Required Changes

Code Changes:

• Modify line 228 in src/wrap/tokenize.rs to add bracket detection as proposed
• Scope: Single line change to the scan_while condition
• Risk: Low—the change makes the else branch stop at brackets, allowing the loop to handle them in the next iteration

Test Changes:

• Add test cases for patterns like text[link](url) and word[link](url)[another](url2) to src/wrap/tests.rs
• Coverage gap: Current tests only verify links at string start

Documentation:

• No documentation updates required since this is a bug fix to support valid markdown, not a new feature

The fix is straightforward, low-risk, and addresses a legitimate parsing gap for valid markdown syntax.

[leynos]

Closed by #258