fix: split trailing punctuation in sentence mode (#21)

[Alex-Wengg]

Summary

• Fixes Punctuation behind number caused number not processed #21 — trailing punctuation glued to a word ("eight,") used to block the cardinal/aviation taggers in sentence mode, producing wrong output like "United 780 eight," instead of "United 788,".
• Adds a pretokenize helper that peels leading/trailing ASCII punctuation (, . ; : ! ? ( ) [ ] { } ") into separate pretokens while remembering each token's original leading separator, so output spacing matches input exactly.
• Factors the three duplicated sliding-window loops (normalize_sentence_inner, tn_normalize_sentence_with_max_span, tn_normalize_sentence_with_max_span_lang) into a single shared sentence_loop that consumes pretokens.

Behavior

Input	Before	After
United seven eighty eight, please ...	United 780 eight, please ...	United 788, please ...
United seven eighty eight. please ...	United 780 eight. please ...	United 788. please ...
United seven eighty eight , please ...	already worked	United 788 , please ... (unchanged)
I have (twenty one) apples!	I have (twenty one) apples!	I have (21) apples!
don't eat twenty one apples	don't eat 21 apples	don't eat 21 apples (unchanged)

Apostrophes and hyphens are intentionally excluded from the split set so contractions (don't) and hyphenated words (twenty-one) stay intact.

Test plan

• cargo fmt
• cargo build --features ffi
• cargo test --features ffi — full suite passes (lib + en/de/fr/es/zh/ja/hi/lang/extensive/ffi/doc)
• 4 new regression tests in tests/en_tests.rs:
  • test_issue_21_trailing_comma
  • test_issue_21_trailing_period
  • test_issue_21_space_before_punct_preserved
  • test_issue_21_other_punctuation (parens, quotes, contractions)

---

[devin-ai-integration]

Devin Review found 1 potential issue.

View 5 additional findings in Devin Review.

[devin-ai-integration]

🔴 Pretokenizer breaks TN whitelist abbreviations ending with '.' in sentence mode

The pretokenize function splits trailing periods from whitespace tokens (e.g., "Dr." → ["Dr", "."]). In sentence_loop, pretokens are then joined with spaces to form spans for the parser (.join(" ")), so the parser sees "Dr ." instead of "Dr.". This causes two classes of TN sentence-mode regressions:

1. Entries with only the period form (e.g., "e.g.", "etc.", "Prof.", "Inc.", "approx." — ~25 entries in the English TN whitelist alone): the abbreviation is never matched. "e.g." → pretokens ["e.g", "."] → single-pretoken span "e.g" has no whitelist hit, two-pretoken span "e.g ." also has no hit → abbreviation passes through unnormalized.

2. Entries with both forms (e.g., "Dr"/"Dr.", "vs"/"vs."): the bare form matches, but the period remains as a separate pretoken, producing a spurious period: "Dr. Smith" → "doctor. Smith" instead of "doctor Smith".

This affects all 7 TN language whitelists (en, fr, es, de, zh, hi, ja) in tn_normalize_sentence, tn_normalize_sentence_with_max_span, and tn_normalize_sentence_with_max_span_lang.

Trace for "e.g." through pretokenize + sentence_loop

word = "e.g." → chars [('e',0), ('.',1), ('g',2), ('.',3)]
start scan: 'e' not punct → start=0
end scan: chars[3]='.' is punct → end=3; chars[2]='g' not punct → stop
Core: word[0..3] = "e.g", trailing: "."
Pretokens: ["e.g", "."]

In sentence_loop:
Span [0..2]: "e.g" + " " + "." = "e.g ." → WHITELIST.get("e.g .") → None
Span [0..1]: "e.g" → WHITELIST.get("e.g") → None (only "e.g." is in the whitelist)
Fallback: output "e.g" then ".", no normalization occurs

Prompt for agents

The root cause is that sentence_loop joins all pretokens with spaces when constructing spans for the parser. This destroys the original adjacency of split-off punctuation: pretokens ["Dr", "."] from the word "Dr." become span "Dr ." instead of "Dr.", breaking TN whitelist matching.

The fix needs to reconstruct the original text when building spans, not unconditionally insert spaces. The Pretoken.sep field already records the original separator ("" for adjacent/split pretokens, " " for whitespace-separated). The span construction at src/lib.rs:1115-1119 could use sep values for pretokens after the first:

  pretokens[i..end].iter().enumerate().fold(String::new(), |mut s, (j, p)| {
      if j > 0 { s.push_str(p.sep); }
      s.push_str(&p.text);
      s
  })

However, this would make the span include the glued punctuation (e.g., "twenty one,"), which was the original problem the pretokenizer aimed to fix. A more nuanced approach is needed, such as:

1. Reconstruct spans using sep (preserving adjacency) so parsers see original text like "Dr." or "e.g.", and ALSO try the sub-span without trailing punctuation pretokens so "twenty one" can still be tried without the comma.

2. Alternatively, make the pretokenizer TN-aware: skip splitting when the full token matches a whitelist entry.

3. Or exclude '.' from is_split_punct and handle trailing periods differently (e.g., the parser strips trailing periods before matching, or the sentence_loop retries without the last character on failure).

All TN language whitelists in src/tn/{en,fr,es,de,zh,hi,ja}/whitelist.rs are affected. Affected public APIs: tn_normalize_sentence, tn_normalize_sentence_with_max_span, tn_normalize_sentence_with_max_span_lang.

---

Was this helpful? React with 👍 or 👎 to provide feedback.