Improve tokenization docs

README.md

@@ -25,7 +25,6 @@ cargo install --path .
 
 ## Command-line usage
 
-
 ```bash
 mdtablefix [--wrap] [--renumber] [--breaks] [--ellipsis] [--fences] [--footnotes] [--in-place] [FILE...]
 ```
@@ -111,8 +110,8 @@ A brief intermission for pizza.
 
 ## Library usage
 
-The crate exposes helper functions for embedding the table-reflow logic in
-Rust projects:
+The crate exposes helper functions for embedding the table-reflow logic in Rust
+projects:
 
 ```rust
 use mdtablefix::{process_stream_opts, rewrite, Options};
@@ -159,8 +158,9 @@ For an overview of how the crate's internal modules relate to each other, see
 
 ## Testing
 
-The test suite is structured using the `rstest` crate. See [Rust testing with
-rstest fixtures](docs/rust-testing-with-rstest-fixtures.md) for details.
+The test suite is structured using the `rstest` crate. See
+[Rust testing with rstest fixtures](docs/rust-testing-with-rstest-fixtures.md)
+for details.
 
 ## License
 

src/wrap.rs

@@ -118,7 +118,24 @@ fn tokenize_inline(text: &str) -> Vec<String> {
     tokens
 }
 
-/// Tokenise Markdown into fences, inline code and plain text.
+/// Split the input string into [`Token`]s by analysing whitespace and
+/// backtick delimiters.
+///
+/// The tokenizer groups consecutive whitespace into a single
+/// [`Token::Text`] and recognises backtick sequences as inline code spans.
+/// When a run of backticks is encountered the parser searches forward for an
+/// identical delimiter, allowing nested backticks when the span uses a longer
+/// fence. Unmatched delimiter sequences are treated as literal text.
+///
+/// ```rust,ignore
+/// use mdtablefix::wrap::{Token, tokenize_markdown};
+///
+/// let tokens = tokenize_markdown("Example with `code`");
+/// assert_eq!(
+///     tokens,
+///     vec![Token::Text("Example with "), Token::Code("code")]
+/// );
+/// ```
 pub(crate) fn tokenize_markdown(input: &str) -> Vec<Token<'_>> {
     let mut out = Vec::new();
     let mut in_fence = false;