The tokenize_markdown function in src/lib.rs at line 231 collects all characters of the input text into a Vec<char>, which allocates memory proportional to the text length. For very large markdown files, this could cause high memory usage.
Consider refactoring to use lazy iteration over characters instead of collecting them into a vector.
References:
The
tokenize_markdownfunction insrc/lib.rsat line 231 collects all characters of the input text into aVec<char>, which allocates memory proportional to the text length. For very large markdown files, this could cause high memory usage.Consider refactoring to use lazy iteration over characters instead of collecting them into a vector.
References: