refactor!: vendor the tokenizer stack into lance

[Xuanwo]

This PR vendors the tokenizer stack Lance actually uses into a new rust/lance-tokenizer crate and rewires FTS and inverted-index code to depend on it instead of tantivy and lindera-tantivy. It keeps the existing document and query tokenization semantics in-tree, renames the old FTS document adapter module to document_tokenizer, and preserves upstream license headers on vendored code.

[codecov]

Codecov Report

❌ Patch coverage is 74.57405% with 194 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-tokenizer/src/lindera.rs	0.00%	56 Missing ⚠️
rust/lance-tokenizer/src/jieba.rs	0.00%	39 Missing ⚠️
rust/lance-tokenizer/src/ascii_folding_filter.rs	74.22%	23 Missing and 2 partials ⚠️
rust/lance-tokenizer/src/ngram_tokenizer.rs	82.53%	21 Missing and 1 partial ⚠️
rust/lance-tokenizer/src/stemmer.rs	68.25%	20 Missing ⚠️
rust/lance-tokenizer/src/stop_word_filter.rs	76.36%	13 Missing ⚠️
rust/lance-index/src/scalar/inverted/tokenizer.rs	55.55%	7 Missing and 1 partial ⚠️
rust/lance-tokenizer/src/alphanum_only.rs	87.50%	3 Missing ⚠️
rust/lance-tokenizer/src/analyzer.rs	92.85%	3 Missing ⚠️
...lance-index/src/scalar/inverted/tokenizer/jieba.rs	0.00%	2 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

[BubbleCal]

Awesome work!