Unicode segmenter performance

[clipperhouse]

In this PR, as an experiment I’ve swapped out the Unicode segmenter in Bleve with this one.

Benefits

• ~2x throughput improvement

Before
BenchmarkTokenizeEnglishText-8   	   17751	     65341 ns/op	   49528 B/op	      11 allocs/op

After
BenchmarkTokenizeEnglishText-8   	   34965	     32749 ns/op	   49528 B/op	      11 allocs/op

(Run on my M2 MacBook.)

• Updates Unicode 8 → Unicode 15

Testing & compatibility

• Both segmenters pass the official Unicode test suites
• Bleve tests pass
• Added adaptors to the underlying UAX29 package to determine Bleve token types
• Added fuzzing to the UAX29 package
• Known differences:
  • The original segementer splits runs of spaces into separate tokens; UAX29 concatenates runs of spaces into a single token. This should be irrelevant since Bleve filters whitespace in any case.
  • The original segmenter doesn’t handle emoji skin tone modifiers, the new one does. I believe the old tokenizer doesn’t handle Thai script, or at least numbers in that script.

[abhinavdangeti]

Thank you for submitting this @clipperhouse , we'll review it soon.

[clipperhouse]

Thanks @abhinavdangeti. I did some more testing of differences between segmenters here: https://github.com/clipperhouse/segmenter-repro

[clipperhouse]

Updated benchmark, using new sample text (~110K in size, multilingual). Note allocations.

Previous
BenchmarkTokenizeEnglishText-4   	     285	   4167394 ns/op	  26.15 MB/s	     16042 tokens	 1348515 B/op	     610 allocs/op

New
BenchmarkTokenizeEnglishText-4   	     614	   1986500 ns/op	  54.86 MB/s	     16042 tokens	 1310730 B/op	       2 allocs/op

[clipperhouse]

@abhinavdangeti I think this is in good shape for your review. Happy to discuss.

[clipperhouse]

I see that the go:build syntax isn’t compatible with old Go versions, fix incoming.

[clipperhouse]

@abhinavdangeti OK, try those workflows again? Thanks.

[clipperhouse]

I rebased a bit for a cleaner merge.

[clipperhouse]

(Rebased)

[clipperhouse]

@abhinavdangeti Friendly ping, let me know if you'd like to pursue this.

[clipperhouse]

Hiya @abhinavdangeti, the above pushes are just rebases, no updates here in a while. Would you like to run checks?

[abhinavdangeti]

Thanks @clipperhouse , re-running the checks.

[clipperhouse]

Looks good, thanks. Ready for review at your convenience.

[clipperhouse]

Hi @abhinavdangeti, I check on this every few months just to keep it fresh. If this is in your “someday” file, great. If not, feel free to close it.

Most recently, I removed my refactoring of the allocations, and just focused on a like-for-like swap of the segmenter. Simpler for review.