Tokenizer TS Web Browser Compatability

[Wolfleader101]

Hello,

The TS package doesnt work on web browser due to its usage of fs and path, is there any plans to support web browsers?

Cheers!

[shengyfu]

Sorry, web browser support is not our priority as of now, but feel free to create a PR if you really need it, the fs/path are only used to load the bpe model files, they can be replaced with http stream in browser setting.

[devm33]

@shengyfu Would you be able to take a look at #59? As you mentioned fs and path are only used if the bpe files are not provided. Deferring their imports until the time of use enables browser support and is a minimal change. Emulated the approach from the web-tree-sitter library.