Fixing the last deviations from sentencepiece indicated by test-tokenizer-1

[goerch]

The last deviations are fixed now, too.

We don't seem to need Unicode normalization immediately.

[cebtenzzre]

I would like a more complete understanding of why the API change is needed. The fact that sentencepiece accepts it isn't good enough - does the text parameter represent a UTF-8 string, or something else?

[goerch]

I would like a more complete understanding of why the API change is needed. The fact that sentencepiece accepts it isn't good enough - does the text parameter represent a UTF-8 string, or something else?

As far as I understand it, 0x00 is a valid Unicode point which converts to 0x00 in UTF8 and vice versa.

[cebtenzzre]

As far as I understand it, 0x00 is a valid Unicode point which converts to 0x00 in UTF8.

Yes, but it is a non-printable control character. As far as POSIX is concerned, text files must not contain NUL bytes - therefore, text should not contain NUL bytes.

[goerch]

As far as I understand it, 0x00 is a valid Unicode point which converts to 0x00 in UTF8.

Yes, but it is a non-printable control character. As far as POSIX is concerned, text files must not contain NUL bytes - therefore, text should not contain NUL bytes.

The small print says: 'Although POSIX.1-2017 does not distinguish between text files and binary files (see the ISO C standard)...'

[cebtenzzre]

If the POSIX description isn't sufficient, what about MIME types?

$ printf 'hello\n' >text.bin
$ printf 'hello\0\n' >not_text.bin
$ file -i text.bin
text.bin: text/plain; charset=us-ascii
$ file -i not_text.bin
not_text.bin: application/octet-stream; charset=binary

[goerch]

@cebtenzzre : I believe that the NUL character is improbable and badly supported by software around. But this PR is aiming to improve sentencepiece compatibility for a specific test case. So either we work with this PR or change the test case. How do you propose I should change the test case? Ignore token 3 and code point 0? Then I can simply change the test case and retract all changes to the llama.cpp kernel.

[slaren]

I am not sure why POSIX is relevant here. Is there any reason to believe that the tokenizer needs to respect any part of the POSIX spec? I think this is simpler that, if the sentencepiece tokenizer can encode NUL characters, but our implementation can't, then it is a bug in our implementation and it should be fixed. Supporting non-NUL terminated strings seems the natural way to do it.

[cebtenzzre]

Okay, I'll defer to your judgement in this case.