Incorrect position check in IsUtf8?

[pedasmith]

It looks like a lot of the UTF8 checker code on GitHub is all copied with what appears to be the same issue: if the utf8 multi-byte sequence is at the end of the buffer, it's incorrectly marked as invalid. AFAICT, all of the checks for position >= length -2 (or whatever) should be position > length-2. In most projects, the offending file is utf8checker.cs (but in your case, I think it's in program.cs)

[jmalarcon]

Hi Peter:

Thanks for taking the time to write about this, even when my quick and dirt utility program is written and documented in Spanish.

I see what you mean. The fragment of the code that I'm using in the main program to heuristically detect UTF8 is not writetn by me. As I tell in the comments, it's based in utf8checker. AFAIK the code is based on the Unicode indications here and passes the test data sources indicated in the Codeplex page. That was the reason to choose it in the first time, long ago, but probably has some errors, I don't know.

Do you know about any good C# Open Source implementation that gets everything right?
If you could point me to one it'll be helpful and I'd update the code in this repo.

Or do you think that the only thing needed is to change the >= for > and that's it?

Thanks in advance!

[pedasmith]

FYI: I've just created a near drop-in IsUtf8 replacement; that project includes a bunch of unit tests.

[jmalarcon]

Thank you!

I'll take a look at it and fix my tool.