Warn that Chars iterator does not iterate "characters"

[kornelski]

The Chars iterator iterates over Unicode Scalar Values, but when people think about "characters" they usually mean something closer to what Unicode calls "grapheme clusters".

This leads to surprising results and thus potential errors, for example:

"éé".chars().count() == 3
"🇺🇸".chars().count() == 2

I suggest adding a warning to the documentation that this iterator isn't iterating over "characters", and that users should consider using UnicodeSegmentation::graphemes iterator instead.

[steveklabnik]

Rust's char type is a unicode scalar value, and its documentation does currently say 'codepoints', explicitly: http://doc.rust-lang.org/stable/std/primitive.str.html#method.chars

We also document these differences in the book: http://doc.rust-lang.org/stable/book/strings.html#indexing

[kornelski]

The documentation is technically correct, but I think insufficient to prevent users from falling into a trap.

The difference between Unicode codepoint and a character is subtle and tricky, and to make it worse, in most cases (and all examples in Rust documentation) it appears to be the same thing.

The documentation shows only simple cases where one codepoint is one grapheme cluster, which—along with the name of the char type—could lead users to wrongly assume that codepoints are characters.

I could make pull requests for the docs adding explanatory notes to chars iterator, chars method and adding tricky cases to the book. Do you think that would be a good addition?

[steveklabnik]

Yeah, I guess what I meant was "here's what we have, what specifically could improve?"

A PR would be a great way to workshop it :)

[steveklabnik]

@pornel are you still persuing a PR, or should I write something?

[kornelski]

Sorry, I've been a bit ill. It's still on my todo list.

[steveklabnik]

It's all good! Feel better. I have other work I can do, was just curious.