Discussion: Non-Ascii test cases.

[Insti]

While updating the anagram test cases (issue: #413)
Discussion of handling non-ascii characters came up, and we decided that we would NOT use non-ascii characters in those tests.

@NobbZ made a good point:

I do not think, that we should test for anything that is not in ASCII.

1. Most languages do not cope very well with codepoints beyond ASCII
2. There are letters beyond ASCII, that would need another step of normalization. Eg the german "es-zett" (ß) which is is only allowed as lowercase (but there is a capital version available which is allowed to be used on titlepages and headlines), on capitalisation it is usually turned into "SS". There has been the long-s, and similar characters in ancient greek (gamma was one of them AFAIK).

Isogram (as of 20161031) also has non-ascii test cases.

Are there other problems that have non-ascii test cases?

I've created this issue so we can discuss the general policy of whether non-ascii characters should be used in test cases and have a a thread to point to when it comes up again in the future.

Proposal:
All test cases should only use ASCII characters
(Unless extended character handling is integral to the problem.)

[rbasso]

Aye!

I believe that the same logic can be applied to a broader class of questions:

Q: Should we test for ...?
A: No! Unless that is a fundamental part of the problem.

Anytime we testing for something that is nothing fundamental, we reduce diversity in the solutions and it gets boring to review code. 😁

[ErikSchierboom]

Agreed. I think having the non-ASCII test in Isogram does not really add anything. In fact, I think most people will be confused by it. If someone could come up with another exercise in which non-ASCII character handling does make sense, I'm all for it. But for the current exercises, I think it should be removed.

[NobbZ]

I do like to have some of non US-ASCII test cases as an optional part for a very small subset of exercises. They can teach you a lot of unicode-handling if and only if you are open to it and want to learn it.

Anagram is not one of them, since it requires you to normalize charakters and some of them can't be normalized. In the example of the german ß vs. SS, is MASSE an anagram of Maße? How is it the other way round?

Word-Count on the other hand does gain a lot by adding the Unicode sugar to separate words from each other. But, well, the normalisation problem persists, but its influence is by far not that strong.

[behrtam]

I would agree that non-ASCII test cases don't make sense in every exercise, especially if they add unnecessary complexity for example if normalization is needed.
But we should add unicode character where ever it makes sense. It is 2016, emojis have conquered the world and not everyone can stay in the US bubble. People need to learn how to deal with unicode at some point in the tracks.

[Insti]

It is 2016, emojis have conquered the world and not everyone can stay in the US bubble. People need to learn how to deal with unicode at some point in the tracks.

I agree, this is why there need to be specific exercises that deal with multi-language text handling, and all the gotchas and edge cases involved in that. But that shouldn't be required in an exercise that is about the algorithm for detecting anagrams.

[petertseng]

Looks like we're going sort of Unix philosophy with many exercises - do one thing and do it well. This is why we are cutting Unicode from a few text exercises. I welcome the move!

For the time when that new Unicode exercise(s) solidifes, I have a list of things I've seen over the months that could stand to go in it:

• Consider creating new Unicode exercise go#200 (because of ledger: handling unicode go#195)
• Potential new test in anagram: Same bytes different characters. #318
• Add test case to Hamming that shows the difference between len() and chars().count() rust#128

[petertseng]

Are there other problems that have non-ascii test cases?

We can try git grep -Pnl "[\x80-\xFF]" to find them, though I admit this can have false positives. But at least it shouldn't have false negatives... right?

Edit: Actually, it may not have false positives either...

For me, this has found:

exercises/atbash-cipher/canonical-data.json
exercises/bob/canonical-data.json
exercises/forth/canonical-data.json
exercises/isogram/canonical-data.json
exercises/pangram/canonical-data.json
exercises/run-length-encoding/canonical-data.json
exercises/scrabble-score/canonical-data.json