Discrepancy in output length with special characters

[sylvestre]

With

echo "mÃ" > /tmp/a

With our implementation:

$ cat /tmp/a| ./target/debug/coreutils wc  -L -l
      1      2

With GNU:

$ cat /tmp/a| /usr/bin/wc -L -l  
      1      1

Found with the wc differential fuzzer

[cakebaker]

I can't reproduce this issue locally. GNU wc returns the same result as uutils wc:

$ echo "mÃ" > /tmp/a
$ cat /tmp/a | /usr/bin/wc -L -l
      1       2

[sylvestre]

try with a different local ?
like with LANG=C ?

[cakebaker]

I tried LC_ALL=C, LANG=C and LC_COLLATE=C, but no difference. And based on the doc of -L I would expect it to return 2 (and not 1) because there are two characters.

[BenWiederhake]

@sylvestre Do you still have the file somewhere? I suspect that this is just an encoding issue, because I can't reproduce it either. For reference, here's my /tmp/a:

$ hexdump -C /tmp/a
00000000  6d c3 83 0a                                       |m...|
00000004

[RenjiSann]

I can reproduce:

$ echo "mÄ" | ../gnu/src/wc -L -l      
      1       1
$ echo "mÄ" | ./target/debug/wc -L -l 
      1       2

Also, a similar error is happening with -m (number of characters).

$ echo "mÄ" | ./target/debug/wc -m -c
      3       4
$ echo "mÄ" | ../gnu/src/wc -m -c      
      4       4

# Without the `\n` added by echo
$ echo -n "mÄ" | ./target/debug/wc -m -c
      2       3
$ echo -n "mÄ" | ../gnu/src/wc -m -c 
      3       3

[rwdj]

Confirming an issue with both -L and -m:

$ echo "mÄ" > /tmp/f
$ hexdump -C /tmp/f
00000000  6d c3 84 0a                                       |m...|
00000004
$ LC_ALL=C cargo run -q -- wc -mL /tmp/f
3 2 /tmp/f
$ LC_ALL=C /usr/bin/wc -mL /tmp/f
4 1 /tmp/f

Although still a bug, it does appear encoding-related:

$ LC_ALL=en_US.UTF-8 cargo run -q -- wc -mL /tmp/f
3 2 /tmp/f
$ LC_ALL=en_US.UTF-8 /usr/bin/wc -mL /tmp/f
3 2 /tmp/f

[rwdj]

To be more specific, it's a matter of if bytes and multibyte chars are the same which depends on locale.

$ LC_ALL=en_US.UTF-8 cargo run -q -- wc -cm /tmp/f
3 4 /tmp/f
$ LC_ALL=C cargo run -q -- wc -cm /tmp/f
3 4 /tmp/f
$ LC_ALL=en_US.UTF-8 wc -cm /tmp/f
3 4 /tmp/f
$ LC_ALL=C wc -cm /tmp/f
4 4 /tmp/f

In libc, this is detected by checking the length of a multibyte char is equal to a byte (locale-dependent): MB_CUR_MAX.
It appears uutils's wc uses utf8::BufReadDecoder thereby assuming always UTF-8.