Inconsistency between tr and cut on multibyte character

[jyte]

Hi,

I downloaded the last release and have a strange behaviour.

I have the following test.txt file (utf8 encoded)

zone1§zone2§zone3

$ ./coreutils cut -d§ -f2 test.txt
zone2
$ cat test.txt | ./coreutils tr '§' ';'
zone1;;zone2;;zone3

the cut command behave as I would expect (although the output is different from the gnu cut)
However, tr should not double ;

Basically, it seems to work fine when tr replace n byte(s) character by another n byte(s) character, but is broken if I try to replace a n byte(s) character by a m byte(s) character.

[jadijadi]

When testing with GNU coreutils 9.4, the tr behaves as we do:

cat /tmp/test2.txt | tr '§' ';'
zone1;;zone2;;zone3

Same as us:

cat /tmp/test2.txt | cargo run tr '§' ';'
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/coreutils tr '§' ';'`
zone1;;zone2;;zone3

The question is if we have to keep the compatibility with GNU coreutils or the consistency with our cut and as we expect from a UTF-8 char.

Side note: GNU breaks if § used as a separator.

cut -f2 -d§ /tmp/test.txt
cut: the delimiter must be a single character
Try 'cut --help' for more information.

[tavianator]

tr is treating § as two separate characters, basically $'\xc2\xa7' (the UTF-8 encoding of §). As an extension, the replacement character ; is repeated to match the length of the pattern string, becoming ;;. So the command is basically saying "replace $'\xc2' with ; and $'\xa7' with ;. I think this behaviour is not a bug.

[jyte]

From tr manual :

tr - translate or delete characters

For tr I guess it all depends on what you put behind a "character".

But on a broader thinking, my understanding is that those tools are primarily made to parse and process text. If my language prints text with characters coded on single and/or multibytes such as α β or γ for greek, or e é for french, I should be able to process text with such characters without even having to wonder on how many bytes the character is coded.

I agree that historically, one character = 1 byte = 8 bits, and we had differents ASCII tables to code different characters for various languages. This means that by using Latin-1 charset system, I have

$ printf 'aébéc' | cut -dé -f2
b
$ printf 'aébéc' | tr 'é' 'e'
aebec

But now, on UTF-8 systems, I have the following behaviour

# GNU utils
$ printf 'aébéc' | cut -dé -f2
cut: the delimiter must be a single character
Try 'cut --help' for more information.
$ printf 'aébéc' | tr 'é' 'e'
aeebeec

# uutils
$ printf 'aébéc' | ./coreutils cut -dé -f2
b
$ printf 'aébéc' | ./coreutils tr 'é' 'e'
aeebeec

From my point of view, switching from Latin-1 to UTF-8 break GNU tools in some cases, and it needs fixing.

Another example, from cut manual

  -b, --bytes=LIST        select only these bytes
  -c, --characters=LIST   select only these characters

So, we clearly have a difference between a byte and a character. Let's see what happens...

$ printf 'aébéc' | cut -b1
a
$ printf 'aébéc' | cut -b2
�

This is fine since we cut on byte index.

$ printf 'aébéc' | cut -c1
a
$ printf 'aébéc' | cut -c2
�

Ok, so now we cut on character index and... Oh, same result as a byte cut (on both uutils and GNU utils). Why 2 different parameters if we expect the same processing in both cases ? I believe this is a bug and should output : é

And as a final nail in the coffin, to argue that GNU is broken and needs fixing and that uutils should not accept GNU breakage as the target behaviour, here is the macos BSD utils behaviour on UTF-8

$ printf 'aébéc' | cut -dé -f2
b
$ printf 'aébéc' | tr 'é' 'e'
aebec
$ printf 'aébéc' | cut -b1
a
$ printf 'aébéc' | cut -b2
?
$ printf 'aébéc' | cut -c1
a
$ printf 'aébéc' | cut -c2
é

Which is pretty damn fine on all account.

[tavianator]

Hmm actually looking at the POSIX spec for tr, everything is defined in terms of "characters". In POSIXese, a character is

A sequence of one or more bytes representing a member of a character set.

Note

This term corresponds to the ISO C standard term multi-byte character, where a single-byte character is a special case of a multi-byte character. Unlike the usage in the ISO C standard, character here has no necessary relationship with storage space, and byte is used when storage space is discussed.

See the definition of the portable character set in 6.1 Portable Character Set for a further explanation of the graphical representations of (abstract) characters, as opposed to character encodings.

So I've change my mind. tr should work on multi-byte characters, and I think GNU tr is non-conforming to POSIX in locales with multi-byte characters.

[tavianator]

This is in fact a known issue with GNU tr: https://unix.stackexchange.com/a/228570/56202

[jyte]

should a dedicated issue be opened for cut -c ?