OS interaction and unicode

[Leont]

To quote perlunicode:

One reason that Perl does not attempt to resolve the role of Unicode in these situations is that the answers are highly dependent on the operating system and the file system(s). For example, whether filenames can be in Unicode and in exactly what kind of encoding, is not exactly a portable concept. Similarly for qx and system: how well will the "command-line interface" (and which of them?) handle Unicode?

• chdir, chmod, chown, chroot, exec, link, lstat, mkdir, rename, rmdir, stat, symlink, truncate, unlink, utime, -X
• %ENV
• glob (aka the <*>)
• open, opendir, sysopen
• qx (aka the backtick operator), system, readdir, readlink

Given we're handling readdir now, I'm wondering if we shouldn't also handle most other interfaces with the OS. Not sure we can actually do that for all of these though, some don't overload cleanly.

Also, this may be material for a separate module that's used by utf8::all, instead of being direct part of it.

[doherty]

Yes, we should try to handle all of that "appropriately" (whatever that means) -- and I don't see a particular reason to do it outside this module. Not sure I have the time or ability to pull it off, however.

[HayoBaan]

• chdir, chmod, chown, chroot, exec, link, lstat, mkdir, rename, rmdir, stat, symlink, truncate, unlink, utime, -X
  I don't think these need to be changed as they do not return strings and take their input just fine.
• %ENV
  This one is tricky. While we certainly can update it to have UTF-8 decoded strings, this will result in a global change, and may break other code. I haven't yet found a way to localise the changes to the calling module only…
• glob (aka the <*>)
  Done as per my contribution Added: utf-8 aware glob, File::Find and Cwd::* #24
• open, opendir, sysopen
  See comment on chdir etc.
• qx (aka the backtick operator), system, readdir, readlink
  • readdir – is done,
  • system – is not necessary (see comment on chdir etc.)
  • readlink is likely doable,
  • qx may be interesting as its output may need decoding.

[doherty]

"Fix all the things" is not a very useful bug report.

@HayoBaan: Would you open a bug for each thing or closely-related-set-of-things with a quick description of what's needed? When you're done, please close this issue with a list of the individual issues you created.

[HayoBaan]

@doherty Actually from my comment it boils down to just:
readlink (which is easy to do) and possibly qx / backtick (though I currently wouldn't know how to replace this with our own version…)
The rest is either done/not necessary or (in the case of %ENV), way to dangerous to implement.

[HayoBaan]

So with readlink out of the way: if anyone has an idea how to override the default code for the qx and backtick operator, I would like to hear it so we can implement these as well.

Also a solution for manipulating %ENV without affecting non utf8::all code would be welcome :-)

[Leont]

So with readlink out of the way: if anyone has an idea how to override the default code for the qx and backtick operator, I would like to hear it so we can implement these as well.

That is easy, you can override readpipe in much the same way you override readlink and readdir :-)

Also a solution for manipulating %ENV without affecting non utf8::all code would be welcome :-)

I can think of ways, but none of them is appealing (e.g. ties…)

[HayoBaan]

So with readlink out of the way: if anyone has an idea how to override the default code for the qx and backtick operator, I would like to hear it so we can implement these as well.

That is easy, you can override readpipe in much the same way you override readlink and readdir :-)

Awesome, why didn't I think of that earlier? (I do now remember reading about the base of qx and the backtick, but couldn't bring it to memory, thanks for reminding me). Perhaps I can have this one done soon too now! 😎

Also a solution for manipulating %ENV without affecting non utf8::all code would be welcome :-)

I can think of ways, but none of them is appealing (e.g. ties…)

I had been thinking in that direction too, but didn't like the idea either 😞

[HayoBaan]

Hmm, there seems to be a little snag when overriding the readpipe; after you've done that, the arguments to qx and the backtick operator aren't interpolated any longer.

Instead of ls -1 corpus | sort the argument the overridden readpipe gets is $cmd.
Is there perhaps some compiler hint that needs to be set to have it interpolate???
(the readpipe tests, of course works as expected and does interpolate the arguments).

Forget this, see next comment 😉

[HayoBaan]

Ok, guys, I thought I'd do one other test and you're going to laugh now (I know, I did!) 😃
It turns out we had been trying to do something that wasn't necessary at all:
Without doing any readpipe override, the results from the qx, backtick, and readpipe itself already give the expected result (i.e., get decoded/encoded as specified with utf8:all). As it turns out, this is the result of the use open qw(:encoding(UTF-8) :std); that we implement…
I'll add this nice fact to the documentation and I will also add the readpipe testcase just to make sure 😉

So that leaves us with just the evil of %ENV 😄

[Leont]

It turns out we had been trying to do something that wasn't necessary at all

Awesome!

[doherty]

Good news! :D

I'll open a new issue for %ENV and close this one.

[Leont]

chdir, chmod, chown, chroot, exec, link, lstat, mkdir, rename, rmdir, stat, symlink, truncate, unlink, utime, -X

I don't think these need to be changed as they do not return strings and take their input just fine.

That is not quite true. Currently they take internal encoding, they should take character when utf8-all is enabled. It will do the wrong thing (from an u-a perspective) when a string is in octet-mode instead of utf-8 mode.