allow a deeper integration with nushell

[fdncred]

As a maintainer of nushell, I just want to say that we love the effort of uutils/coreutils. Good work y'all and please keep up this excellent work!

We love it so much that we really want to have a much tighter integration than just calling external executables. I myself, have started to experiment by using the uu_cp and uu_mv crates as internal nushell commands in this PR nushell/nushell#9463.

There are some rough edges that we'd need to work together on and I've tried to document different strategies in my PR. Without going into a lot of detail I'll just list a few things.

• Nushell doesn't use clap so working with command line arguments and the uu Options structs is interesting. Right now, I've kind of hacked it together where nushell checks params, then uu checks params, which isn't ideal, but it's a start.
• I've experimented with calling a command's uumain directly and passing args to it (and that can work), or rolling our own type of uumain. The benefit with rolling-our-own is we get the expected nushell "pretty" error messages. Error handling in general will need help and I'm not quite sure what to do here, specifically in regards to passing back a Span so we know where the error is.
• Streaming. Some uu_ commands stream output as output is generated, this is great and we'd need a way to hook into this output in order to consume it as nushell data structures that generate the nushell tabular or record output. I'm not really sure what is appropriate here.

In general, as you might guess, some things in coreutils may need to be exposed publicly, like command Options, to enable others, like nushell, to utilize them, and other things may need a different level of surgery in nushell or in coreutils.

One thing is for sure, we'd love to partner up and find a way forward because, let's be honest, dealing with some of these file-io and other commands is just difficult, and it would just be great to have one source of truth that works cross-platform like the uutils/coreutils.

[amtoine]

as a maintainer of Nushell myself, i totally support the initiative of @fdncred here 😊

[sylvestre]

Sure, let's work together to make this happen!

[dmatos2012]

As a small small contributor of both nushell and coreutils, Id love to see this happen and would like to help in making this possible. :)

[fdncred]

@dmatos2012 The easiest way for you (or anyone else that is interested) to help is to start playing with my nushell PR listed above. If you checkout my PR and compile it with cargo build --features=nuuu then cp and mv will be using the uutils version. You can play around and see that some mv syntax does not work like mv a b c test/. It works in uutils mv but nushell doesn't allow it. This type of testing starts to reveal changes that will need to be made somewhere.

The current task is to really to find a way forward. e.g. How do we handle spans, which nushell requires for errors? How do we handle parameter parsing? How do we handle streaming.

I started with uu_cp because there's not a lot of output with copying. I figured it may be an easy place to start. Then I went to uu_mv, which quickly revealed syntax that I think just needs to be added to nushell.

[tertsdiepraam]

I've been thinking about this in the background and looked at nushell/nushell#9463. Here's what I think we should do:

Create a list of suitable utils for nu

Not all utils are suitable for nu. For example, using the uutils ls over the nu ls would probably be a bad idea. I like the start of mv and cp. In general, I think the utils that are suitable are:

• relatively complex,
• have very little output or the output is not the main function of the util,
• do not rely on bash-isms.

For example:

• Suitable: b2sum (and similar), cp, dd, rm
• Unsuitable: test, ls, sort, true, false

There is a grey area of utils which have some significant output, but are relatively simple, such as wc. This could potentially be supported by creating some wordcount crate extracted from uutils to be shared between the projects.

Refactor those utils to expose their functionality

The nu cli interface is better than anything uutils could ever provide, so nu should do the argument parsing in my opinion. I think using uumain (like in nushell/nushell#9463) is probably also more complicated than necessary and might introduce subtle bugs. I think we should instead provide an interface that exposes the internals of the util directly:

pub fn cp(opts: CpOptions) -> UResult<()> { ... }

This can then be used by nu.

Ensure that uutils does not exit or panic

uutils has largely been written to be run as a separate process and has used std::process::exit quite freely in the past. We've been reducing this dramatically, but it's not yet fixed. There's also the occasional panic. Both of these would exit the nu process, which is of course unacceptable. The refactored utils should be checked to ensure that does not happen.

Non-goal for now: streaming output

If we focus on utils that are not focused on output, we can ignore the issue of generating streamed structured output from uutils. I'd love to tackle this in the future, but it would be very complicated. So, I think it's easier to focus on the quite large collection of other utils first.

I'd love to hear what you think of this!

[fdncred]

Thanks @tertsdiepraam. I appreciate your thoughtfulness. Here's some of my thoughts.

I generally agree with this approach, which is why I started with cp. JT likes to say, "crawl, walk, run" which is basically what your proposal is. Start with the low hanging fruit and then look at the next lowest, and so on. So, yes, please. :) I'd love to move forward with this. I'm happy to just start with cp as a proof-of-concept. It seemed pretty straight forward.

Another thing that I started playing around with was porting your tests into nushell. As far as I'm concerned, the more tests, the better, especially when bridging platforms like this. It was not trivial for the couple tests I moved over. I'd like to somehow just dump the tests your community has created for individual commands. We may have to think a little more on that front.

I do know that with the mv command, there is some functionality that didn't work out-of-the-box, or said another way, nushell may have to change the way parameters are setup now for mv. IIRC, it was mv file file file dir. Nushell should support that, but the current code did not, so we'll have to investigate if this is a simple change or a more complex one.

Exposing uutils Options is a must. Agreed. If nushell is handling the command line parsing then we'll have the spans for "pretty" errors.

Exit/Panic - Ya, that would be bad but it seems like it's a known issue that we can continue to work on.

ls - I'm very keen to use your ls and stream the output somehow with a callback Fn or something. We have permission issues sometimes and those are just a pain to figure out because it's hard to reproduce them. I'm fine with waiting though. Another one that we have permission issues with is cd. I'm not sure I saw a uutils cd command though.

I'll take a look through the commands and throw a list together of ones that I'd consider the "crawl" stage, the post it here, but like I said, let's just plan on cp being first and let's just do one command. We just had a release, 0.83.1, so we have ~3 weeks until the next release 0.84.0. It would be cool to have uu_cp inside of nushell by then. If we can't do it by then, that's no biggy though.

[tertsdiepraam]

Excellent! I love the "crawl, walk, run" saying!

Let's start with cp then. I think that it would be quite easy on the uutils side. We could just expose:

coreutils/src/uu/cp/src/cp.rs

Line 214 in f37318d

pub struct Options {

and

coreutils/src/uu/cp/src/cp.rs

Line 1132 in f37318d

fn copy(sources: &[Source], target: &TargetSlice, options: &Options) -> CopyResult<()> {

and the types used in Options.

(Source and TargetSlice are just aliases for PathBuf and Path respectively)

I can add some additional documentation to those types and maybe clear up the naming a bit and then the rest would be up to you.

[fdncred]

I took a swing at bucketing the commands into crawl, walk, run here https://docs.google.com/spreadsheets/d/1MP9V8BTtcy_3DIUT8025yTcGrjkIJ-ZKFT0uvJFw0Lo/edit#gid=0

I guessed at a bunch of them and some may be bucketed wrong, but it's a start.

[tertsdiepraam]

I've made a PR to expose most of the necessary types of cp for use in nu: #5152. Feedback is welcome!

[dmatos2012]

Now, building up on my comment on the PR, I have some pain points specifically for us to solve here. @fdncred. You can check my WIP branch building on yours (very ugly code now), to see some of updates.

• Particularly pain point for me was parsing as GNU wants. I agree with @tertsdiepraam that we should not behave exactly like GNU. Specific problems:

• Some GNU flags have only a short flag, but no long flag, and with it their behavior changes, so makes it very difficult to implement. For instance, I made it so that cp -u a b would work by having u being a NamedArg. So in this case -u is None and it uses a default. But if you do cp -u none a b ,-u is none, but if you do cp a b -u it wouldnt work since Nu would complain about "String is required for argument".

• And as far as I am concerned, I dont think nu allows for no long flag, but only short flag, so we have to get around that as well.

• GNU is quite specific with how it uses flags. So short flag, cp -u a b implies an specific update, update=older, but if you use the long flag, you use it differently cp --update=older a b (where = is required). Right now in my impl I have just parsed and made it work by doing cp --update older a b and cp -u older a b (without the = for simplicity). Therefore, I think we need to make a decision on how we are going to support the arguments, and what is allowed or not. Otherwise, it would become a headache to get the behavior exactly using the nushell parser.

• Finally there is the problem of setting some Options that are private in uutils, such as Attribute that would be quite cumbersome for us to implement, so we need to keep working with them to make this fully functional and painless for both.

Once we define those, I think I could try to port more tests and improve the code, and add some docs.

Hopefully we can keep working together, to get close to a fully functional nushell using the cp from uutils :)

[tertsdiepraam]

Excellent observations! Thanks @dmatos2012!

Pedantic correction: in GNU, -u and --update are not separate but only the long version allows an optional argument. I think optional arguments in general might be a bit difficult to support in the nu parser.

One workaround is to create flags like --update-older, --update-all & --update-none and then make -u the short version of --update-older. Though that is a bit ugly I suppose.

An example of a short argument that does not have a long counterpart at all is -H. But there we could just come up with a long name for that.

As for Attributes, I think the solution there is to extract all of the parsing to a single function in uutils and expose that to avoid duplication.

For the first version, I think we should simplify as much as possible (e.g. no attributes or other complicated flags).

Edit: Maybe SyntaxShape could help for --update? Would something like this work?

SyntaxShape::OneOf(vec![
    SyntaxShape::Keyword(
        vec![b'='],
        SyntaxShape::String,
    ),
    SyntaxShape::Nothing,
])

[fdncred]

@dmatos2012 Great write up. Thanks for your effort! I like your "ugly" code, mine would've looked worse. 😆

I agree with your points and @tertsdiepraam has some interesting thoughts for work-arounds. One comment I'll throw in is no one said we have to support every parameter in cp (or any other command). We can approach cp with the crawl, walk, run strategy as well, if we want to. In my MVP comment in the code, I was only supporting a handful of parameters. Having said that, I would prefer to support all the parameters, but I'd like to leave our options open for now.

As you brought up, we also need to determine how compatible we want to be with the GNU flags. I'm not sure myself. I want people to feel at home but nushell is already pretty different. I'm not convinced it's important that we be 100% compatible with GNU flags. I think if we have a way to do the same thing but with a different way, perhaps like --update-older, then that's not a bad way to go. We'd probably just want to include something in the help to explain that.

I'm going to try to get some other nushell core-team members to chime in here about these points. This is the "devil in the details" I was sure that we'd come across. There will be other problems with other commands like mv, I'm sure.

You can't tell but I'm bubbling over with excitement for this integration!