Standardise STDOUT output for CLI commands

[emmacasolin]

Specification

Our new commandments for a better, brighter, command line:

• Send output to stdout. The primary output for your command should go to stdout. Anything that is machine readable should also go to stdout—this is where piping sends things by default.
• Send messaging to stderr. Log messages, errors, and so on should all be sent to stderr. This means that when commands are piped together, these messages are displayed to the user and not fed into the next command.
• When we are displaying output text to indicate success, we need to be brief. Feedback is always preferred, even though deviating from UNIX conventions.
• The output from stderr can be redirected to /dev/null, but we're going to provide more verbosity options that suppresses all non-essential text (feedback, warnings, etc.).
• We should ALWAYS be using camelCase for anything going to stdout. As we prioritize parsability, this needs to be the default. No more manually outputting strings like Name: amy.
• Nested structures should always be avoided in dicts formats wherever possible. Instead, the nested structure should be spread into the data.
• Streaming with the JSON format should output the objects in JSONL format. To parse it with jq, one needs to use --slurp

STDOUT

Traditionally, we have constructed human readable dictionaries as such:

process.stdout.write(
  binUtils.outputFormatter({
    type: options.format === 'json' ? 'json' : 'list',
    data: [`Root certificate:\t\t${response.getCert()}`],
  }),
);

This is clunky, inconsistent, worse for parsability and more...

In all cases where the output is "semantically" a list, the output should use either json or list. In all other cases, the list option should be replaced with dict. This should be done like this:

process.stdout.write(
  binUtils.outputFormatter({
    type: options.format === 'json' ? 'json' : 'dict',
    data: { rootCertificate: response.getCert() },
  }),
);

The data parameter must be kept as close to the return value of the RPC call possible. This ensures that our output is predictable. Furthermore

STDERR

Log messages, errors, and so on should all be sent to stderr. This means that when commands are piped together, these messages are displayed to the user and not fed into the next command.

Most STDERR messages should either be using the raw or list format. This is because they are intended to be human readable. Under json format, the data should always be { message: "..." }.

process.stderr.write(
  binUtils.outputFormatter({
    type: 'list',
    data: ['Message 1'],
  }),
);

process.stderr.write(
  binUtils.outputFormatter({
    type: 'raw',
    data: 'Message 1',
  }),
);

process.stderr.write(
  binUtils.outputFormatter({
    type: 'json',
    data: { message: "Message 1" },
  }),
);

STDIN

STDIN should always be checked for interactivity before asking prompts. This means that if a Polykey Client session has not already been opened, we should error before asking for a password prompt on a non-interactive terminal.

If input or output is a file, support - to read from stdin or write to stdout. This lets the output of another command be the input of your command and vice versa, without using a temporary file. For example:

cat secret.txt | polykey secret create - vault:secret

Piping

There is currently issues with piping input into Polykey-CLI: #139

TBD...

Additional context

• General CLI review issue: Publishing the PK CLI for General Use Polykey#268
• https://clig.dev/#output

Tasks

1. Ensure that the output formatter can correctly format output for all possible format options without modifying content
2. Replace all usage of the list format option with dict (except in situations where the output is more semantically suited to a list)
3. Ensure that there are no tabs or spaces in json output (as this needs to be machine readable)
4. Replace usage of the output formatter with just outputting a single string in cases where this is appropriate
5. Implement -q argument to suppress non-essential output.
6. Review commands for new specification changes

• Agent - 7/7
• Bootstrap - 1/1
• Identities - 13/13
• Keys - 12/12
• Nodes - 6/6
• Notifications - 3/3
• Secrets - 12/12
• Vaults - 12/12

[CMCDragonkai]

I was thinking that for commands that output a stream of data, the human readable formats don't currently have anything separating each "unit" of output. That is, for dict, we might see something like:

pid	3497062
nodeId	vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g
pid	3497062
nodeId	vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g
pid	3497062
nodeId	vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g

Which represents 3 units of output.

In JSON, this is distinguished simply by the newline:

{"pid":3497062,"nodeId":"vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g"}
{"pid":3497062,"nodeId":"vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g"}
{"pid":3497062,"nodeId":"vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g"}

So for the human readable output, it may be a good idea to have a double line separation between each unit. The end result should look like:

pid	3497062
nodeId	vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g

pid	3497062
nodeId	vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g

pid	3497062
nodeId	vng8qsa39osoqrj3o1ftnbgig80drej2on6fok35jvsm5457up72g

And this would be similar for other kinds of outputs like list:

a
b
c

a
b
c

a
b
c

But to do this, you'd have to add \n after each process.stdout.write, but only when it's not json output, since json output now already has a \n appended afterwards in the output formatter, and double lines is not valid for jsonlines format convention.

Something like this:

for (const _ of [1,2,3,4]) {
  process.stdout.write(binUtils.outputFormatter({ ... }));
  if (options.format !== 'json') process.stdout.write('\n');
}

To make this more robust, we may add a new output formatter async generator (consumer), that takes input and makes these decisions automatically rather than having to put it in each command.

[CMCDragonkai]

According to MatrixAI/Polykey#446 (comment) our dictionary formatter doesn't support recursive dictionaries.

I think we need to special case handling of object values, that is if they don't have a proper toString format, then we should be providing a recursive outputting of that object.

It could work through simply calling the outputFormatter again on that.

[CMCDragonkai]

Actually to handle the additional indentation, one has to keep track of how many levels of indentation required as a prefix to each line. This may require some additional bookkeeping. A sort of indentation counter.

However if the object has a custom toString defined, we just use that. See: https://stackoverflow.com/a/47903498/582917 for the distinction. That way if a custom object has a toString() then it can be used directly.

So generally we would be sending POJOs to the output formatter.

[CMCDragonkai]

This issue is to be moved to Polykey-CLI.

[addievo]

table format needs an update for different types of values

For instance for this value:

NodeId vq480ofld7vrol0ura1viif9474pb7ptrqm9s8ld23vn5t92oif4g, Address 127.0.0.1:55551, bucketIndex 249

The table output, outputs like this:

Private Zenhub Image

Which is not ideal.

[CMCDragonkai]

Prefer to upload images to GH, not zenhub, otherwise the uploads could break.

You should see that table output should always take an array of dictionaries. Then the initial keys of the first dictionary should be used as the columns.

It should be tab separated as per CLI standardisation between each cell.

[CMCDragonkai]

There is a bit of a problem with maintaining padding though, if you have to loop over all records to find out the maximum size of each value, which is a bit annoying.

Basically I would want the CLI output to be a valid TSV (tab separated values).

However I'm not sure about tabs and newlines within the data field.

See: http://dataprotocols.org/linear-tsv/ and https://github.com/eBay/tsv-utils/blob/master/docs/comparing-tsv-and-csv.md

[CMCDragonkai]

I think a quoted version of TSV would be suitable. If the field ends up having a tab or newline, it would be wrapped in quotes and printed out with \t or \n. Similar to how the values were printed in the current agent status.

[CMCDragonkai]

Imagine:

first	second	third
"abc\t\nfoo"	123	lol

Of course you can see that even using tabs is misaligned. Usually the CLI tool column is supposed to be piped to understand it.