The `pk secrets env` command for meeting Development Environment Usecase

[CMCDragonkai]

Specification

1. The pk secrets env command should operate similar to the unix env command
2. We can use pk secrets env to do double duty: inject environment variables into a new subcommand, or allow sourcing of environment variables into an existing subshell
3. The task for the development environment usecase is specified in https://github.com/MatrixAI/Polykey-Desktop/issues/77
4. The pk secrets env someprogram arg1 arg2 should do a proper process replacement when possible

Additional context

• Exposing different interfaces for use of secrets within polykey Polykey#55 (comment) - old comment about the env interface and the "chording" idea

Copied excerpt:

Now when we want to start development on the project, we have 2 options:

1. Open up a shell with Polykey sourcing the environment variables
2. Use Polykey to inject environment variables into a sub-shell

Either result is the same, because you'll be running a shell with environment variables set so that your test runs of your software can inherit this environment configuration. Let's demonstrate both.

In the first case, we will reuse the source command and ask pk to constuct the equivalent of the .env file but entirely in-memory and only for this specific usage. We must also use process substitution <(...) so that the pk command output can be redirected into a temporary file descriptor that is read by the source command.

# one secret
source <(pk secrets env my-software-project:AWS_ACCESS_KEY_ID)

# multiple secrets
source <(pk secrets env my-software-project:AWS_ACCESS_KEY_ID my-software-project:GOOGLE_MAPS_API_KEY)

# globbing style (you can use globstar as well, this will only export immediate files)
source <(pk secrets env my-software-project:*)

# use the -e flag to export all variables
source <(pk secrets env -e my-software-project:*)
# use the -- to separate so you can export just one out of many
source <(pk secrets env -- my-software-project:AWS_ACCESS_KEY_ID -e my-software-project:GOOGLE_MAPS_API_KEY)
source <(pk secrets env -e -- my-software-project:AWS_ACCESS_KEY_ID my-software-project:GOOGLE_MAPS_API_KEY)

Now your shell has the relevant environment variables set by Polykey, and they will exist for as long as this current shell is alive.

In the second case, we ill use pk secrets env command to run a subshell, it can run any subprogram, but in the context of a development environment, you usually want a shell.

# one secret
pk secrets env my-software-project:AWS_ACCESS_KEY_ID bash

# multiple secrets
pk secrets env my-software-project:AWS_ACCESS_KEY_ID my-software-project:GOOGLE_MAPS_API_KEY bash

# globbing style
pk secrets env my-software-project:* bash

# the -e flag has no effect when using it this way
# this is because the subprogram determines whether to export variable or not
# it usually exports the variable
pk secrets env -e my-software-project:* bash

Now you have a subshell that has the environment variables configured, it will also automatically export it to child programs. When you have finished your development, you can just exit the shell, and the environment variables are gone!

Tasks

1. Ensure that pk secrets env can be used to inject environment variables and run a subprocess
2. Ensure that pk secrets env can be used to source environment variables and output something that can be used as a file to be sourced by a shell
3. Deal with the fact that nodejs doesn't have an exec that can replace the current node process: Ability to replace current Node process with another nodejs/node#21664
   • You can deal with this the way babel deals with it

• Babel attempts to use kexec where available, which is a POSIX-only module that replaces the process literally. Absent that, it falls back to its own implementation that works like the other two examples.

[CMCDragonkai]

Note that the kexec library has been abandoned but there's a recent PR: jprichardson/node-kexec#40 that updates it to be usable. We may need to bring in that source code into our own source tree to simplify usage if upstream doesn't update. This will introduce our own binary/C++ code here which can be useful to gain experience with writing native code in this form.

[scottmmorris]

So would the kexec.cc file need to be rewritten in JS for it to be usable inside js-polykey?

[CMCDragonkai]

No it would have to stay as a C++ file, cause it's native functionality. It's mostly about updating the C++ code so that it is compatible with our current node version.

…

On 7/14/21 1:05 PM, scottmmorris wrote: So would the |kexec.cc| file need to be rewritten in JS for it to be usable inside js-polykey? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/MatrixAI/js-polykey/issues/205#issuecomment-879549093>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4OHKVPQI6FCUD733G5KTTXT5JBANCNFSM5ACCXO5A>.

[CMCDragonkai]

At this point, the main points of review are:

• Globbing library
• The usage of kexec
• And UX of this library
• Testing of env command

Testing of the env subcommand requires a separate process context because eventually it should be using a fork exec pattern. That means it should be replacing the current process context. Because it's running inside a testing process, that would break the tests.

So we will need to use a variant of cli utility to run a subprocess. But the exact way in which this is done is a bit complicated.

The other challenge with testing this, is that if you are testing a subshell like bash or sh, you'll need https://github.com/nodejitsu/nexpect.

However if you write a subprogram like this:

#!/usr/bin/env sh

echo $INPUTVARIABLE

During the testing:

pk secrets env vault1:INPUTVARIABLE ./myscript.sh

The $INPUTVARIABLE will be output to STDOUT. This allows you test that the env variables are actually passed in.

The above script uses sh, sh is always available on Linux platforms. But on Windows, they don't exist.

There are also platform requirements. For example on Windows, you'll need to use a powershell script.

Let's put the platform specific testing into a separate issue later.

[CMCDragonkai]

Put on hold until we have client-refactoring merged.

[CMCDragonkai]

I found my old comment from last year talking about this: MatrixAI/Polykey#55 (comment). It also has a "chording" idea that may be relevant.

[CMCDragonkai]

Example of this being done for Hashicorp Vault: https://github.com/channable/vaultenv. Note that process replacement doesn't exist on windows. It will always be using a child process.