Skip to content

Document secrets#568

Merged
thaJeztah merged 2 commits into
docker:vnext-enginefrom
mdlinville:529_secrets
Jan 9, 2017
Merged

Document secrets#568
thaJeztah merged 2 commits into
docker:vnext-enginefrom
mdlinville:529_secrets

Conversation

@mdlinville
Copy link
Copy Markdown

Describe the proposed changes

Add docs for using secrets, addresses #529.

Unreleased project version

Engine 1.13

Related issue

#529

Related issue or PR in another project

see #529

Please take a look

@ehazlett @thaJeztah @mrjana @stevvooe and please add anyone else who should review.

@mdlinville mdlinville added this to the engine/1.13.0 milestone Nov 15, 2016
@mdlinville mdlinville changed the title 529 secrets Document secrets Nov 15, 2016
@mdlinville mdlinville changed the base branch from master to vnext-engine November 15, 2016 01:24
Comment thread _data/toc.yaml Outdated
title: Manage nodes in a swarm
- path: /engine/swarm/services/
title: Deploy services to a swarm
- path: /engine/swarm/store_sensitive_strings/
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not secrets?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I suppose.... I just don't think secrets means anything to anyone. :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Let's use sensitive data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: it looks like in the rest of the documentation there is a parenthetical "(secrets)" after "manage sensitive strings" - should that also be included here so folks can more easily find it in the TOC if they're specifically looking for the word "secrets"?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread engine/swarm/store_sensitive_strings.md Outdated
itself has already been propagated to the swarm node using mutual TLS.

>**Warning**: RAFT data is encrypted in Docker 1.13 and newer. If some of your
Swarm nodes run an earlier version, the secrets are stored unecrypted in
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the secrets may be stored unecrypted in those nodes' RAFT logs.

I'm not sure that we opaquely store secret data in older versions.

@diogomonica @cyli ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're in a situation where there is a 1.12 manager in a 1.13 cluster, and someone adds a secret, the 1.12 manager will write the secrets unencrypted to disk.

The upgrade path should be: upgrade all the managers to 1.13, and then start using secrets.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably add something about how to securely upgrade from a 1.12, or a note that to use secrets securely, make sure that all of the manager nodes are running 1.13

Copy link
Copy Markdown
Contributor

@stevvooe stevvooe Nov 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diogomonica While RAFT messages may be written to disk opaquely, I'm not sure that a manager with 1.12 snapshot structure definition will actually do this. None of our types have opaque field preservation, as far as I can tell. Looks like this was removed in proto3, as described in protocolbuffers/protobuf#272.

The upgrade path should be: upgrade all the managers to 1.13, and then start using secrets.

I agree and this language is sufficient to protect users against leaks, even if not precisely accurate. We may want to make this stronger and warn users to make sure the entire cluster is on 1.13 before performing cluster mutations.

The only data loss scenario @aaronlehmann and I could work out was if a 1.12 engine was elected leader in a mixed-version cluster. We may be able to downweight older engines in the master election process to minimize this possibility.

The other scenarios for data loss considered are 1.12 proxying to 1.13 leader, but this is a non-issue, since the JSON API prevents opaque field transfer.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only data loss scenario @aaronlehmann and I could work out was if a 1.12 engine was elected leader in a mixed-version cluster.

Or if that engine is upgraded to 1.13 and later becomes the leader. It will be missing the data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diogomonica While RAFT messages may be written to disk opaquely, I'm not sure that a manager with 1.12 snapshot structure definition will actually do this. None of our types have opaque field preservation, as far as I can tell. Looks like this was removed in proto3, as described in protocolbuffers/protobuf#272.

It might be written as a WAL.
Update: tested this manually - it is not written to the snapshot, but it is written to the WAL.

Comment thread engine/swarm/store_sensitive_strings.md Outdated
revoke its access to a given secret at any time.

The Engine managing a given worker node creates a bind mount within
`/run/secrets/` on the worker node for each secret. From the point of view of
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ehazlett /run/secrets runs afoul of FHS standards. Do we link to /var/run to /run?

From https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard:

In FHS 3.0, /var/run is replaced by /run; a system should either continue to provide a /var/run directory, or provide a symbolic link from /var/run to /run, for backwards compatibility.[10]

I think we just need to make sure that /run is not a fork of /var/run. Accessing /var/run/secrets or /run/secrets should be equivalent, recommending the later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes /var/run is linked to /run

@stevvooe
Copy link
Copy Markdown
Contributor

A few nits, but, otherwise, a great doc!

Copy link
Copy Markdown
Contributor

@diogomonica diogomonica left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall.

We're missing examples and a description of the advanced syntax for secrets, and more importantly, an example of how to rotate a secret.

Comment thread engine/swarm/store_sensitive_strings.md Outdated
@@ -0,0 +1,234 @@
---
title: Manage sensitive strings (secrets) for Docker services
description: How to securely store, retrieve, and use sensitive strings (secrets) in Docker services
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the use of strings is weird. what about sensitive data everywhere?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you store sensitive data that isn't a string?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String has a very specific meaning for developers and programmers. I think we want to avoid using that term. The term blob would technically be more adequate, but I think sensitive data reads a lot better.

Comment thread engine/swarm/store_sensitive_strings.md Outdated

## About secrets

In terms of Docker Swarm services, a _secret_ is a string, such as a password,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

string -> a blob of data

Comment thread engine/swarm/store_sensitive_strings.md Outdated
## About secrets

In terms of Docker Swarm services, a _secret_ is a string, such as a password,
credit card number, SSH private key, SSL certificate, or another piece of
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely not a credit card number. Lets remove that.

Comment thread engine/swarm/store_sensitive_strings.md Outdated
In terms of Docker Swarm services, a _secret_ is a string, such as a password,
credit card number, SSH private key, SSL certificate, or another piece of
sensitive data that should not be transmitted over a network or stored in a
Dockerfile or in your application's source code in plain text. Starting in
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

starting with?

Comment thread engine/swarm/store_sensitive_strings.md Outdated
Docker 1.13, Docker can encrypt and manage these secrets for you. Consider the
following examples when you might want to manage sensitive strings using Docker:

- Your service relies on a password-protected database, and you want to protect
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and you want to securely distribute the credentials to your application containers

Comment thread engine/swarm/store_sensitive_strings.md Outdated
itself has already been propagated to the swarm node using mutual TLS.

>**Warning**: RAFT data is encrypted in Docker 1.13 and newer. If some of your
Swarm nodes run an earlier version, the secrets are stored unecrypted in
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably add something about how to securely upgrade from a 1.12, or a note that to use secrets securely, make sure that all of the manager nodes are running 1.13

Comment thread engine/swarm/store_sensitive_strings.md Outdated
`/run/secrets/` on the worker node for each secret. From the point of view of
the container, the secret is available as plain text. The `/run/secrets/`
directory is an in-memory filesystem. When the node stops or leaves the swarm,
or when the service is stopped or removed, the secrets are no longer accessible.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add: and are completely removed from the node's memory.

Comment thread engine/swarm/store_sensitive_strings.md Outdated
## Read more about `docker secret` commands

Use these links to read about specific commands, or continue to the
[example about using secrets with a service](store_sensitive_strings.md@example-use-secrets-with-a-service).
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like store sensitive data instead of sensitive strings

Comment thread engine/swarm/store_sensitive_strings.md Outdated
When evaluating the value of the environment variable,
first check whether that value corresponds to a file path available within the
container read the credentail from the file if so. The `mistysj/wordpress` image
adds the following check to see if the value of `WORDPRESS_DB_PASSWORD` is a
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably document why we don't support environment variables in the first place. They are less secure than files.

See: moby/moby#9176 (comment)

Comment thread engine/swarm/store_sensitive_strings.md Outdated
- [`--secret`](../reference/commandline/service_create.md#create-a-service-with-secrets) flag for `docker service create`
- [`--secret-add` and `--secret-rm`](../reference/commandline/service_update.md#adding-and-removing-secrets) flags for `docker service update`

## Example: Use secrets with a service
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need an example for rotation of a secret (add a new secret, remove the old one, and ensure the new secret has the same target as the old one.

We should probably also give advanced examples on how to change the permissions of the secret file, and the use of source=bla,target=bla,uid=1000,gid=1000,mode=0700

@thaJeztah
Copy link
Copy Markdown
Member

Hm, for fun I tried adding a binary as secret, and that works 😄

$ docker run --name hola hello-world

$ docker cp hola:/hello .
$ cat hello | docker secret create hello
r4gmhiw6fmiorqfk5reqbyryr

$ docker service create --name foo --secret source=hello,target=hello,mode=0755 nginx:alpine


$ docker exec $(docker ps -n1 -q --filter label=com.docker.swarm.service.name=foo) /run/secrets/hello

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Should we document what size limits there are to secrets? (I don't think we put a size limit on there, but given that secrets are in memory, using many / big data there could have consequences perhaps?)

@ehazlett
Copy link
Copy Markdown
Contributor

Yes there is a size limit of 500 KB (https://github.com/docker/swarmkit/blob/master/manager/controlapi/secret.go#L22) per secret. We should document this.

@mdlinville
Copy link
Copy Markdown
Author

@diogomonica @ehazlett @stevvooe @cyli @toli PTAL. I think the topic is starting to shape up nicely. I addressed the feedback so far, and added a section on rotating secrets. Still waiting on an update to the official mysql image so that I can use that in the example instead of mysql/mysql-server.

@mdlinville
Copy link
Copy Markdown
Author

@stevvooe
Copy link
Copy Markdown
Contributor

LGTM

@mdlinville
Copy link
Copy Markdown
Author

Added info about binary content and 500kb limit.

Comment thread engine/swarm/secrets.md Outdated
different target file name within the container. The WordPress container
will use the mount point `/run/secrets/wp_db_password`. Also specify that
the secret is not world-readable, by setting the mode to `0400`.
- Sets the environment variable `WORDPRESS_DB_PASSWORD` to this file path. The
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WORDPRESS_DB_PASSWORD_FILE (correctly set in the example down below)

Comment thread engine/swarm/secrets.md Outdated
decrypted secret is mounted into the container in an in-memory filesystem at
`/run/secrets/<secret_name>`. You can update a service to grant it access to
additional secrets or revoke its access to a given secret at any time. When the
task container stops running, the secret is unavailable to that container. When
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: do you mean if you docker start the stopped container?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I was trying to cover the case where a node is running more than one task, each of which may have access to the same secret. When one of the containers stops, it doesn't have access to the secret anymore. If none of a node's task containers need access to a given secret anymore, I think the secret is flushed from the node as well, unless the node is acting as a backup master. Right? Is there a better way to put this?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your new change is great - I was wondering if you were specifically trying to call out doing a commit in a running container so it'd be available later, or restarting a container, or otherwise checking the contents of the stopped container on disk, etc.

Comment thread engine/swarm/secrets.md Outdated
will use the mount point `/run/secrets/wp_db_password`. Also specify that
the secret is not world-readable, by setting the mode to `0400`.
- Sets the environment variable `WORDPRESS_DB_PASSWORD` to this file path. The
`mistysj/wordpress` image is a fork of the official WordPress image which
Copy link
Copy Markdown
Contributor

@cyli cyli Nov 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example below uses the wordpress:latest image now (this comment refers to mistysj/wordpress). :) Thanks for getting the official image updated!

Comment thread engine/swarm/secrets.md

## Example: Rotate a secret

This example builds upon the previous one. In this scenario, you create a new
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"create a new secret with..."

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@mdlinville
Copy link
Copy Markdown
Author

Addressed @cyli feedback, tried to reword the area about when secrets get removed from a node's memory. PTAL at that section again and give me suggestions how I can improve it.

Copy link
Copy Markdown
Contributor

@diogomonica diogomonica left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this is looking better and better. We need

  • Another document that talks about lock/unlock/rotation that we link from this document when we talk about encryption
  • An easier example for secrets, for example (in one swarm manager cluster):
# docker secret create secret_data.txt
# docker service create --name redis --secret=secret_data.txt redis:alpine 
# docker ps
# docker exec -it redis cat /run/secrets/secret_data.txt

I think this would show exactly how this works faster than the MySQL example.

Comment thread engine/swarm/secrets.md Outdated
those containers that need access to it. A _secret_ is a blob of data that is
encrypted during transit and at rest in a Docker swarm. A given secret is only
accessible within those containers which have been granted explicit access to it,
and only while those containers are running.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we replace "accessible within those containers" to "accessible to those services"?

Other than that, beautiful.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread engine/swarm/secrets.md Outdated
runtime but you don't want to store in the image or in source control, such as:

- Usernames and passwords
- Database names
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

database credentials

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really did mean the database name, separately from the credentials (username and password). For instance, you can specify that as a file with the new wordpress image enhancements.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded a bit.

Comment thread engine/swarm/secrets.md Outdated

When you add a secret to the swarm, Docker sends the secret to the swarm manager
over a mutual TLS connection. The secret is stored in the RAFT log, which is
encrypted. The entire RAFT log is replicated across the other managers, ensuring
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably link here to another document talking about unlock and unlock-key, and how the key used for encryption is stored and how it can be managed/rotated/etc.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for #694 to be merged.

Comment thread engine/swarm/secrets.md Outdated
secret with the new MySQL password, update the `mysql` and `wordpress` services
to use it, then remove the old secret.

1. Create the new secret, and name it `mysql_password_2016`.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use mysq_password_v2 instead of 2016

Comment thread engine/swarm/secrets.md Outdated
## Build support for Docker Secrets into your images

If you develop a container that can be deployed as a service and accepts a
sensitive string, such as a credential, as an environment variable, consider
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and requires sensitive data, such as...

Comment thread engine/swarm/secrets.md Outdated
being passed directly.

>**Note**: Docker secrets do not set environment variables directly. This was
a conscious decision, since environment variables can leak between containers
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would do since it's easy for environment variables to be unintentionally leaked

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@cyli
Copy link
Copy Markdown
Contributor

cyli commented Nov 22, 2016

@mstanleyjones This looks great, thank you! I just wasn't sure if you were trying to emphasize something in particular, and this removes that confusion. Appreciate all your hard work on this! Other than @diogomonica's suggestion for lock and unlock, this all LGTM!

Comment thread engine/swarm/secrets.md Outdated
## Read more about `docker secret` commands

Use these links to read about specific commands, or continue to the
[example about using secrets with a service](secrets.md.md@example-use-secrets-with-a-service).
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be secrets.md@example-use-secrets-with-a-service?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably @ -> # as well?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@ehazlett
Copy link
Copy Markdown
Contributor

one minor comment otherwise LGTM -- nice work!

Comment thread engine/swarm/secrets.md Outdated
lwxgks2rojdbvaui5kay1suvr
```

The value returned is not the password, but the ID (digest) of the secret.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, sorry I missed this earlier. The ID is not the digest - the ID is opaque.

Comment thread engine/swarm/secrets.md Outdated

```bash
$ docker secret ls
ID NAME CREATED UPDATED SIZE
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, many apologies for springing this on you at the last minute, but moby/moby#28727 (comment) removes displaying sizes and digests at all when you inspect. Could you delete the last column in this sample output? (the size column?)

@mdlinville
Copy link
Copy Markdown
Author

Addressed latest feedback. Thanks @cyli !

@mdlinville
Copy link
Copy Markdown
Author

Modified the instructions in the mysql lab now that docker-library/mysql#237 is merged and the mysql image has been updated!

Comment thread engine/swarm/secrets.md Outdated
The secret is stored in the encrypted RAFT logs for the swarm.

4. Create the MySQL service. The `mysql` image now supports
additional environemnt variables which read their values from a file within
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"environment" mispelling :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, non-blocking nitpick - this reads to me like the environment variable reads values from a file? Whereas it's the script within the image which reads the file pointed at from the environment variable. I'm not sure the best way to say this in a clear way, but maybe something like "additional environment variables, which lets mysql read secret values from a file..."

That may make the next sentence a bit redundant though.

@cyli
Copy link
Copy Markdown
Contributor

cyli commented Nov 23, 2016

@mstanleyjones Thank you! I have a minor spelling/phrasing nitpick, but everything looks great! 👍 LGTM!

@cyli
Copy link
Copy Markdown
Contributor

cyli commented Dec 21, 2016

@thaJeztah Also, 👍 on the separate wordpress and mysql users!

@thaJeztah
Copy link
Copy Markdown
Member

wow your shell-fu is WAY better than mine!

"Learn by example" - I looked into some of @tianon's scripts to see how it's done 👑

Also, in the revised example, should we mention that the service update to mysql causes the container to restart and thus would cause a disruption in the wordpress service?

Probably good to mention.

My goals were, first of all, to avoid having password in plain text on-disk. I know a lot of work has gone into the secrets feature to avoid just that, and ending up with writing to plain text files in the documentation didn't feel "right".

At first, I wanted to avoid the docker exec as a whole, and spin up a mysql container to use as client (which is the "neat" way to do it). Unfortunately, we don't have secrets yet on docker run, so that didn't work.

As a follow up to this PR, we should do some thinking about how to actually use the secrets in various situations. For example, the WordPress image still sets the secrets through environment variables, so a stray phpinfo() page (yup expect those to be in a lot of setups) will happily show the password;

screen shot 2016-12-22 at 01 20 50

Preventing that needs changes to the configuration files of images, so we may want some examples for that.

@cyli
Copy link
Copy Markdown
Contributor

cyli commented Dec 22, 2016

My goals were, first of all, to avoid having password in plain text on-disk. I know a lot of work has gone into the secrets feature to avoid just that, and ending up with writing to plain text files in the documentation didn't feel "right".

Yep, don't disagree, although in theory it's on your local machine, so less terrible. It can also be stored in a password manager such as the lastpass CLI, which would also let you pipe it into secrets, but wasn't sure if that'd be too much for an example.

At first, I wanted to avoid the docker exec as a whole, and spin up a mysql container to use as client (which is the "neat" way to do it). Unfortunately, we don't have secrets yet on docker run, so that didn't work.

Can we use a service with --restart-condition=none?

docker service create \
  --name=mysql_password_rotation \
  --network=wpnet \
  --secret=mysql_password \
  --secret=mysql_password_v2 \
  --restart-condition=none \
  mysql \
  bash -c 'mysqladmin --host=mysql --user=wordpress --password="$(< /run/secrets/mysql_password)" password "$(< /run/secrets/mysql_password_v2)"'

I'm not sure if we're starting to get into weird service esoteric stuff, now, though :)

As a follow up to this PR, we should do some thinking about how to actually use the secrets in various situations. For example, the WordPress image still sets the secrets through environment variables, so a stray phpinfo() page (yup expect those to be in a lot of setups) will happily show the password

Ah good catch. :| I didn't realize it'd be part of the process's environment - I'd only checked mysql's, which didn't seem to be. cc @tianon

@thaJeztah
Copy link
Copy Markdown
Member

I'm not sure if we're starting to get into weird service esoteric stuff, now, though :)

definitely taking it too far. we can revisit once (if) secrets are available for docker run

Ah good catch. :| I didn't realize it'd be part of the process's environment - I'd only checked mysql's, which didn't seem to be.

Was already discussing with him; we could probably unset the env-vars directly after they have been consumed. Help me remind, I can try a pull request (or bake Tianon some cookies for christmas)

@mdlinville
Copy link
Copy Markdown
Author

Pushed some updates. This puts back the mysql service and also uses @thaJeztah magical way to update the password in MySQL. The only thing it doesn't do yet is use a non-root user for the database. Need to figure out how to do that.

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some hints what needs to be changed to make the wordpress user work

Comment thread engine/swarm/secrets.md Outdated
--mount type=volume,source=mydata,destination=/var/lib/mysql \
--secret source=mysql_password,target=mysql_password \
-e MYSQL_ROOT_PASSWORD_FILE="/run/secrets/mysql_password" \
mysql:latest
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need two secrets here; one for the root password and one for the wordpress user;

docker service create \
  --name mysql \
  --replicas 1 \
  --network=mysql_private \
  --mount type=volume,source=mydata,destination=/var/lib/mysql \
  --secret=mysql_root_password \
  --secret=mysql_password \
  -e MYSQL_ROOT_PASSWORD_FILE="/run/secrets/mysql_root_password" \
  -e MYSQL_PASSWORD_FILE="/run/secrets/mysql_password" \
  -e MYSQL_DATABASE=wordpress \
  -e MYSQL_USER=wordpress \
  mysql

here;

  • MYSQL_ROOT_PASSWORD_FILE sets the path to the secret for the root password
  • MYSQL_PASSWORD_FILE sets the path to the secret for the regular user's (wordpress) password
  • MYSQL_DATABASE sets the name of the database to create the first time the service is started
  • MYSQL_USER sets the name of the user to create the first time the service is started

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure to use the right secret for the right user.

Comment thread engine/swarm/secrets.md Outdated
--publish 30000:80 \
--secret source=mysql_password,target=wp_db_password,mode=0400 \
-e WORDPRESS_DB_PASSWORD_FILE="/run/secrets/wp_db_password" \
-e WORDPRESS_DB_HOST="mysql:3306" \
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to add WORDPRESS_DB_USER here, to set the name to use for connecting to the MySQL database;

-e WORDPRESS_DB_USER=wordpress \

Comment thread engine/swarm/secrets.md Outdated
--replicas 1 \
--network mysql_private \
--publish 30000:80 \
--secret source=mysql_password,target=wp_db_password,mode=0400 \
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a volume here, so that the WordPress data is preserved when updating the service;

--mount type=volume,source=wpdata,destination=/var/www/html \

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What WordPress data is that? I thought it stored everything in the MySQL database....

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The WordPress installation (theme, uploads, configuration, etc)

Comment thread engine/swarm/secrets.md Outdated
```bash
$ docker service rm wordpress

$ docker rm -f mysql
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will be docker service rm mysql, or combine with the previous one (docker service rm wordpress mysql)

also, the named volumes can be cleaned up afterwards

@mdlinville
Copy link
Copy Markdown
Author

OK, the latest update uses a separate wordpress database, user, and secret. I still need to set the the MySQL root password, but it's not used after MySQL starts up.

This is getting very close to being ready to go, IMHO. I guess the only outstanding concern is the WordPress environment variable, right @thaJeztah ?

@mdlinville
Copy link
Copy Markdown
Author

BTW I don't need to use WORDPRESS_DB_USER because it defaults to wordpress.

@thaJeztah
Copy link
Copy Markdown
Member

I guess the only outstanding concern is the WordPress environment variable

That's something to be addressed in the official WordPress image; I haven't had time to look into that yet, but it's orthogonal to this PR

BTW I don't need to use WORDPRESS_DB_USER because it defaults to wordpress.

Oh, yes, it probably defaults to that; doesn't hurt to be explicit, but I'm also fine to skip setting it

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some last bits and nits, but I think we're almost there

Comment thread engine/swarm/secrets.md Outdated
l1vinzevzhj4goakjap5ya409
```

The value returned is not the password, but the ID (digest) of the secret.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit; remove (digest) here; the ID of secrets have no relation to the content of the secrets (otherwise they may potentially "leak" the content).

Comment thread engine/swarm/secrets.md Outdated

- Because the scale is set to `1`, only a single MySQL task runs.
Load-balancing MySQL is left as an exercise to the reader and involves
than just scaling the service.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing "more" before "than"

Comment thread engine/swarm/secrets.md Outdated
initializing the system database for the first time. Afterward, the
passwords are stored in the MySQL system database itself.
- Sets environment variables `MYSQL_USER` and `MYSQL_DATABASE`. A new
database called `wordpress` is created when the image starts, and the
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "when the image starts" -> "when the container starts"

Comment thread engine/swarm/secrets.md
wvnh0siktqr3 mysql replicated 1/1 mysql:latest
```

At this point, you could actually revoke the `mysql` service's access to the
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread engine/swarm/secrets.md Outdated
Rotating passwords or other secrets will often involve additional steps outside
of Docker.

1. Create the new password and store it in file in the `tempdir` directory
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove "and store it in file in the ..." part, as it's no longer needed

Comment thread engine/swarm/secrets.md Outdated

```bash
$ docker service update \
--secret-rm mysql_password mysql
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something we need to fix; doing --secret-rm and --secret-add should be possible in a single update

Comment thread engine/swarm/secrets.md Outdated
First, find the ID of the `mysql` container task.

```bash
$ docker ps | grep mysql |awk {'print $1'}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to use the built-in filter functionality here (and -q to only show the ID;

docker ps -q --filter name=mysql

Comment thread engine/swarm/secrets.md Outdated
Verify that the blog post you wrote still exists, and if you changed any
configuration values, verify that they are still changed.

6. Revoke accsss to the old secret from the MySQL service and
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo accsss

@mdlinville
Copy link
Copy Markdown
Author

Fixed latest problems found by @thaJeztah and also got a review from my friend @bskaggs as well. He spotted things I could no longer see, I've been looking at this too long!

Comment thread engine/swarm/secrets.md Outdated
```bash
$ docker service rm wordpress mysql

$ docker volume rm mydata
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to add wpdata here as well now

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left one quick nit, and possibly someone else may want to have a final look, because i'm now probably blind as well 😄

@mdlinville
Copy link
Copy Markdown
Author

@thaJeztah nit fixed, and a few more picky formatting tweaks. I think this is ready to merge but I'd like input from @cyli @diogomonica @NathanMcCauley

Comment thread engine/swarm/secrets.md Outdated
`service update` command redeploys the service.

```bash
$ docker exec -it <container_id> cat /run/secrets/my_secret_data
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: since we had the example above of docker exec $(docker ps --filter name=redis -q) cat /run/secrets/my_secret_data, can we use that same command here instead of <container_id>?

Comment thread engine/swarm/secrets.md Outdated

### Intermediate example: Use secrets with a Nginx service

> **Note**: This example uses a single-Engine swarm for simplicity, and only
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Since all of these examples (redis, nginx, and wordpress) presume a single-Engine swarm, should that part of this note be moved to the top of the examples section? (Similarly for the wordpress example)

Comment thread engine/swarm/secrets.md Outdated

> **Note**: Normally you would create a Dockerfile which copies the `site.conf`
> into place, build the image, and run a container using your custom image.
> This example does it all in one step.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like "This example does not require a custom image; it copies the site.conf into place and runs the container all in one step"?

Comment thread engine/swarm/secrets.md Outdated
yvsczlx9votfw3l0nz5rlidig mysql_root_password 12 seconds ago 12 seconds ago
```

The secrets are stored in the encrypted WAL logs for the swarm.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Should we just say "Raft logs" here, since above we mention that secrets are stored in the raft logs? It's true that they are written to the WAL (and snapshot, later on), but that might be too much raft detail for this document, particularly since we never say what WAL stands for.

Comment thread engine/swarm/secrets.md
> a file on disk. You must use a query or a `mysqladmin` command to change the
> password in MySQL.

1. Generate a random alphanumeric password for MySQL and store it as a Docker
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention that this first password is for the wordpress user for mysql? Or at least a non-root user?

Comment thread engine/swarm/secrets.md Outdated
queries or commands, as opposed to just changing a single environment variable
or a file, since the image only sets the MySQL password if the database doesn’t
already exist, and MySQL stores the password within a MySQL database by default.
Rotating passwords or other secrets will often involve additional steps outside
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nonblocking nitpick: should this be "may"? If it's something simple like an API key the service just access, just changing the service spec should be fine. Alternately, this is likely always the case with databases, so maybe we can call out DBs specifically here?

Comment thread engine/swarm/secrets.md Outdated
`/run/secrets/mysql_password`.

Even though the MySQL service has access to both the old and new secrets
now, the MySQL root password has not yet been changed.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't change the root password at all in this example, right? Could we clarify this example to say that we don't need to adjust the service's access to the MySQL root password, since it's not being modified?

Comment thread engine/swarm/secrets.md Outdated
Even though the MySQL service has access to both the old and new secrets
now, the MySQL root password has not yet been changed.

3. Now, change the MySQL password for the `wordpress` user using the `mysql`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"mysqladmin CLI", as opposed to "mysql CLI"

Misty Stanley-Jones added 2 commits January 4, 2017 11:37
Fixes #529

Signed-off-by: Misty Stanley-Jones <misty@docker.com>
@mdlinville
Copy link
Copy Markdown
Author

Addressed @cyli feedback, and also pushed a second commit which updates the syntax of the -f flag to docker create, as per moby/moby#29889.

@cyli
Copy link
Copy Markdown
Contributor

cyli commented Jan 5, 2017

LGTM!

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still LGTM

we can update the syntax once moby/moby#29955 was decided on.

Thanks again, @mstanleyjones !

@thaJeztah thaJeztah merged commit 2c26e6a into docker:vnext-engine Jan 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants