Remove secrets from Docker Images by gdbelvin · Pull Request #1191 · google/keytransparency

gdbelvin · 2019-02-07T14:19:12Z

Private keys should not be part of the docker images.
This was an old hack that predated docker's secrets feature.

TODO / Help wanted: add a kubernetes configmap that does the same thing.

gdbelvin · 2019-02-07T14:25:58Z

@DazWilkin I also verified that in this configuration, docker-compose up works successfully. Can you verify with respect to #1190 ?

codecov · 2019-02-07T14:27:52Z

Codecov Report

Merging #1191 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #1191      +/-   ##
=========================================
- Coverage   30.33%   30.3%   -0.03%     
=========================================
  Files          48      48              
  Lines        3867    3867              
=========================================
- Hits         1173    1172       -1     
- Misses       2512    2513       +1     
  Partials      182     182

Impacted Files	Coverage Δ
core/client/client.go	`28.28% <0%> (-0.66%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c05089...1e210f7. Read the comment docs.

DazWilkin · 2019-02-07T17:36:40Z

Checking it now.

DazWilkin · 2019-02-07T18:35:32Z

sequencer service needs the secret references too.

DazWilkin · 2019-02-07T18:49:46Z

Otherwise, it appears to work. I continue to have issues with context deadline exceeded

gdbelvin · 2019-02-07T18:51:01Z

Just added the keys to the sequencer. See if that doesn't solve the context deadline exceeded issue...
If the sequencer isn't starting because it doesn't have access to those keys, it would prevent sequencing progress from happening which would cause a context exceeded error

DazWilkin · 2019-02-07T18:51:48Z

I'd added they keys myself to get it to work but it doesn't address my issue :-(

gdbelvin · 2019-02-07T18:51:52Z

The other place to look for these context deadline errors is the trillian log sequencer. If either the KT sequencer or the log sequencer are having issues clients will see deadline exceeded.

DazWilkin · 2019-02-07T18:52:38Z

checking

DazWilkin · 2019-02-07T18:56:37Z

Everything else appears (!?) to be OK.

The only other recurring log entry that I see across the services is in server and it's:
interceptor.go:38] auth interceptor: no hander for /google.keytransparency.v1.KeyTransparency/GetRevision

NB Typo in hander sb handler

But, otherwise, everything appears OK. Should I publish my logs?

DazWilkin · 2019-02-07T19:03:07Z

I have been unclear as to whether I should use 0.0.0.0:8080 (sequencer) or 0.0.0.0:443 (server).

Trying again using :443 for both authorized-keys create-keyset and then...

keytransparency-client post dazwilkin@google.com \
--client-secret=./client_secret.190205.json \
--insecure \
--data='dGVzdA==' \
--password=${PASSWORD} \
--kt-url=0.0.0.0:443 \
--verbose \
--logtostderr
Go to the following link in your browser then type the authorization code: 
https://accounts.google.com/o/oauth2/auth?access_type=online&client_id=[[REDACTED]]
[[REDACTED]]
2019/02/07 10:59:56 ✓ Signed Map Head signature verified.
2019/02/07 10:59:56 ✓ Log inclusion proof verified.
I0207 10:59:56.987019  158923 client.go:145] Trusted root updated to TreeSize 1
2019/02/07 10:59:56 ✓ Log root updated.
2019/02/07 10:59:56 ✓ Commitment verified.
2019/02/07 10:59:57 ✓ VRF verified.
2019/02/07 10:59:57 ✓ map inclusion proof verified.
2019/02/07 10:59:57 Got current entry...
2019/02/07 10:59:57 Sending Update request...
2019/02/07 10:59:58 ✓ Signed Map Head signature verified.
2019/02/07 10:59:58 ✓ Log inclusion proof verified.
Error: update failed: context deadline exceeded

gdbelvin · 2019-02-07T19:34:13Z

That timeout error is happening really quickly. I think the client is picking up an odd default timeout value of 0s for some reason.

gdbelvin · 2019-02-07T19:36:20Z

The other thing that is happening is that we're starting the timeout counter before the user goes through the OAuth flow

DazWilkin · 2019-02-07T19:38:10Z

So, you're telling me that my 'open link', allow, copy-paste then enter is too slow? ;-)

gdbelvin · 2019-02-07T19:41:37Z

It's embarrassing :-) Fix: #1195

RJPercival · 2019-02-08T13:52:10Z

This isn't an issue introduced by this PR, but shouldn't the key generation scripts create and chmod the files before writing the keys to them?

Use go env GOPATH for situations where the GOPATH environment variable may be unset https://golang.org/cmd/go/#hdr-GOPATH_environment_variable

NeverTestGitHub

google vrp test

RJPercival · 2019-05-13T12:50:58Z

scripts/gen_monitor_keys.sh

-mkdir -p "${GOPATH}/src/github.com/google/keytransparency/genfiles"
-cd "${GOPATH}/src/github.com/google/keytransparency/genfiles"
+mkdir -p "$(go env GOPATH)/src/github.com/google/keytransparency/genfiles"
+cd "$(go env GOPATH)/src/github.com/google/keytransparency/genfiles"


super-nit: Won't work as expected if $GOPATH contains more than one path

RJPercival · 2019-05-13T12:52:32Z

scripts/gen_server_keys.sh

 	a) ADDRESS=${OPTARG};;
 	s) SAN_DNS=${OPTARG};;
-	*) echo "usage: ./generate.sh -d <domain> -a <ip_address> -s <san_extension_DNS>"; exit 1;;
+	*) echo "usage: ./gen_server_keys.sh -d <domain> -a <ip_address> -s <san_extension_DNS>"; exit 1;;


nit: Can use $0 or $(basename $0) instead of the filename here.

* master: (106 commits) Remove unused logVerifier (google#1324) Verify Revisions in StreamRevisions (google#1323) Pair verifier functions (google#1322) Split VerifyRevision into Verify{LogRoot,MapRevision (google#1318) Make Previous hash check optional (google#1307) Remove VerifySignedMapRoot from VerifierInterface (google#1320) Remove trailing whitespace (google#1321) Encapsulate Client Verifier State in test vectors (google#1316) Pass along err message (google#1314) Remove unnessesary func() (google#1319) New test vector transcript format (google#1315) Track map revision inside mutation (google#1310) Move verifier to its own package (google#1312) go generate ./... (google#1306) Fix proto copying in revisions and paginator tests. (google#1309) Fix proto copying in server_test. (google#1308) go mod tidy (google#1305) Use new TrillianMapWrite API (google#1304) Configurable maximum queue depth for metric reporting. (google#1303) Proposal to refine docker deployment (google#1302) ...

gdbelvin requested a review from RJPercival February 7, 2019 14:19

gdbelvin force-pushed the genkeys branch 2 times, most recently from 8f01e17 to 52b9c84 Compare February 7, 2019 14:24

gdbelvin requested a review from DazWilkin February 7, 2019 14:26

gdbelvin mentioned this pull request Feb 7, 2019

Integration tests are not working with docker-compose mysql #1184

Closed

DazWilkin mentioned this pull request Feb 7, 2019

Prefer absolute references to Google Trillian? #1193

Closed

gdbelvin added the security label Feb 8, 2019

gdbelvin added 5 commits February 8, 2019 16:09

Use $(go env GOPATH)

3fd178f

Use go env GOPATH for situations where the GOPATH environment variable may be unset https://golang.org/cmd/go/#hdr-GOPATH_environment_variable

Remove secrets from the docker images

bbec99b

Bump docker version

2b67c35

Add secrets to sequencer

8130699

Set permission bits before writing file

e494bd5

gdbelvin force-pushed the genkeys branch from 11aa66f to e494bd5 Compare February 8, 2019 16:17

NeverTestGitHub approved these changes Mar 11, 2019

View reviewed changes

RJPercival approved these changes May 13, 2019

View reviewed changes

gdbelvin mentioned this pull request Jun 19, 2019

Proposal to refine docker deployment #1302

Merged

gdbelvin requested a review from a team as a code owner July 18, 2019 10:51

googlebot added the cla: yes label Jul 18, 2019

gdbelvin added 3 commits July 19, 2019 11:57

Merge branch 'master' into genkeys

e8fa468

Set docker-compose versions

2080af2

Merge branch 'master' into genkeys

1e210f7

gdbelvin merged commit 2aa29cd into google:master Jul 19, 2019

gdbelvin deleted the genkeys branch July 19, 2019 12:03

Comments

Conversation

gdbelvin commented Feb 7, 2019

Uh oh!

gdbelvin commented Feb 7, 2019

Uh oh!

codecov bot commented Feb 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

gdbelvin commented Feb 7, 2019

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

gdbelvin commented Feb 7, 2019

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

DazWilkin commented Feb 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

gdbelvin commented Feb 7, 2019

Uh oh!

gdbelvin commented Feb 7, 2019

Uh oh!

DazWilkin commented Feb 7, 2019

Uh oh!

gdbelvin commented Feb 7, 2019

Uh oh!

RJPercival commented Feb 8, 2019

Uh oh!

NeverTestGitHub left a comment

Choose a reason for hiding this comment

Uh oh!

RJPercival May 13, 2019

Choose a reason for hiding this comment

Uh oh!

RJPercival May 13, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Feb 7, 2019 •

edited

Loading

DazWilkin commented Feb 7, 2019 •

edited

Loading