enable kubelet server cert bootstrapping by sjenning · Pull Request #187 · openshift/machine-config-operator

sjenning · 2018-11-20T22:43:26Z

@smarterclayton @ericavonb @mrogers950 @rphillips

wait for openshift/cluster-machine-approver#3 to merge and ensure that machine-approver is running

sjenning · 2018-11-20T23:24:52Z

/hold

smarterclayton · 2018-11-21T01:23:00Z

Also add RotateKubeletClientCertificate, and let's make sure rotate certificates is set here too.

3.10 on always rotated both, but at a slower interval. We want 4.0 to rotate both, more quickly (days).

Yes, this is set for workers, but not masters. I didn't want to make that change as part of this until I understood why this was done.

sjenning · 2018-11-21T02:08:57Z

Test cluster is working with the approver starting and approving new machines as expected, but I think I need to move the approver to runlevel 0. At runlevel 1 is starts late and kubelets will not accept incoming connections until the server cert is installed. openshift/cluster-machine-approver#4

sjenning · 2018-11-21T02:26:11Z

/retest

smarterclayton · 2018-11-21T06:08:17Z

I think you'll need to ensure the signer is running early in bootstrap.

…

On Tue, Nov 20, 2018 at 11:36 PM OpenShift CI Robot < ***@***.***> wrote: @sjenning <https://github.com/sjenning>: The following test *failed*, say /retest to rerun them all: Test name Commit Details Rerun command ci/prow/e2e-aws c78e5cd <c78e5cd> link <https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/187/pull-ci-openshift-machine-config-operator-master-e2e-aws/333> /test e2e-aws Full PR test history <https://openshift-gce-devel.appspot.com/pr/openshift_machine-config-operator/187>. Your PR dashboard <https://openshift-gce-devel.appspot.com/pr/sjenning>. Please help us cut down on flakes by linking to <https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests> an open issue <https://github.com/openshift/machine-config-operator/issues?q=is:issue+is:open> when you hit one in your PR. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/guide/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. I understand the commands that are listed here <https://go.k8s.io/bot-commands>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#187 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pz87-jrQzR18BkP78-0U5swV5b_lks5uxNhQgaJpZM4YsD7O> .

sjenning · 2018-11-21T14:59:46Z

i think openshift/cluster-machine-approver#4 may have not gone in before the previous retest
/retest

rphillips · 2018-11-26T16:13:06Z

EOL got stripped on this file

sjenning · 2018-11-26T18:29:21Z

# crictl logs f2fd6df8c27e0
2018/11/26 18:16:29 socat[4847] E connect(5, AF=1 "/var/run/openshift-sdn/cni-server.sock", 40): No such file or directory
User "sa" set.
Context "kubelet" modified.
I1126 18:16:29.545148    4826 start_network.go:200] Reading node configuration from /tmp/sdn-config.yaml
Invalid NodeConfig /tmp/sdn-config.yaml
  servingInfo.certFile: Invalid value: "/etc/kubernetes/pki/kubelet.crt": could not read file: stat /etc/kubernetes/pki/kubelet.crt: no such file or directory
  servingInfo.keyFile: Invalid value: "/etc/kubernetes/pki/kubelet.key": could not read file: stat /etc/kubernetes/pki/kubelet.key: no such file or directory

# crictl inspect f2fd6df8c27e0
...
      {
        "containerPath": "/etc/kubernetes/pki",
        "hostPath": "/var/lib/kubelet/pki",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": false
      },
...

The sdn pod is not starting and is blocking the cluster control plane from starting. It is not clear why the sdn pod needs the kubelet server key/cert. Seem to be a remnant of using the NodeConfig code in the sdn process. Discussing with @squeed and @dcbw.

sjenning · 2018-11-26T20:18:38Z

needs openshift/origin#21551 plus a PR to the cluster-network-operator to not set the kubelet serving info in the generated NodeConfig.

sjenning · 2018-11-27T21:24:58Z

next in line openshift/cluster-network-operator#43

sjenning · 2018-11-28T16:46:24Z

/retest

sjenning · 2018-11-28T20:31:02Z

/retest

sjenning · 2018-11-29T14:27:45Z

/retest

smarterclayton · 2018-12-02T19:42:14Z

Please rebase - this is critical for getting metrics from nodes.

openshift-bot · 2018-12-03T15:53:34Z

@sjenning: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sjenning · 2018-12-03T16:19:19Z

/hold cancel

sjenning · 2018-12-03T17:03:58Z

/retest

sjenning · 2018-12-03T18:10:13Z

/retest

smarterclayton · 2018-12-03T19:41:38Z

/lgtm

openshift-ci-robot · 2018-12-03T19:41:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sjenning, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

DEVEXP-418: fetch arch from GOARCH, allow for z/ppc

openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Nov 20, 2018

openshift-ci-robot requested review from crawford and wking November 20, 2018 22:43

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 20, 2018

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 20, 2018

smarterclayton reviewed Nov 21, 2018

View reviewed changes

rphillips reviewed Nov 26, 2018

View reviewed changes

sjenning force-pushed the kubelet-server-cert-rotation branch 2 times, most recently from da1c326 to f9fe715 Compare November 26, 2018 17:36

sjenning mentioned this pull request Nov 27, 2018

remove kubelet pki from sdn pod openshift/cluster-network-operator#43

Merged

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 3, 2018

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2018

sjenning mentioned this pull request Dec 3, 2018

kubelet uses self-signed serving certs openshift/installer#486

Closed

sjenning added 2 commits December 3, 2018 10:22

enable kubelet server cert bootstrapping

f239e27

update test_data

6b49762

sjenning force-pushed the kubelet-server-cert-rotation branch from f9fe715 to 6b49762 Compare December 3, 2018 16:23

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 3, 2018

openshift-ci-robot assigned smarterclayton Dec 3, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 3, 2018

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 3, 2018

openshift-merge-robot merged commit 84bf7d6 into openshift:master Dec 3, 2018

osherdp pushed a commit to osherdp/machine-config-operator that referenced this pull request Apr 13, 2021

Merge pull request openshift#187 from gabemontero/nonx86

b723076

DEVEXP-418: fetch arch from GOARCH, allow for z/ppc

sinnykumari mentioned this pull request Jun 8, 2023

OCPBUGS-13656: MCO-632: Update kube deps to 1.27.2 #3735

Merged

Conversation

sjenning commented Nov 20, 2018

Uh oh!

sjenning commented Nov 20, 2018

Uh oh!

smarterclayton Nov 21, 2018

Choose a reason for hiding this comment

Uh oh!

sjenning Nov 21, 2018

Choose a reason for hiding this comment

Uh oh!

sjenning commented Nov 21, 2018

Uh oh!

sjenning commented Nov 21, 2018

Uh oh!

smarterclayton commented Nov 21, 2018 via email

Uh oh!

sjenning commented Nov 21, 2018

Uh oh!

rphillips Nov 26, 2018

Choose a reason for hiding this comment

Uh oh!

sjenning commented Nov 26, 2018

Uh oh!

sjenning commented Nov 26, 2018

Uh oh!

sjenning commented Nov 27, 2018

Uh oh!

sjenning commented Nov 28, 2018

Uh oh!

sjenning commented Nov 28, 2018

Uh oh!

sjenning commented Nov 29, 2018

Uh oh!

smarterclayton commented Dec 2, 2018

Uh oh!

openshift-bot commented Dec 3, 2018

Uh oh!

sjenning commented Dec 3, 2018

Uh oh!

sjenning commented Dec 3, 2018

Uh oh!

sjenning commented Dec 3, 2018

Uh oh!

smarterclayton commented Dec 3, 2018

Uh oh!

openshift-ci-robot commented Dec 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants