Skip to content
This repository was archived by the owner on Oct 24, 2023. It is now read-only.
This repository was archived by the owner on Oct 24, 2023. It is now read-only.

KMS encryption produces zero byte azure.json #49

@jackfrancis

Description

@jackfrancis

On a master immediately after provisioning:

$ sudo systemctl status kubelet
● kubelet.service - Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Wed 2018-11-21 00:43:09 UTC; 11min ago
 Main PID: 9924 (code=exited, status=255)

Nov 21 00:43:09 k8s-master-16337524-0 kubelet[9924]: I1121 00:43:08.985421    9924 mount_linux.go:211] Detected OS with systemd
Nov 21 00:43:09 k8s-master-16337524-0 kubelet[9924]: I1121 00:43:08.991898    9924 server.go:376] Version: v1.10.9
Nov 21 00:43:09 k8s-master-16337524-0 kubelet[9924]: I1121 00:43:08.991942    9924 feature_gate.go:226] feature gates: &{{} map[PodPriority:true]}
Nov 21 00:43:09 k8s-master-16337524-0 kubelet[9924]: F1121 00:43:08.992081    9924 server.go:233] failed to run Kubelet: could not init cloud provider "azure": No credentials provided for AAD ap
Nov 21 00:43:09 k8s-master-16337524-0 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Nov 21 00:43:09 k8s-master-16337524-0 systemd[1]: Stopped Kubelet.
Nov 21 00:43:09 k8s-master-16337524-0 systemd[1]: kubelet.service: Start request repeated too quickly.
Nov 21 00:43:09 k8s-master-16337524-0 systemd[1]: Failed to start Kubelet.
Nov 21 00:43:09 k8s-master-16337524-0 systemd[1]: kubelet.service: Unit entered failed state.
Nov 21 00:43:09 k8s-master-16337524-0 systemd[1]: kubelet.service: Failed with result 'start-limit-hit'.
$ ls -la /etc/kubernetes/azure.json 
-r-------- 1 root root 0 Nov 21 00:42 /etc/kubernetes/azure.json
$ cat /etc/kubernetes/encryption-config.yaml
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
    - secrets
    providers:
    - kms:
        name: azurekmsprovider
        endpoint: unix:///opt/azurekms.socket
        cachesize: 0
    - identity: {}
$ cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubelet
ConditionPathExists=/usr/local/bin/kubelet
Requires=kms.service

[Service]
Restart=always
EnvironmentFile=/etc/default/kubelet
SuccessExitStatus=143
ExecStartPre=/bin/bash /opt/azure/containers/kubelet.sh
ExecStartPre=/bin/mkdir -p /var/lib/kubelet
ExecStartPre=/bin/mkdir -p /var/lib/cni
ExecStartPre=/bin/bash -c "if [ $(mount | grep \"/var/lib/kubelet\" | wc -l) -le 0 ] ; then /bin/mount --bind /var/lib/kubelet /var/lib/kubelet ; fi"
ExecStartPre=/bin/mount --make-shared /var/lib/kubelet
# This is a partial workaround to this upstream Kubernetes issue:
#  https://github.com/kubernetes/kubernetes/issues/41916#issuecomment-312428731
ExecStartPre=/sbin/sysctl -w net.ipv4.tcp_retries2=8
ExecStartPre=-/sbin/ebtables -t nat --list
ExecStartPre=-/sbin/iptables -t nat --list
ExecStart=/usr/local/bin/kubelet \
        --enable-server \
        --node-labels="${KUBELET_NODE_LABELS}" \
        --v=2 \
        --volume-plugin-dir=/etc/kubernetes/volumeplugins \
        $KUBELET_CONFIG $KUBELET_OPTS \
        $KUBELET_REGISTER_NODE $KUBELET_REGISTER_WITH_TAINTS

[Install]
WantedBy=multi-user.target

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions