Problem description
moby/swarmkit#2246 stops generating and storing keys in PKCS1 format and stores them in PKCS8 format instead. Older versions of docker can read unencrypted PKCS8 format, but not encrypted PKCS8 format. Therefore, on swarm nodes, once TLS keys have been rotated to be in PKCS 8 format, if they are encrypted (such as on managers with autolock enabled), downgrading to a version of docker that cannot read encrypted PKCS8 keys can be problematic.
Project version(s) affected
Future version of docker that includes changes from moby/swarmkit#2246.
Suggestions for a fix
We should document the downgrade process.
If, after upgrading to a version of docker that stores keys in PKCS8 format, and you want to downgrade, check if your manager keys are stored as encrypted PKCS8 keys (worker keys are not encrypted). If you have autolock enabled on your swarm, then it's possible that you have PKCS8 encrypted keys (if the TLS certs have been renewed since you upgraded, or you installed the newer version of docker and now want to downgrade, or if you enabled autolock post-upgrade). In order to downgrade, we need to decrypt the keys, so make sure that autolock is disabled, and before downgrading each manager node, verify that the keys are no longer encrypted.
They should go from looking like this (an encrypted PKCS8 key):
-----BEGIN ENCRYPTED PRIVATE KEY-----
kek-version: 45
raft-dek: CAESMPNMlb2Ava3k6VobbLc8EmyfF0+a+F9FNEXeZclCM8a9373rQMvp/91i7oMlYZ2UBhoYbc1r7GBuGTvnlVrCAFkbCPr2dQSYZ+RC
MIHuMEkGCSqGSIb3DQEFDTA8MBsGCSqGSIb3DQEFDDAOBAjSTabG59CbxgICCAAw
HQYJYIZIAWUDBAEqBBBIDnXxvmr8s+gTa/19wtr/BIGg/tVp7CHJKRcRB1hwKSo5
lbnqYcn8bFwgJYwj7olylCVcLmvkllOk9GV9YP2RahlDEzbFWJiZFQHbTuC9A5eM
kmKZJxTpqHXsLUpTdzG6FdlYn1qI735jgzrzjeFTIhcV+DcECekSQK1WKjIz6xYn
DhxGHm+59Q9RaTv7Y+33lkSt7g9AoM2+BID8hkxWa4mhfWrGJL2ZATrLquoraG4E
fg==
-----END ENCRYPTED PRIVATE KEY-----
to this (an unencrypted PKCS8 key):
-----BEGIN PRIVATE KEY-----
kek-version: 54
raft-dek: EiDpQCHiKY7nIqXmswxGDqhvoAz0JonZgzajSiwi1sdnqg==
MIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHkwdwIBAQQgCxvNf1aSPYT7J1rC
NO/Fi9RXkscrml/9nFfQBfVUiIqgCgYIKoZIzj0DAQehRANCAASMoPKHOh5Gq5Ij
qf6dq5wnjO8+gjj3qvbUIx67LyFVwbWGbbbYimJCEQR99oHRfqZuuYUSUyedEY1o
RIJ/Bz3i
-----END PRIVATE KEY-----
This will guarantee that the older version of docker can read the key. Autolock can be re-enabled once all the managers have been downgraded.
(As a note, in case it's useful for documentation purposes, PKCS1 encrypted keys look like:
-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,ed7a4a5bd75da23e0d0b432e273ff4e6
kek-version: 11
raft-dek: CAESMCk0I41T62g3mx/VB0LvzLlqAG85FJ2qNrkWRMUrd67eLj4qn7PuTK3GCz5tzJs06hoYmSHtET6mNHXSnBFLH30euPubHShF8IMm
4PXBvWv+NcPLuxb1b58LLM/WiEZp4BL5YrhMvpnCiHieHxjeV0P/rIp5agNNjkcB
gCzekta+RPMqHL5FVlApHgOfuHhz2vyfwtwvqqDZd+BCfBEM2XOvrBf9LP0QNcC+
itb4HYH85urXcJte48QTHT0CgNWzT7QclChAJS9QCRM=
-----END EC PRIVATE KEY-----
and unencrypted PKCS keys look like:
-----BEGIN EC PRIVATE KEY-----
kek-version: 0
raft-dek: EiDOkUNCMN6P/WfXo/oJ4glDQHTD385kNkZZ5FAeHi4R/g==
MHcCAQEEIMrThOSIgvnK85Hcz3VGJv/aOWTn/8P/FSOK/UBnNT7toAoGCCqGSM49
AwEHoUQDQgAEXKmyOnq3oxeKxdCmW98TzosMzl2+gd8o3XWYJZ6Ea7s5k5trG4YB
lb/PlonS4yuk29FnArldwYBwyjogoiWvcg==
-----END EC PRIVATE KEY-----
The unencrypted version is missing the Proc-Type and DEK-Info headers)
cc @alyyousuf7 @friism @aluzzardi @diogomonica
Problem description
moby/swarmkit#2246 stops generating and storing keys in PKCS1 format and stores them in PKCS8 format instead. Older versions of docker can read unencrypted PKCS8 format, but not encrypted PKCS8 format. Therefore, on swarm nodes, once TLS keys have been rotated to be in PKCS 8 format, if they are encrypted (such as on managers with autolock enabled), downgrading to a version of docker that cannot read encrypted PKCS8 keys can be problematic.
Project version(s) affected
Future version of docker that includes changes from moby/swarmkit#2246.
Suggestions for a fix
We should document the downgrade process.
If, after upgrading to a version of docker that stores keys in PKCS8 format, and you want to downgrade, check if your manager keys are stored as encrypted PKCS8 keys (worker keys are not encrypted). If you have autolock enabled on your swarm, then it's possible that you have PKCS8 encrypted keys (if the TLS certs have been renewed since you upgraded, or you installed the newer version of docker and now want to downgrade, or if you enabled autolock post-upgrade). In order to downgrade, we need to decrypt the keys, so make sure that autolock is disabled, and before downgrading each manager node, verify that the keys are no longer encrypted.
They should go from looking like this (an encrypted PKCS8 key):
to this (an unencrypted PKCS8 key):
This will guarantee that the older version of docker can read the key. Autolock can be re-enabled once all the managers have been downgraded.
(As a note, in case it's useful for documentation purposes, PKCS1 encrypted keys look like:
and unencrypted PKCS keys look like:
The unencrypted version is missing the
Proc-TypeandDEK-Infoheaders)cc @alyyousuf7 @friism @aluzzardi @diogomonica