Randomly set credentials written in cleartext to world-readable file

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1918303

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Bionic)', 'cloud-init (Ubuntu Focal)', 'cloud-init (Ubuntu Groovy)']
assignee = oddbloke
assignee_name = Dan Watkins
date_closed = 2021-05-05T20:34:41.802942+00:00
date_created = 2021-03-09T15:37:00.582455+00:00
date_fix_committed = 2021-03-23T16:24:33.988640+00:00
date_fix_released = 2021-05-05T20:34:41.802942+00:00
id = 1918303
importance = critical
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1918303
milestone = None
owner = wasp0x01
owner_name = Carl Pearson
private = False
status = fix_released
submitter = wasp0x01
submitter_name = Carl Pearson
tags = ['verification-done', 'verification-done-bionic', 'verification-done-focal', 'verification-done-groovy', 'verification-done-xenial']
duplicates = [1888483]

Launchpad user Carl Pearson(wasp0x01) wrote on 2021-03-09T15:37:00.582455+00:00

Summary

cloud-init allows administrators to set passwords for user accounts via the chpasswd configuration module. Administrators can instruct cloud-init to set a random password generated at runtime using the 'R' or 'RANDOM' keywords.

However, cloud-init appears to write all randomly generated passwords in cleartext to stderr. Cloud-init's default logging configuration, in file /etc/cloud/cloud.cfg.d/05_logging.cfg, redirects both stdout and stderr to the log file /var/log/cloud-init-output.log. The file /var/log/cloud-init-output.log is world readable. Thus, any unprivileged account on the system can view the cleartext password for any account which had a random password generated at runtime. The credentials are not redacted in the log.

Reproduction

Pre-requisites: A device with Ubuntu Server 20.04 installed. Ubuntu Server comes with cloud-init pre-installed out of the box, but the latest release of cloud-init as of this report (21.1) is not available in 20.04's apt repositories. You may need to install v21.1 manually. You will also need an exsiting admin account with root privileges.

1. Login as admin.
2. Create an unprivileged user account, bob, and set a password. We will use this account to demonstrate unprivileged account access to generated passwords.
   sudo adduser bob
3. Create another unprivileged user account, alice, and set a password. We will change this account's password with cloud-init.
   sudo adduser alice
4. Create and open configuration file /etc/cloud/cloud.cfg.d/95_chpasswd.cfg using vim or other editor of your choice.
   sudo vim /etc/cloud/cloud.cfg.d/95_chpasswd.cfg
5. Add the following chpasswd configuration content to the file then save and exit.
   chpasswd:
   list: |
   alice:RANDOM
6. cloud-init only runs the chpasswd function on first boot of the OS that cloud-init knows about. For proof of concept purposes, we need to simulate a new instance. Run:
   sudo cloud-init clean
   to reset cloud-init's state.
7. Reboot the system.
   sudo reboot
8. Login as unprivileged user bob.
9. View the password by runnnig
   cat /var/log/cloud-init-output.log | grep alice
10. Alice's temporary password should appear on terminal in the form alice:
11. Logout and log back in to the system as alice using the temporary password. You should get access and prompted to set a new password, which confirms the password bob retrieved from the logs is the actual password for alice's account.

Impact

Any unprivileged user on the system can retrieve all cloud-init randomly set credentials. These could potentially be used to access other accounts.

Notes

If 'expire: false' is added to the chpasswd config, then leaked passwords remain valid until manually changed and increases the risk of unauthorized account access. Otherwise, the default behaviour prompts accounts to set a new password at next login, reducing the time window for unauthorized access.

Accounts not used for interactive login might not get passwords changed or accounts might get a password set but then not authenticate for some time. The precise impact and duration of valid exposed credentials appears dependent somewhat on each cloud-init customer's environment and how they use cloud-init to set credentials.

I'm not sure the best approach to patch this but perhaps the credentials could be written to cloud-init's protected directories or files which restrict access to root users only, such as /var/run/cloud-init/instance-data-sensitive.json?

Line 214 of https://github.com/canonical/cloud-init/blob/master/cloudinit/config/cc_set_passwords.py checks if any random passwords were set and if so prints each one to stderror. This might be the root cause.

Tested on Ubuntu Server 20.04.02, cloud-init latest release 21.1 as of report time. If I can provide any further information please let me know. Thanks!
-Carl

[ubuntu-server-builder]

Launchpad user Carl Pearson(wasp0x01) wrote on 2021-03-09T15:37:00.582455+00:00

Launchpad attachments: cloud-init.tar.gz

[ubuntu-server-builder]

Launchpad user Carl Pearson(wasp0x01) wrote on 2021-03-09T15:37:30.399548+00:00

My chpasswd config file
Launchpad attachments: 95_chpasswd.cfg

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-03-09T22:23:10.282555+00:00

Hi Carl,

Thanks for this detailed and thoughtful bug report, we really appreciate it. We've had some preliminary internal discussions, and we have a sync with our security team scheduled for tomorrow afternoon to discuss whether or not we think this issue warrants a coordinated release. Below is my initial analysis.

In order to allow people to access systems using these randomly generated passwords (without needing another access vector in order to know the passwords), cloud-init emits them to the serial console. In order to have log messages emitted to the console readily available within the system also, it writes that same content to /var/log/cloud-init-output.log. As a result, those passwords are written to that file, which is world-readable.

In the first instance, we should consider whether or not emitting these random passwords to the console is acceptable default behaviour. For users who are using this functionality as the only way they access the instance, it is: there's no other way for them to know what password to use. The question then becomes, I think, what proportion of users of this functionality use it in this way? If it's the majority, then perhaps emitting them by default is justifiable. If not (or if we think this behaviour is simply too dangerous to be the default), then we could consider toggling it with a new configuration option, defaulting to false: users are already passing in cloud-config to get into this situation, so working around this change in behaviour should be reasonably straightforward. (I don't think we can remove support for the emission altogether: that will break existing workflows, and in a way that is likely to move such folks to simply using hard-coded passwords: not exactly a security fix.)

Given my (parenthetical) conclusion, we clearly do also need to address how we are going to remove access to these passwords from unprivileged users within the system. Two obvious options present themselves: (a) we could make cloud-init-output.log only root-readable, or (b) we could modify the code which emits these messages to either redact them when going to cloud-init-output.log or omit them from there entirely. cloud-init-output.log and the serial console are tied together very tightly[0], so (b) could prove to be difficult: we'll perform some further analysis. (It's possible that the default config in [0] could be extended to omit the lines using grep, for example?)

(For (a), I also just noticed that the content in cloud-init-output.log is also present in the journal for the various cloud-init units. Whatever resolution we reach regarding cloud-init-output.log, we should confirm that the permissions that journalctl enforces for such access are acceptable to us.)

Thanks again for the report! Any comments, questions, or suggestions (either from you, Carl, or the other cloud-init folks with access to this report) are more than welcome!

Dan

[0] The default configuration should give you a sense of how thorough that link is: "output: {all: '| tee -a /var/log/cloud-init-output.log'}"

[ubuntu-server-builder]

Launchpad user Carl Pearson(wasp0x01) wrote on 2021-03-10T05:35:21+00:00

Hi Dan,
Thank you for the update and the detailed analysis. I'm not as familiar with some of the cloud-init use cases so good to know what some of the constraints are. Yep, restricting read access to cloud-init-output.log or redacting the messages when sent to the log both sound like workable strategies to me, although as you noted both have some caveats to work around. 
Thanks and have a great day,
Carl
On Tuesday, March 9, 2021, 2:30:42 PM PST, Dan Watkins 1918303@bugs.launchpad.net wrote:

Hi Carl,

Thanks for this detailed and thoughtful bug report, we really appreciate
it.  We've had some preliminary internal discussions, and we have a sync
with our security team scheduled for tomorrow afternoon to discuss
whether or not we think this issue warrants a coordinated release.
Below is my initial analysis.

In order to allow people to access systems using these randomly
generated passwords (without needing another access vector in order to
know the passwords), cloud-init emits them to the serial console.  In
order to have log messages emitted to the console readily available
within the system also, it writes that same content to /var/log/cloud-
init-output.log.  As a result, those passwords are written to that file,
which is world-readable.

In the first instance, we should consider whether or not emitting these
random passwords to the console is acceptable default behaviour.  For
users who are using this functionality as the only way they access the
instance, it is: there's no other way for them to know what password to
use.  The question then becomes, I think, what proportion of users of
this functionality use it in this way?  If it's the majority, then
perhaps emitting them by default is justifiable.  If not (or if we think
this behaviour is simply too dangerous to be the default), then we could
consider toggling it with a new configuration option, defaulting to
false: users are already passing in cloud-config to get into this
situation, so working around this change in behaviour should be
reasonably straightforward.  (I don't think we can remove support for
the emission altogether: that will break existing workflows, and in a
way that is likely to move such folks to simply using hard-coded
passwords: not exactly a security fix.)

Given my (parenthetical) conclusion, we clearly do also need to address
how we are going to remove access to these passwords from unprivileged
users within the system.  Two obvious options present themselves: (a) we
could make cloud-init-output.log only root-readable, or (b) we could
modify the code which emits these messages to either redact them when
going to cloud-init-output.log or omit them from there entirely.  cloud-
init-output.log and the serial console are tied together very
tightly[0], so (b) could prove to be difficult: we'll perform some
further analysis.  (It's possible that the default config in [0] could
be extended to omit the lines using grep, for example?)

(For (a), I also just noticed that the content in cloud-init-output.log
is also present in the journal for the various cloud-init units.
Whatever resolution we reach regarding cloud-init-output.log, we should
confirm that the permissions that journalctl enforces for such access
are acceptable to us.)

Thanks again for the report!  Any comments, questions, or suggestions
(either from you, Carl, or the other cloud-init folks with access to
this report) are more than welcome!

Dan

[0] The default configuration should give you a sense of how thorough
that link is: "output: {all: '| tee -a /var/log/cloud-init-output.log'}"

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1918303

Title:
  Randomly set credentials written in cleartext to world-readable file

Status in cloud-init:
  New

Bug description:
  ## Summary

  cloud-init allows administrators to set passwords for user accounts
  via the chpasswd configuration module. Administrators can instruct
  cloud-init to set a random password generated at runtime using the 'R'
  or 'RANDOM' keywords.

  However, cloud-init appears to write all randomly generated passwords
  in cleartext to stderr. Cloud-init's default logging configuration, in
  file /etc/cloud/cloud.cfg.d/05_logging.cfg, redirects both stdout and
  stderr to the log file /var/log/cloud-init-output.log. The file
  /var/log/cloud-init-output.log is world readable. Thus, any
  unprivileged account on the system can view the cleartext password for
  any account which had a random password generated at runtime. The
  credentials are not redacted in the log.

  ## Reproduction

  Pre-requisites: A device with Ubuntu Server 20.04 installed. Ubuntu
  Server comes with cloud-init pre-installed out of the box, but the
  latest release of cloud-init as of this report (21.1) is not available
  in 20.04's apt repositories. You may need to install v21.1 manually.
  You will also need an exsiting admin account with root privileges.

  1. Login as admin.
  2. Create an unprivileged user account, bob, and set a password. We will use this account to demonstrate unprivileged account access to generated passwords.
  sudo adduser bob
  3. Create another unprivileged user account, alice, and set a password. We will change this account's password with cloud-init.
  sudo adduser alice
  4. Create and open configuration file /etc/cloud/cloud.cfg.d/95_chpasswd.cfg using vim or other editor of your choice.
  sudo vim /etc/cloud/cloud.cfg.d/95_chpasswd.cfg
  5. Add the following chpasswd configuration content to the file then save and exit.
  chpasswd:
    list: |
      alice:RANDOM
  6. cloud-init only runs the chpasswd function on first boot of the OS that cloud-init knows about. For proof of concept purposes, we need to simulate a new instance. Run:
  sudo cloud-init clean
  to reset cloud-init's state.
  7. Reboot the system.
  sudo reboot
  8. Login as unprivileged user bob.
  9. View the password by runnnig
  cat /var/log/cloud-init-output.log | grep alice
  10. Alice's temporary password should appear on terminal in the form alice:
  11. Logout and log back in to the system as alice using the temporary password. You should get access and prompted to set a new password, which confirms the password bob retrieved from the logs is the actual password for alice's account.

  ## Impact

  Any unprivileged user on the system can retrieve all cloud-init
  randomly set credentials. These could potentially be used to access
  other accounts.

  # Notes

  If 'expire: false' is added to the chpasswd config, then leaked
  passwords remain valid until manually changed and increases the risk
  of unauthorized account access. Otherwise, the default behaviour
  prompts accounts to set a new password at next login, reducing the
  time window for unauthorized access.

  Accounts not used for interactive login might not get passwords
  changed or accounts might get a password set but then not authenticate
  for some time. The precise impact and duration of valid exposed
  credentials appears dependent somewhat on each cloud-init customer's
  environment and how they use cloud-init to set credentials.

  I'm not sure the best approach to patch this but perhaps the
  credentials could be written to cloud-init's protected directories or
  files which restrict access to root users only, such as /var/run
  /cloud-init/instance-data-sensitive.json?

  Line 214 of https://github.com/canonical/cloud-
  init/blob/master/cloudinit/config/cc_set_passwords.py checks if any
  random passwords were set and if so prints each one to stderror. This
  might be the root cause.

  Tested on Ubuntu Server 20.04.02, cloud-init latest release 21.1 as of report time. If I can provide any further information please let me know. Thanks!
  -Carl

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1918303/+subscriptions

[ubuntu-server-builder]

Launchpad user Seth Arnold(seth-arnold) wrote on 2021-03-11T00:03:18.821952+00:00

Hello Carl, thanks for the excellent report.

Please use CVE-2021-3429 for this issue.

Normally I'd say printing passwords to logs is a mistake but I can see how we got here. Probably we should make this configurable.

I think we should also change the permissions to match the systemd journal files.

On my 20.04 laptop, files in /var/log/journal/*/ are readable by group systemd-journal and group adm. adm is allowed to read many log files on Debian and derivatives, though normally passwords aren't included in logs.

Thanks

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-03-12T17:21:16.910237+00:00

We've identified two remediations here: we're going to stop writing the passwords to this file at all, and also remove its world-readable permissions. I have an implementation of this first part locally.

For the latter, we would ideally set the user/group and permissions to match other such log files. However, this presents a challenge: cloud-init's code doesn't directly create cloud-init-output.log. Rather, it is created by the tee process which we pipe all of our output in the default configuration[0]:

output: {all: '| tee -a /var/log/cloud-init-output.log'}

The two other places that "cloud-init-output.log" is mentioned in the cloud-init codebase are in the collect-logs code and the cloudinit.apport module: both to read from the file in a booted system.

So from a code perspective, we don't really have a good place to put our permission handling logic. Doing it unconditionally would be incorrect: that would create a cloud-init-output.log file on systems which aren't configured to have such a file.

We could introspect the configured value (if "cloud-init-output.log" in ...) and pre-create the file if we find that string (the presence of which almost certainly indicates that cloud-init-output.log will be created. This would solve this specific issue, but leaves this class of bug open: if someone has configured a different log file to emit output to, our handling will not trigger, leaving that log file vulnerable.

Given the freeform nature of the configuration, I don't think we can fully solve the problem for non-default log settings: we can't know what log files will be created by a given setting (e.g. "| custom_logging_script" could write anywhere at all, depending on the implementation of custom_logging_script). However, we can set the umask on the subprocess we start to pipe our output to: this will at least allow us to ensure that the file is not world-readable (though does not allow us to set the user/groups we would prefer).

So, my proposal for the permissions change is in two parts. Firstly, set the umask of the receiving process to 0x0026: this means that files it creates will not be world-readable. Secondly, add special-case handling for the default configuration: look for "/var/log/cloud-init-output.log" in the given configuration and, if present, create the file with 640 permissions and the appropriate user:group before launching the receiving process. (I think root:adm is probably what we want?)

[0] https://github.com/canonical/cloud-init/blob/master/config/cloud.cfg.d/05_logging.cfg#L71

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-03-12T17:27:43.858389+00:00

Firstly, set the umask of the receiving process to 0x0026

Oops, forgot to update this after poking at it some more: I mean 0o037 (i.e. 000 011 111).

[ubuntu-server-builder]

Launchpad user Dan Watkins(oddbloke) wrote on 2021-03-23T16:25:02.038168+00:00

A fix for this landed in b794d42

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2021-03-23T17:16:12.468439+00:00

cloud-init (21.1-19-gbad84ad4-0ubuntu1) hirsute; urgency=medium

• d/cloud-init.postinst: Change output log permissions on upgrade
  (LP: #1918303)

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2021-03-23T22:44:54.786879+00:00

Hello Carl, or anyone else affected,

Accepted cloud-init into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2021-03-23T22:49:10.844626+00:00

Hello Carl, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2021-03-23T23:00:57.187113+00:00

Hello Carl, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2021-03-23T23:05:58.349913+00:00

Hello Carl, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/21.1-19-gbad84ad4-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.