Add LXD datasource

[blackboxsw]

Add DataSourceLXD which knows how to talk to the dev-lxd socket to obtain all instance metadata API: https://linuxcontainers.org/lxd/docs/master/dev-lxd.

This first branch is to get deliver feature parity with the existing NoCloud datasource which is currently used to intialize LXC instances on first boot.

Introduce a SocketConnectionPool and LXDSocketAdapter to support performing HTTP GETs on the following routes which are surfaced by the LXD host to all containers:

• http://unix.socket/1.0/meta-data
• http://unix.socket/1.0/config/user.user-data
• http://unix.socket/1.0/config/user.network-config
• http://unix.socket/1.0/config/user.vendor-data

These 4 routes minimally replace the static content provided in the following nocloud-net seed files:
/var/lib/cloud/nocloud-net/{meta-data,vendor-data,user-data,network-config}

The intent of this branch is to set a foundation for LXD socket communication that will allow us to build network hot-plug features by eventually consuming LXD's websocket upgrade route 1.0/events to react to network, meta-data and user-data config changes over time.

In the event that no custom network-config is provided, default to the same network-config definition provided by LXD to the NoCloud network-config seed file.

version: 1
config:
- type:physical
  name: eth0        # or one of enp5s0, enp0s5 or enc9
        dhcp4: true

Supplemental features above NoCloud datasource:

• surface all custom instance data config keys via cloud-init query ds which aids in discoverability of features/tags/labels as well as conditional #cloud-config jinja templates operations based on custom config options.
• TBD: better cloud-init query support for dot-delimited keys `cloud-init query ds[1.0].config.

Proposed Commit Message

Test Steps:

git checkout upstream/ubuntu/bionic debian
dch --newversion 24.0 --distribution bionic
# Add LXD entries via this patch against debian/cloud-init.templates to support dpkg-reconfigure setting LXD as a viable datasource
https://paste.ubuntu.com/p/Cdbz6wRcDf/plain/
patch -p1 < <patch from pastebin> 
git add debian
git commit -am 'local packaging test for LXD'
build-package
sbuild --dist=bionic --arch=amd64 --arch-all ../out/cloud-init*dsc
lxc launch ubuntu-daily:bionic dev-b
lxc file push cloud-init*deb dev-b/
lxc exec dev-b bash
$ apt install /cloud-init*deb
$ sudo dpkg-reconfigure cloud-init # select LXD and NoCloud Datasources
$ sudo cloud-init clean --logs --reboot 
$ sudo cloud-init status --long
$ sudo cloud-init query ds
$ sudo cloud-init query --format "{{platform}} {{subplatform}}"

Checklist:

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

[TheRealFalcon]

Thanks @blackboxsw . This is looking really good!

Some observations:
If you specify user.meta-data when launching the instance, it appends the data that you've sent to its own metadata, rather than overwriting it. While playing around, I threw in an extra log to log all of the crawled metadata and got this:

2021-09-30 14:43:57,538 - DataSourceLXD.py[DEBUG]: {'meta-data': '#cloud-config\ninstance-id: me\nlocal-hostname: me\ninstance-id: i-87018aed\nlocal-hostname: myhost.internal', '1.0': {'config': {'user.user-data': "#cloud-config\nruncmd:\n - echo 'hi' > /root/hi", 'user.meta-data': 'instance-id: i-87018aed\nlocal-hostname: myhost.internal', 'user.vendor-data': "#cloud-config\nbootcmd:\n - echo 'hi from vendor' > /var/tmp/vendor"}}, 'user-data': "#cloud-config\nruncmd:\n - echo 'hi' > /root/hi", 'vendor-data': "#cloud-config\nbootcmd:\n - echo 'hi from vendor' > /var/tmp/vendor"}

This isn't a problem, but in cloud-init query we only report the user metadata. It might be confusing not seeing the pre-existing metadata that we don't overwrite.

cloud-init query -a includes:

 "ds": {
  "1.0": {
   "config": {
    "user.meta_data": "instance-id: i-87018aed\nlocal-hostname: myhost.internal",
    "user.user_data": "#cloud-config\nruncmd:\n - echo 'hi' > /root/hi",
    "user.vendor_data": "#cloud-config\nbootcmd:\n - echo 'hi from vendor' > /var/tmp/vendor"
   }
  }

regardless of the user. Vendordata and userdata should be redacted from the non-root user.

Regardling the security.devlxd, I'm not sure there's much more we can do. If it's not set, we'll use the LXD datasource. If it is set, we'll use NoCloud. The big problem comes if somebody toggles the flag after the container has already started. I can't really see a nice way of handling this. We have no mechanism to dynamically switch datasources, and I don't think we should. After first boot, we're not likely to see metadata changes anyway, so even if the socket gets shut off, I don't think it's necessarily catastrophic for the instance. I think we just need to document that setting the flag to true will force us to use the NoCloud datasource, and toggling the flag for an already running instance can lead to unexpected results.

[TheRealFalcon]

Hmmm, I also just noticed this for each subsequent boot:

2021-09-30 15:13:40,527 - stages.py[DEBUG]: cache invalid in datasource: DataSourceLXD
2021-09-30 15:13:40,527 - handlers.py[DEBUG]: finish: init-local/check-cache: SUCCESS: cache invalid in datasource: DataSourceLXD
2021-09-30 15:13:40,527 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance

[github-actions]

Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close.

If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging mitechie, and he will ensure that someone takes a look soon.

(If the pull request is closed and you would like to continue working on it, please do tag mitechie to reopen it.)

[TheRealFalcon]

This looks really good! I left some comments, but they're all very minor.

[TheRealFalcon]

So < Jammy, we'll need quilt patches to put LXD after NoCloud?

[blackboxsw]

For the record, the order in ds-identify doesn't really matter as it is discovered based on the order configuration file on disk in the datasource_list config option in /etc/cloud/cloud.cfg or /etc/cloud/cloud.cfg.d/90_dpkg.cfg (included in cloud images), or cloudinit/settings.py or kernel_cmdline in the function read_datasource_list

you can see the ds-identify sources this order in /run/cloud-init/ds-identify.log even if LXD is listed before NoCloud in ./tools/ds-identify as I had here... you'll still see datasource_list sourced.

/etc/cloud/cloud.cfg.d/90_dpkg.cfg set datasource_list: [ NoCloud, ConfigDrive, LXD, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Scaleway, AliYun, Ec2, CloudStack, Hetzner, IBMCloud, Oracle, Exoscale, RbxCloud, UpCloud, VMware, Vultr, None ]
DSLIST=NoCloud ConfigDrive LXD OpenNebula DigitalOcean Azure AltCloud OVF MAAS GCE OpenStack CloudSigma SmartOS Bigstep Scaleway AliYun Ec2 CloudStack Hetzner IBMCloud Oracle Exoscale RbxCloud UpCloud VMware Vultr None
check for 'NoCloud' returned found
check for 'LXD' returned found
Found 2 datasources found=all: NoCloud LXD

That ordering found by ds-identify sets the ordering of python datasources discovery:

root@dev-bb:~# grep LXD /var/log/cloud-init.log 
2021-10-30 02:34:33,682 - __init__.py[DEBUG]: Looking for data source in: ['NoCloud', 'LXD', 'None'], via packages ['', 'cloudinit.sources'] that matches dependencies ['FILESYSTEM']

All said though, it's best if we keep this ordering aligned. Because in absence of other defaults, eventually we'd fallback to ds-itentify ordering in the DI_DSLIST_DEFAULT.

[TheRealFalcon]

I know it's not likely, but if config is null in the metadata definition, then it will be None here, and the .get after it will fail.

[blackboxsw]

+1, reasonable. better to be defensive and schema warn than to fall over with an ugly trace
Well this was wrong to begin with as read_metadata actually returns the config key nested under the API version as in {"1.0": {"config": {...}}} That said, I've changed this to check of LXD_SOCKET_API_VERSION exists from read_metadata. When it does the "config" key is guaranteed to be a dict.

[TheRealFalcon]

This can return None which would traceback on the following line.

[TheRealFalcon]

Not something we need to change now, but I'd like to get away from setting things to sources.UNSET. Mypy raises hell all over this module because we're setting it to a string here, but the "actual" type after it's been set is a dict...and same with the crawled_metadata line two lines below.

[blackboxsw]

+1 let's look generally at better mypy cleanup on the 2nd pass here.

[blackboxsw]

Thanks @blackboxsw . This is looking really good!

Some observations: If you specify user.meta-data when launching the instance, it appends the data that you've sent to its own metadata, rather than overwriting it.

Thanks I added a ds["1,0"]["meta-data"] key too so we can see both from cloud-init query. LXD team might change how this is surfaced to LXD datasource mid-cycle so surfacing the raw content we see is probably the best plan forward. If that content changes to cloud-init.(network-config|vendor-data|user-data) keys we will surface those as well.

regardless of the user. Vendordata and userdata should be redacted from the non-root user.
Ahh right. need to make that change still in this PR.

Regardling the security.devlxd, I'm not sure there's much more we can do. If it's not set, we'll use the LXD datasource. If it is set, we'll use NoCloud.
I have documented that devlxd: true config is required. But I think this DS still needs to handle failure case across boot where devlxd is turned off. It still attempts to load the DS from cache and query instance data directly. If the socket file no longer exists (maybe due to a container being sandboxed for security triage or something) we would need to react with a warning in this case.

We have no mechanism to dynamically switch datasources, and I don't think we should. After first boot, we're not likely to see metadata changes anyway, so even if the socket gets shut off, I don't think it's necessarily catastrophic for the instance. I think we just need to document that setting the flag to true will force us to use the NoCloud datasource, and toggling the flag for an already running instance can lead to unexpected results.

+1

[TheRealFalcon]

also test comments

[TheRealFalcon]

Were you planning on parametrizing LXD_V1_METADATA? If not, these if statements seem pointless since it'll only ever be one value.

[blackboxsw]

Originally, yes. I think it's probably worth me parametrizing this test. will tackle tomorrow.

[blackboxsw]

strike that dropping parametrization for this pass. I'll add a followup branch with intrumented VM support.

[TheRealFalcon]

Should LXD be added here? https://github.com/canonical/cloud-init/blob/main/cloudinit/settings.py#L21

[blackboxsw]

Should LXD be added here? https://github.com/canonical/cloud-init/blob/main/cloudinit/settings.py#L21

yes it should and should have the same ordering we want LXD after NoCloud & ConfigDrive.

[TheRealFalcon]

There's still a couple of comments from my previous review that are unaddressed. I think the None checks and pep8 import stuff are minor enough we can do without, but I think we need to fix the wrong link in the docs (or tell me if I'm misunderstanding it).

[TheRealFalcon]

Did you see this one?

[TheRealFalcon]

Thanks!