Fix error on upgrade caused by new vendordata2 attributes

[TheRealFalcon]

Proposed Commit Message

Fix error on upgrade caused by new vendordata2 attributes

In #777, we added 'vendordata2' and 'vendordata2_raw' attributes to
the DataSource class, but didn't use the upgrade framework to deal
with an unpickle after upgrade. This commit adds the necessary
upgrade code.

Additionally, added a smaller-scope upgrade test to our integration
tests that will be run on every CI run so we catch these issues
immediately in the future.

LP: #1922739

Additional Context

We haven't caught this is our current upgrade test because we haven't been checking for error status of cloud-init status. I added a check for that in the larger upgrade test as well.

Test Steps

Since the latest cloud-init version is already in LXD images, it's not as straightforward. You can launch a xenial instance and then upgrade to the latest PPA/deb. For other releases, you can launch an instance, manually downgrade to 20.4.1, run a cloud-init clean, then upgrade to latest. I've manually reproduced the bug both ways and manually verified the bug both ways.

I also locally reproduced/verified the new integration test by launching the test against a xenial instance (CI uses focal).

[realizelol]

Thank you,

this fix is definitvely working.

[OddBloke]

We're missing some non-integration testing here, and I have concerns about how we got here, but this otherwise looks pretty good to me; I'm going to run the test locally now.

[OddBloke]

Should we promote _get_initial_image to the public API?

(Side note, which I noticed when reading _get_initial_images code: should we be caching that value so that we avoid getting different "initial" images if we call that before and after an image publication?)

[TheRealFalcon]

What do you mean by before and after an image publication?

Although, now that I look at it again, should I just make released_image_id public and use that instead?

[OddBloke]

What do you mean by before and after an image publication?

We'll call _get_initial_image once when session_cloud is first instantiated (in the entire test run), and once again here: if a release image is released between those two calls, we'll run this test and the other tests against different images.

Although, now that I look at it again, should I just make released_image_id public and use that instead?

Aha, yeah, I guess we are caching the value already! 👍

[OddBloke]

I'm going to run the test locally now.

I've confirmed that the test in this branch fails when run against an old focal image with a deb built from main:

$ CLOUD_INIT_OS_IMAGE=ubuntu:e0c3495ffd48::ubuntu::focal CLOUD_INIT_CLOUD_INIT_SOURCE=./cloud-init_21.1-34-g45db197c-1\~bddeb\~20.04.1_all.deb pytest tests/integration_tests/test_upgrade.py
...
FAILED tests/integration_tests/test_upgrade.py::test_upgrade_package - assert False

and passes with a deb built from this branch:

$ CLOUD_INIT_OS_IMAGE=ubuntu:e0c3495ffd48::ubuntu::focal CLOUD_INIT_CLOUD_INIT_SOURCE=./cloud-init_21.1-36-g5933b497-1\~bddeb\~20.04.1_all.deb pytest tests/integration_tests/test_upgrade.py
...
PASSED

[TheRealFalcon]

@OddBloke I believe I addressed your comments. I had a question/comment about the unit test.

I added a new test. It failed without this branch against the already existing pickles. Is there a reason to add a new pickle? I'm not sure I understand the reason for adding new pickles provided we start at a sufficiently old starting point.

[OddBloke]

Thanks!