Fall back to cached local ds if no valid ds found

[PengpengSun]

Proposed Commit Message

Fall back to cached local ds if no valid ds found

Rebooting an instance which has finished VMware guest
customization with DataSourceVMware will load
DataSourceNone due to metadata is NOT available.

This is mostly a re-post of PR#229, few differences are:
1. Let ds decide if fallback is allowed, not always fall back
   to previous cached LOCAL ds.
2. No comparing instance-id of cached ds with previous instance-id
   due to I think they are always identical.

Fixes GH-3402 (GitHub Issue number. Remove line if irrelevant)
LP: #1835205 (Launchpad bug number. Remove line if irrelevant)

Additional Context

Fixes #3402

Test Steps

Checklist

• My code follows the process laid out in the documentation
• I have updated or added any unit tests accordingly
• I have updated or added any documentation accordingly

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[holmanb]

When this happens, does ds-identify still correctly identify the platform?

[holmanb]

Could we make this method name more descriptive please? Also we should add a docstring that will communicate to other datasource authors what the purpose of this method is. There are many fallbacks in cloud-init, so this isn't very meaningful on its own.

[PengpengSun]

Thanks for the review.
Yes, I've renamed this method(please suggest a better name if needed), and added a docstring.

[PengpengSun]

When this happens, does ds-identify still correctly identify the platform?

Not sure if I get your question correctly, with my understanding, ds-identify identifies the platform before init --local, so this shall not have impact on ds identify process. The purpose of this change is to add an opportunity for the cached ds to be restored after no data can be gotten from ds in the ds list.

[PengpengSun]

Hi @holmanb , I'm removing WIP from title, please kindly review this again and let me know if any updates needed. Thanks!

[holmanb]

This is undesirable for two reasons:

• old pickles will never acquire this attribute
• encapsulation: we shift pickle handling outside of the class definition which should handle pickle issues directly

How about the following modification?

diff --git a/cloudinit/distros/__init__.py b/cloudinit/distros/__init__.py
index 2c4d54871..e225e4ee8 100644
--- a/cloudinit/distros/__init__.py
+++ b/cloudinit/distros/__init__.py
@@ -185,6 +185,8 @@ class Distro(persistence.CloudInitPickleMixin, metaclass=abc.ABCMeta):
             self._dhcp_client = None
         if not hasattr(self, "_fallback_interface"):
             self._fallback_interface = None
+        if not hasattr(self, "check_if_fallback_is_allowed"):
+            self.check_if_fallback_is_allowed = lambda: False
 
     def _validate_entry(self, entry):
         if isinstance(entry, str):
diff --git a/cloudinit/stages.py b/cloudinit/stages.py
index 13bc26d26..c228805d1 100644
--- a/cloudinit/stages.py
+++ b/cloudinit/stages.py
@@ -378,11 +378,7 @@ class Init:
                 if existing != "check":
                     raise e
                 ds = self._restore_from_cache()
-                if (
-                    ds
-                    and hasattr(ds, "check_if_fallback_is_allowed")
-                    and ds.check_if_fallback_is_allowed()
-                ):
+                if ds and ds.check_if_fallback_is_allowed():
                     LOG.info(
                         "Restored fallback datasource from checked cache: %s",
                         ds,

[PengpengSun]

Hi @holmanb,
Thanks for the review! I've updated the code accordingly(I think it'd be sources/init.py), PTAL.

[PengpengSun]

tox is not happy with the last commit, need to fix it.

[PengpengSun]

Hi @holmanb , I've updated code again which fixed the test error, PTAL, thanks!

[holmanb]

LGTM, thanks @PengpengSun!

[ani-sinha]

@holmanb is it possible to backport this patch to 24.1 train maybe in 24.1.5?