[enhancement](schema.py): Add support for the "not" keyword in json schema

[mostafaCamel]

Enhancement

Problem

Currently ,schema.py in cloud-init does not provide support for the not keyword.

How it was identified

While working on #6070 #6370 and unit-testing changes made to schema-cloud-config-v1.json (changes included using the not keyword for the first time in this json schema) , the function validate_cloudconfig_schema failed with

.tox/py3/lib/python3.12/site-packages/jsonschema/_keywords.py:334: in allOf
    yield from validator.descend(instance, subschema, schema_path=index)
.tox/py3/lib/python3.12/site-packages/jsonschema/validators.py:432: in descend
    for error in errors:
                 ^^^^^^
.tox/py3/lib/python3.12/site-packages/jsonschema/_keywords.py:377: in not_
    if validator.evolve(schema=not_schema).is_valid(instance):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cloudinit/config/schema.py:504: in is_valid
    validator = self.evolve(schema=_schema)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
.tox/py3/lib/python3.12/site-packages/jsonschema/validators.py:342: in evolve
    NewValidator = validator_for(schema, default=self.__class__)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cause

The overriden method is_valid is lacking a condition (present in the original method in the jsonschema package) on whether or not to call .evolve when schema is None

Given that cloud-init is also doing an overriding method "is_valid_pre_4_0_0" (here) , I checked the last pre-4.0 version in the jsonschema library and the is_valid in this old version is not using this condition so is_valid_pre_4_0_0 in cloud-init does not need code changes.

I aso confirmed that is_valid in jsonschema v4.0.0 does have the condition

Proposed fix

Something like this

--- a/cloudinit/config/schema.py
+++ b/cloudinit/config/schema.py
@@ -500,11 +500,15 @@ def get_jsonschema_validator():
 
         It does ignore instances of `SchemaDeprecationError`.
         """
+        if _schema is None:
+            validator = self
+        else:
+            validator = self.evolve(schema=_schema)
         errors = filter(
             lambda e: not isinstance(  # pylint: disable=W1116
                 e, SchemaDeprecationError
             ),
-            self.evolve(schema=_schema).iter_errors(instance),
+            validator.iter_errors(instance),
         )
         return next(errors, None) is None
 

[mostafaCamel]

Confirming the compatibility of the not keyword in cloud-init

• cloud-init uses jscon schema Draft4 version (The schema version is different from the jsonschema pypi package version
• The current jsonschema package version definitely has support for the not keyword
• The v2.0.0 jsonschema package version (released in 2015) has support for the not keyword in Draft4
• Python 3.6.0final was released in 2016 . New cloud-init packagaes are backward compatible all the way to python 3.7

So in case a user somehow still has a 10-year-old jsonschema package , they should not get an error because of me introducing the not keyword in a cloud-init schema

[mostafaCamel]

#6469

[blackboxsw]

Great analysis here. Thank you for the detail and references.