feat: support for generating charmcraft YAML from Python classes #1975

tonyandrewmeyer · 2025-08-11T23:05:31Z

A new package, ops-tools is added. This currently only provides a script to update charmcraft.yaml based on config and action Python classes, but is designed to also offer other tools in the future. It is released simultaneously with the other ops-* packages, but is not required by any of the others, or offered as an extra.

Charmers can use the exported functions (which take a Python class object and return a dictionary suitable for serialisation to YAML), but are expected to use the provided script (that works with a charmcraft.yaml file and Python modules).

Classes can provide a to_juju_schema() method if they need to provide YAML in a different way. The script will pass the generated config/action as a base to optionally work from.

The implementation is designed to work with the load_config and load_params functionality in ops. In particular, that means working with four types of class:

A generic Python class (no inheritance other than object).
Standard library dataclasses (I personally believe these are the best choice for almost all cases).
Pydantic dataclasses
Pydantic 2.x BaseModel classes

Only reference documentation is included in this PR (Preview). The intention is that there will be a follow-up PR that includes at least how-to documentation.

More details are available in the spec. This PR replaces #1702.

… don't catch the error themselves.

…break the config into multiple classes.

…if the secret isn't available, but avoids Juju calls when loading config.

…id secret IDs.

In #1975, we'd like to include a couple of docstrings. This PR clears the way for that, as well as any other doctests we wish to use in the future. The PR tweaks some docstrings that are in private methods, so not in in our reference documentation. In addition: * `ActionEvent.set_results` changes from a block of `>>>` text to a set of bullet points. If this was doctest'ed then it'd actually call the method, but doing that with `set_results` itself would be complicated. This seems just as clear to me. * `pebble.Client.exec` has a couple of lines to disable doctest. This could be a doctest, but that would require a real Pebble, which we currently split off into a different set of tests. This is a bit ugly, but putting `doctest: +SKIP` on most of the lines is uglier (in my opinion), and I don't think there's a cleaner way to skip a single docstring. --------- Co-authored-by: Dave Wilding <tech@dpw.me>

tools/src/ops_tools/_update_charmcraft_yaml.py

james-garner-canonical

Partial review for now.

tools/README.md

james-garner-canonical · 2025-08-26T23:49:19Z

tools/pyproject.toml

+]
+
+[project.scripts]
+update-charmcraft-schema = "ops_tools._update_charmcraft_yaml:main"


This name is a bit confusing to me. I also wonder if subcommands might be a nicer way to handle this as we add more commands. How about if the script is ops-tools, and the functionality added in this PR is under ops-tools generate-metadata or similar?

I'm not that keen on having things as sub-commands under a single ops-tools command. For one thing, I don't like the duplication:

uvx --from=ops-tools ops-tools generate-metadata

But also I feel like ops-tools is meant to be a collection of tools, not a single tool with multiple subcommands.

However, I haven't finished going through the CLI spec in detail, so maybe it has recommendations here that should be followed.

I would like to resolve the comment around how this should work (just a diff/lint, merging, replacing) before locking down what the command looks like and is named.

Re: duplication, you shouldn't need any if the command name is the same as the package name, right? Just uvx ops-tools generate-metadata. Good call on figuring out the rest of the commandline interface first though, I agree that's more important, and I have some thoughts there too.

Leaving aside subcommands, since with uv that just means +/- the --from flag, I think the existing interface is nice overall. The main parts I find confusing are what will actually happen with the content we generate.

I'd suggest using mutually exclusive groups for the output options, and defaulting to a non-destructive output method like writing the generated content to stdout. How about:

--update (current behaviour, write to existing file)

--diff (current option)

--output=FILE (default behaviour, with - for stdout)
I'd also be happy with a --stdout flag instead of --output= if that's simpler.

The --merge flag option had me a little confused too, I thought it was the option that turned on the --update behaviour, but that's (currently) unconditional. Instead, it controls whether to keep actions/options that are in the charmcraft.yaml file but not present in our generated output, right? In that case, how about --delete / --no-delete flags for this (borrowing rsync semantics)?

Regarding the name, how about update-charmcraft-yaml instead? The term charmcraft schema seems a little off somehow. All together that gives us:

uvx {--from} ops-tools update-charmcraft-yaml --path=... # This might be nice as an optional positional argument instead? --config=... ... --action=... ... [--delete / --no-delete (default)] [--update / --diff / --output='-' (default)]

It also seems like it would be nice to try this out without consuming a charmcraft.yaml file at all. What do you think about additional entry points (scripts or subcommands) with much simpler interfaces that just write the actions/config options to stdout?

uvx {--from} ops-tools generate-actions CLASS... uvx {--from} ops-tools generate-config CLASS...

I'd suggest writing in the actions.yaml and config.yaml format, but there could always be a flag added to control that.

tools/src/ops_tools/_attrdocs.py

tools/src/ops_tools/_generate_yaml.py

james-garner-canonical · 2025-08-27T00:54:21Z

tools/src/ops_tools/_generate_yaml.py

+    # The many type: ignores are required because we don't want to import
+    # pydantic in this code.
+    options: dict[str, OptionDict] = {}
+    for name, field in cls.model_fields.items():  # type: ignore


This is as far as I've gotten in my review so far. I'm wondering here if we can reuse any of the logic already implemented in ops for extracting fields from classes, e.g. ops.charm._juju_fields.

Co-authored-by: James Garner <james.garner@canonical.com>

tools/src/ops_tools/_update_charmcraft_yaml.py

james-garner-canonical · 2025-08-29T02:06:10Z

tools/src/ops_tools/_generate_yaml.py

+        if hasattr(field.default, 'default_factory')
+        else field.default_factory
+    )
+    # A hack to avoid importing Pydantic here.


What if ops-tools just depended on Pydantic? We don't want to add a heavy dependency to ops itself, but would this all be a lot easier if we included it in ops-tools?

I'm pretty strongly opposed to pulling Pydantic in, even for ops-tools. It would be a bit easier, but not a lot, I feel.

May I ask why -- is it the extra runtime overhead for users?

james-garner-canonical

I've left some major and minor comments.

Major: I think the CLI interface is pretty nice, but what actually happens with the output could be made more clear. I'd like to see non-destructive default behaviour by default, and I'd also like to be able to dump a Python class to a Juju action/config definition without worrying about charmcraft.yaml at all.

More minor: while reading through the code, I've left a number of suggestions that I think might make some of the logic easier to follow. However ...

Big picture questions:

How much simpler would interacting with Pydantic model classes be if ops-tools could just import Pydantic?
- Additionally, if we're importing a module that defines a Pydantic class, won't we have to have Pydantic installed anyway?
- We could either add Pydantic to our own deps
- Or if we're relying on the user doing --with=pydantic or something, then we could move our Pydantic handling logic to a module that imports pydantic and only import that module as needed. This would have the benefit of letting us fudge around the Pydantic version, but seems more complicated.
I may be wrong about this, but my impression is that we're doing similar class traversal logic in a few different places for different purposes. This is some of the trickier code to follow imo. Would it be worthwhile to centralise this logic and parse the classes we're consuming into an intermediate data structure that has everything we want represented nicely?
- I also had kind of assumed that the core class parsing logic would be in ops (internals), and we'd use it in ops-tools, since we already parse classes for loading actions/config/relation data in ops. That wouldn't work if we went down the route of using Pydantic here, but perhaps something to keep in mind.

Co-authored-by: James Garner <james.garner@canonical.com>

dimaqq

First round of reviews.

dimaqq · 2025-08-12T05:51:39Z

tools/src/ops_tools/_attrdocs.py

+        last_attr = None
+        for child in node.body:
+            if isinstance(child, (ast.Assign, ast.AnnAssign)):
+                target = None  # Make the type checker happy.
+                if isinstance(child, ast.Assign):
+                    target = child.targets[0]
+                elif isinstance(child, ast.AnnAssign):
+                    target = child.target
+                assert isinstance(target, ast.Name)
+                last_attr = target.id


I would prefer to see unit tests for this class, the top-level function below in addition to the "functional" tests for YAML generation.

I've been bitten by AST before 🐍

Ouch I left this comment "Pending" on my side for three weeks!

dimaqq · 2025-08-12T23:47:30Z

tools/src/ops_tools/_update_charmcraft_yaml.py

+    with open(args.charmcraft_yaml, 'w') as raw:
+        yaml.safe_dump(charmcraft_yaml, raw)


If we want folks to use this feature, we could either:

advocate that charmers abandon separate config/actions/metadata.yaml (my pref), or

support updating both charmcraft.yaml and metadata.yaml, and autodetect which is present

P.S. I'm all for updating "in place", however we must use a round-trip safe yaml parser/serialiser and have tests to prove it, including YAML with comments in "inconvenient" places. I've run into that with the charm pin updater script, there are plenty of corner cases when key or content is interspersed with comments.

dimaqq · 2025-09-05T06:00:42Z

tools/src/ops_tools/_generate_yaml.py

+    default = None
+    for field in dataclasses.fields(cls):
+        if field.name != name:
+            continue
+        break
+    else:
+        return default


My 2c: if dataclasses doesn't provide a dict, a helper method that builds a dict is the most natural way to go.

tools/src/ops_tools/_update_charmcraft_yaml.py

dimaqq · 2025-09-05T06:09:47Z

tools/src/ops_tools/_update_charmcraft_yaml.py

+            exit_code += 2
+        sys.exit(exit_code)
+
+    if args.merge and 'config' in charmcraft_yaml:


question: Is this for 12-factor charms?

Personally I would like to avoid merge like a plague of rats gorged on plagued infested locusts who ate all the plagued corn...

12F could be solved by a stub / base class for options.

dimaqq · 2025-09-05T06:11:12Z

tools/src/ops_tools/_update_charmcraft_yaml.py

+        config = charmcraft_yaml['config']['options'].update(config['options'])
+    raw_yaml = _insert_into_charmcraft_yaml(raw_yaml, 'config', {'config': config})
+    if actions:
+        if args.merge and 'actions' in charmcraft_yaml:


Are there unit tests and functional / test vector tests for the merge flag?

dimaqq · 2025-09-05T06:13:07Z

tools/tests_tools/test_generate_yaml_config.py

+    generated_schema = ops_tools.config_to_juju_schema(config_class)
+    expected_schema = {
+        'options': {
+            'basic-bool': {
+                'type': 'boolean',
+            },


This is great, but omits YAML formatting.

There needs to be a test for verbatim YAML output.
I'm thinking a stability test perhaps (we may change the code, but the output keeps same "canonical" indent).

dimaqq · 2025-09-05T06:14:09Z

tools/README.md

+actions:
+    run-backup:
+        description: Backup the database.
+        params:
+            filename:
+                type: string
+                description: The name of the backup file.
+            compression:
+                type: string
+                description: The type of compression to use.
+                default: gzip
+                enum: [gzip, bzip2]
+        required: [filename]
+        additionalProperties: false


I'd like to see a test that looks exactly like this.

PietroPasotti · 2025-09-26T10:52:31Z

this brings back some memories: https://github.com/PietroPasotti/jinx

dimaqq

Partial review

dimaqq · 2025-09-29T07:36:28Z

test/charms/test_main/src/charm.py

 # during unit tests, and test_main failures that subprocess out are often
 # difficult to debug. Uncomment this line to get more informative errors when
 # running the tests.
+# When uncommented the test_hook_and_dispatch_with_failing_hook test will fail.


nit could be perhaps written as "test expected to fail if uncommented".

dimaqq · 2025-09-29T07:38:18Z

testing/src/scenario/state.py

+        # TODO: ideally, we would look for ConfigBase classes in the charm
+        # module and autoload from there at this point. Leaving this until the
+        # conversation about if & how the generation is done is resolved.
+


I'm not getting the point of this comment.
Specifically, if I were to read it a few years later, I would be missing the context.

Could future work be tracked in GitHub issues or Jira tickets or roadmap items instead?

dimaqq · 2025-09-29T07:42:50Z

testing/tests/test_e2e/test_config.py

+            framework.observe(self.on.config_changed, self._on_config_changed)
+
+        def _on_config_changed(self, event: ops.ConfigChangedEvent):
+            self.typed_config = self.load_config(Config, 10, c='foo')


Question this is unexpected to me.

I would have thought that default are specified in a model, not in the load config call site.

The defaults can be included in both dataclasses and pydantic models, so why add the extra arguments feature in the load_config method?

P.S. maybe I was lazy reading the spec, but somehow I don't recall this API 🙈

P.P.S. I guess it's because Juju action parameters don't have defaults like Juju config does, isn't it?

That also means that we should be testing both required and non-required parameters somewhere... perhaps a negative test in Scenario?

dimaqq · 2025-09-29T07:52:07Z

tools/tests_tools/test_generate_yaml_config.py

+    my_str: str = 'foo'
+    """A string value."""
+
+    my_secret: ops.Secret | None = None


Issues

What are the semantics for optional-Secret-typed fields?

I guess if a value is passed, the a secret with this id (not label) must exist, or load_config() fails.

And I guess on error, same errors = raise|blocked applies, doesn't it?

And if nothing is passed (None, not empty string), then field if None.

This needs to be documented somewhere.

The spec describes Secret fields, but not optional Secret fields.

The implementation remains the same, doesn't it?

option_type = self.meta.config.get(key) # Convert secret IDs to secret objects. We create the object rather # that using model.get_secret so that it's entirely lazy, in the # same way that SecretEvent.secret is. if option_type and option_type.type == 'secret': assert isinstance(value, str) # Juju will have made sure of this. value = model.Secret( backend=self.model._backend, id=value, _secret_set_cache=self.model._cache._secret_set_cache, ) config[attr] = value

The value seems mandatory?

Copilot

Pull Request Overview

This PR introduces a new ops-tools package that provides utilities for generating charmcraft.yaml configuration and action schemas from Python classes. The main contribution enables developers to maintain a single source of truth for charm configuration in Python code rather than duplicating information across Python and YAML files.

Key Changes

New ops-tools package with schema generation functionality from Python classes (dataclasses, Pydantic models, or plain classes)
Command-line tool update-charmcraft-schema for automated charmcraft.yaml updates
Integration with the workspace, build, and release processes

Reviewed Changes

Copilot reviewed 21 out of 24 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`tools/src/ops_tools/_generate_yaml.py`	Core schema generation logic for config and actions
`tools/src/ops_tools/_update_charmcraft_yaml.py`	CLI tool for updating charmcraft.yaml files
`tools/src/ops_tools/_attrdocs.py`	Extracts attribute docstrings from Python classes
`tools/tests_tools/test_generate_yaml_*.py`	Test coverage for schema generation
`uv.lock`	Dependency lock file updates (beautifulsoup4, coverage, lxml, etc.)
`pyproject.toml`	Workspace configuration for new tools package
`release.py`	Release automation for ops-tools

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-03T10:50:40Z

tools/src/ops_tools/_update_charmcraft_yaml.py

+        action='append',
+        help='Python class with optional module path (can be specified multiple times). '
+        'For example, "src.config:Config". The module defaults to "src.charm".'
+        'The class may be a regular expression.',


Missing space after the period. Should be 'The module defaults to "src.charm". The class may be a regular expression.'

Suggested change

'The class may be a regular expression.',

' The class may be a regular expression.',

Copilot · 2025-11-03T10:50:41Z

tools/src/ops_tools/_update_charmcraft_yaml.py

+        'For example, "src.backup:BackupAction". The module defaults to "src.charm".'
+        'The class may be a regular expression.',


Missing space after the period. Should be 'The module defaults to "src.charm". The class may be a regular expression.'

tools/tests_tools/test_generate_yaml_action.py

benhoyt · 2025-11-04T10:13:52Z

Tracking issue: #2163

tonyandrewmeyer added 30 commits March 29, 2025 19:50

Initial implementation.

4ee7643

Merge origin/main.

da8d265

Tox is green, including docs.

7f143cd

Add extra tests.

739c20c

Tox is green again.

2c9bb9f

Add a Raises section.

a1d4f74

Expand to-do list.

fac6c7c

Adjustments after initial review.

f155bf5

Provide pydantic for static checks.

ea10a4d

Changes from initial review.

bd2a2df

Merge from main.

eb2d6e1

Tweaks after merge.

35b643a

Add TODO.

441c68a

Remove ignore used during dev.

0966028

Add TODO note.

29875b1

Do the status set right at the end, so that it only happens if charms…

f13ed9d

… don't catch the error themselves.

Tweak comment.

4320d97

Use cleaner methods to get the set of fields.

84ba899

Only pass the config that the class defines. This makes it easier to …

b13c742

…break the config into multiple classes.

Provide lazy Secrets in the config - this means you get errors later …

30573f9

…if the secret isn't available, but avoids Juju calls when loading config.

With lazy Secrets, we no longer get errors with missing secrets/inval…

172c692

…id secret IDs.

Clean up the attr-to-type code.

3ee1807

Add a test for the 'only load some config' case.

5f8db50

Move TODO to a better location.

d932a97

Merge origin/main.

80dfa63

Add tests for uncaught InvalidSchemaError.

197431a

Fix docs.

d0b7e2e

Adjustments from spec review.

1c09153

Merge origin/main.

b716878

Provide Python 3.8 support.

97cd70a

tonyandrewmeyer added 4 commits August 15, 2025 11:12

Merge origin/main

eaa7476

Make the entry point name clearer.

b901a0a

Allow merging rather than replacing.

335c01c

Add a diff method.

71d14d6

tonyandrewmeyer mentioned this pull request Aug 15, 2025

ci: enable doctests #1991

Merged

tonyandrewmeyer added 3 commits August 15, 2025 13:47

WiP doctests.

a5b5dc5

Edit the charmcraft.yaml in a way that preserves comments and ordering.

b2efed5

Update for newer pyright.

b485a50

tonyandrewmeyer added 2 commits August 19, 2025 11:57

Merge origin/main.

47e425b

Fix order of output.

f888063

benhoyt reviewed Aug 20, 2025

View reviewed changes

tools/src/ops_tools/_update_charmcraft_yaml.py Outdated Show resolved Hide resolved

tools/src/ops_tools/_update_charmcraft_yaml.py Outdated Show resolved Hide resolved

tonyandrewmeyer requested a review from james-garner-canonical August 26, 2025 01:06

james-garner-canonical reviewed Aug 27, 2025

View reviewed changes

tonyandrewmeyer and others added 5 commits August 27, 2025 13:25

Update tools/README.md

36aa0e0

Co-authored-by: James Garner <james.garner@canonical.com>

Apply suggestions from code review

9044d61

Co-authored-by: James Garner <james.garner@canonical.com>

Tweak args, per review.

89cac58

Merge origin/main.

851956c

Apply suggestions from code review

181ce73

Co-authored-by: James Garner <james.garner@canonical.com>

james-garner-canonical reviewed Aug 29, 2025

View reviewed changes

tools/src/ops_tools/_update_charmcraft_yaml.py Outdated Show resolved Hide resolved

james-garner-canonical reviewed Aug 29, 2025

View reviewed changes

tools/src/ops_tools/_update_charmcraft_yaml.py Outdated Show resolved Hide resolved

james-garner-canonical reviewed Aug 29, 2025

View reviewed changes

james-garner-canonical requested changes Aug 29, 2025

View reviewed changes

Apply suggestions from code review

3d26c4b

Co-authored-by: James Garner <james.garner@canonical.com>

dimaqq reviewed Sep 5, 2025

View reviewed changes

dimaqq reviewed Sep 29, 2025

View reviewed changes

tonyandrewmeyer requested a review from Copilot November 3, 2025 10:47

Copilot AI reviewed Nov 3, 2025

View reviewed changes

		with open(args.charmcraft_yaml, 'w') as raw:
		yaml.safe_dump(charmcraft_yaml, raw)

	'The class may be a regular expression.',
	' The class may be a regular expression.',

		'For example, "src.backup:BackupAction". The module defaults to "src.charm".'
		'The class may be a regular expression.',

feat: support for generating charmcraft YAML from Python classes #1975

Are you sure you want to change the base?

feat: support for generating charmcraft YAML from Python classes #1975

Uh oh!

Conversation

tonyandrewmeyer commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

james-garner-canonical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-garner-canonical left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimaqq left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PietroPasotti commented Sep 26, 2025

Uh oh!

dimaqq left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimaqq Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tonyandrewmeyer commented Aug 11, 2025 •

edited

Loading

james-garner-canonical left a comment •

edited

Loading

dimaqq Sep 29, 2025 •

edited

Loading