Add pants-plugins/schemas to streamline regenerating contrib/schemas

[cognifloyd]

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

• pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
• pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

This PR improves the DX (developer experience) by wiring up the fmt and lint goals to generate contrib/schemas/*.json if needed (or for lint, warn that it needs to be done).

This includes creating a schemas plugin for pants that adds a schemas target.

Developer Experience

Today, if someone changes one of the files used to generate the schemas in contrib/schemas, they will probably forget to run the Makefile target the regenerates them: make schemasgen.

I've missed running that several times. If I'm lucky, CI complains with an error telling me to run that. I've seen CI logs print an error without actually failing the build. I believe that particular Makefile bug is fixed, but that anecdote highlights a usability issue: friction around generated files.

With this PR, we add schema generation into the fmt goal. This also adds it to the lint goal which will tell people if fmt needs to run to update any code (or in this case, regenerate the schema). Then, we only need simple instructions that say something like:

Please run ./pants fmt lint :: before submitting your PR. This will reformat code, if needed, and warn you about any other errors.

Fine grained results cache

After someone runs ./pants fmt :: once, they will benefit from the pants' results cache. For the schemas plugin, here's what that means:

First, the schemas target in contrib/schemas depends on st2common/bin/st2-generate-schemas:

st2/contrib/schemas/BUILD

Lines 1 to 5 in d4fba56

	schemas(
	dependencies=[
	"st2common/bin/st2-generate-schemas",
	],
	)

If the st2-generate-schemas script changes, then pants now knows that the schemas need to be regenerated. Or, in other words, the st2-generate-schemas file is part of the cache key for contrib/schemas/*.json. If that file changes, then pants will re-run the generator.

st2-generate-schemas is a python script, and we use pants' python dependency inference. So pants (effectively) also adds the python files imported by st2-generate-schemas to the cache key. Stepping through the code, we can see what pants would infer:

st2/st2common/bin/st2-generate-schemas

Line 32 in d4fba56

from st2common.cmd import generate_schemas

st2/st2common/st2common/cmd/generate_schemas.py

Lines 26 to 30 in d4fba56

	from st2common.models.api import action as action_models
	from st2common.models.api import pack as pack_models
	from st2common.models.api import policy as policy_models
	from st2common.models.api import rule as rule_models
	from st2common.models.api import sensor as sensor_models

Here we reach the primary definition for the schemas that end up in contrib/schemas: the definition comes from the st2common.model.api.* modules. Now, if anyone changes any of these files, or any of the python bits that get imported to help define things in them, then the next time someone runs the lint goal (or in CI) they will find out that they need to run the fmt goal.

This PR actually only wires up the fmt goal and took advantage of the fact that pants also runs formatters whenever it runs linters, which are side-effect free. And because pants runs the formatters in a sandbox, it doesn't have to materialize any changes during lint.

Continuing the example, since pants cached the results when running the lint goal, running fmt will be very fast. Pants will just materialize the already-generated schemas from its cache.

Developing pants plugins

Pants has extensive documentation on the plugin API, including how to create targets, how to write rules, and there's even a mini tutorial about how to add a formatter (adding formatters is a very common thing for plugins to do).

• Plugins overview
• The Target API
• The Rules API
• Processes (running subprocesses in pants plugins)
• Add a formatter
• Testing Plugins

Why not codegen?

Pants has something called codegen. But it is for code that gets generated without getting checked into the repo. You can export those changes, but they are always exported in a directory under dist/, so we would have to add some custom logic somewhere that copies from dist/ to the contrib/schemas directory.

These schemas are only available in this repo; if someone wants to use them, they have to clone the repo or grab the raw file from github. So, they have to be checked into the repo. We might be able to do something different in the future, but I'm trying not to change more than I have to to introduce pants.

[cognifloyd]

We are using st2common.cmd.generate_schemas:main directly instead of using the st2-generate-schemas script because pants can't run python with a - in the name (because pants relies on importing from the target script, and python cannot import from files with a - in the name).

Eventually I'll get around to plumbing a change in pants to make that possible (it already works with the underlying pex, but pants hasn't learned about that arg yet). In the meantime, we just use the entrypoint itself instead of the script.

[rush-skills]

LGTM