build: enable hermetic library generation

[diegomarquezp]

This is part of the effort to enable multi-version support in hand-written libraries. More context and details in this doc.

This PR adds two workflows and their scripts:

• update_generation_config which updates generation_config.yaml (added in this PR) to contain the latest googleapis_committish (source of truth of library generation) as well as other dependencies such as gapic_generator_version. This job runs nightly and creates a PR that modifies the config yaml.
• hermetic_library_generation which reads generation_config.yaml and pushes a new commit to any PR whose latest commit affects generation_config.yaml (e.g. the nightly PR created by update_generation_config). It essentially translates changes in the config yaml to changes in the source code (of the generated part). This workflow will be a required check.

As an example of how they work in practice:

1. Once per night, update_generation_config creates a PR that updates generation_config.yaml with the latest dependencies and googleapis committish
2. hermetic_library_generation is triggered by this PR and generates the corresponding code changes. A new commit with the code changes is added to the PR. This step may occur for any PR affecting generation_config.yaml.

Follow ups

• Add CLOUD_JAVA_BOT_TOKEN to repository for the hermetic generation workflow to work. We confirmed it works in a fork (job link).

[conventional-commit-lint-gcf]

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

[BenWhitehead]

Will libraries_bom_version and gapic_generator_version be updated by renovate bot, or the update_generation_config action?

[diegomarquezp]

They will be updated by update_generation_config

java-storage/.github/scripts/update_generation_config.sh

Lines 97 to 103 in 68ee05a

	# update gapic-generator-java version to the latest
	latest_version=$(get_latest_released_version "com.google.api" "gapic-generator-java")
	update_config "gapic_generator_version" "${latest_version}" "${generation_config}"
	# update libraries-bom version to the latest
	latest_version=$(get_latest_released_version "com.google.cloud" "libraries-bom")
	update_config "libraries_bom_version" "${latest_version}" "${generation_config}"

[BenWhitehead]

Could that cause any issues if the version in the library or samples updates via renovate-bot first?

[blakeli0]

libraries_bom_version is only used to generate the version in README by the hermetic build scripts, it should not cause any conflicts with the libraries bom version in samples.

There is no gapic_generator_version in pom, I guess you mean sdk-patform-java-config, which includes gax changes? If yes, this is similar to what we have right now with owlbot PRs.
The owlbot PR changes from the generator update may require gax changes. Similarly, the Cloud Java Bot PR may require us to merge the sdk-patform-java-config update first. But there should not be any issues the other way around, we should be free to merge sdk-patform-java-config update anytime, without waiting on Cloud Java Bot PRs created from the gapic_generator_version update.

[BenWhitehead]

Thanks for clarifying

[BenWhitehead]

When we make branches for lts releases will this need to be manually changed?

[blakeli0]

It depends on the type of LTS.

• If it is an LTS that does not need regular updates to googleapis_commitish or generator version, basically the LTS branches we current have. Then we would not set up any automation for it, and only change the generator version when there is a critical bug fix.
• If it is an LTS that requires regular updates, for example, if we want to introduce a Java 21 branch, then yes we will need to update this config on the Java 21 branch.

[BenWhitehead]

I'm okay with the changes as outlined here. How do we get the status check to actually turn green?

[diegomarquezp]

I'm okay with the changes as outlined here. How do we get the status check to actually turn green?

@BenWhitehead We confirmed it works in a diegomarquezp#10 (diegomarquezp#10).

It requires CLOUD_JAVA_BOT_TOKEN to be added, which is a follow up item.