✨ Generate schema for types with custom JSON marshaling as Any type by skaslev · Pull Request #427 · kubernetes-sigs/controller-tools

skaslev · 2020-04-01T07:21:39Z

Currently controller-gen complains about structs with missing json tags even
when those structs implement custom JSON marshalling.

With this change we check if a type implements custom JSON marshalling and if it
does, we output schema for Any type. This still allows the validation type to
overriden with a marker.

Fixes #391

k8s-ci-robot · 2020-04-01T07:21:41Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2020-04-01T07:21:46Z

Welcome @skaslev!

It looks like this is your first PR to kubernetes-sigs/controller-tools 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/controller-tools has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

skaslev · 2020-04-01T07:54:43Z

/assign @DirectXMan12

skaslev · 2020-04-01T08:06:51Z

/retest

k8s-ci-robot · 2020-04-01T08:07:04Z

@skaslev: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

skaslev · 2020-04-01T08:53:43Z

/check-cla

DirectXMan12

can you add a test case for this (just add something to the big integration test)? Otherwise LGTM

DirectXMan12 · 2020-04-18T00:31:31Z

Thanks! We've wanted something like this for a while.

yuzisun · 2020-04-26T18:35:02Z

@skaslev any chance to update the test and get this in? thank you!

DirectXMan12 · 2020-04-27T21:11:07Z

@yuzisun (or anyone else) -- if skaslev doesn't get back to us and you feel like writing the tests, feel free to add an additional commit and submit a new PR (and link to this one).

skaslev · 2020-05-05T10:41:26Z

@DirectXMan12 I added two test cases similar to the problematic types in Knative APIs. The initial version of this PR wasn't entirely correct since it wasn't always applying the markers to the schema for types with custom json serialization. I think the new version works correctly.

Can you take another look?

Currently controller-gen complains about structs with missing json tags even when those structs implement custom JSON marshalling. With this change we check if a type implements custom JSON marshalling and if it does, we output schema for Any type. This still allows the validation type to be overriden with a marker.

This reverts commit 820a4a2, reversing changes made to 67819ee.

To generate CRD validation schemas (also known as structual validation schemas[1]), we bring in controller-tools. This houses a tool called controller-gen which performs various K8s related code generation. Note that we are using a fork of controller-tools due to the upstream (as of v0.3.0) lacking support in two main areas. 1) A type which implements custom JSON marshalling has its validation schema generation skipped. In our case, api.Rule (which is set as the Spec field in CNP and CCNP) implements custom JSON marshalling. With the upstream tool, this skips the validation generation for it. In our fork, we revert this feature. See PR: kubernetes-sigs/controller-tools#427 2) For CNP and CCNP, we require `oneOf` validation for the endpointSelector and nodeSelector fields. This is because we want to validate that either endpointSelector OR nodeSelector is present, but not both, and at least one. In the upstream, there's currently an open PR that implements support for this. We have manually merged it in our fork. See PR: kubernetes-sigs/controller-tools#298 This commit also brings updates to our dependencies: - gomega v1.7.0 -> v1.8.1 - golang.org/x/internal/{tools,event...} [1]: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema Co-authored-by: Chris Tarazi <chris@isovalent.com> Signed-off-by: André Martins <andre@cilium.io> Signed-off-by: Chris Tarazi <chris@isovalent.com>

This reverts commit 820a4a2, reversing changes made to 67819ee.

vincepri · 2020-10-05T18:44:24Z

@DirectXMan12 @skaslev Using the tip of the main branch, I'm now getting an invalid CRD:

  CustomResourceDefinition.apiextensions.k8s.io "kubeadmcontrolplanes.controlplane.cluster.x-k8s.io" is invalid: spec.validation.openAPIV3Schema.properties[spec].properties[kubeadmConfigSpec].properties[initConfiguration].properties[bootstrapTokens].items.properties[token].type: Unsupported value: "Any": supported values: "array", "boolean", "integer", "number", "object", "string"

vincepri · 2020-10-05T18:46:50Z

We should improve the error message saying that when using this capability, folks should manually specify the validation type

alvaroaleman · 2020-10-12T17:59:45Z

Any is just not valid. If folks have a custom marshaller, they must specify the json tag or use a kubebuilder annotation. This PR doesn't fix the issue it claims to fix while introducing a new issue: The schema we generate for types with a custom marshaller and without a kubebuilder annotation is now always invalid. Revert in #505

k8s-ci-robot added the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Apr 1, 2020

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 1, 2020

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Apr 1, 2020

k8s-ci-robot requested review from droot and mengqiy April 1, 2020 07:21

skaslev changed the title ~~🐛 Generate schema for types implementing custom JSON marshaling as Any type~~ 🐛 Generate schema for types implementing custom JSON marshaling as Any type Apr 1, 2020

k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Apr 1, 2020

skaslev changed the title ~~🐛 Generate schema for types implementing custom JSON marshaling as Any type~~ 🐛 Generate schema for types with custom JSON marshaling as Any type Apr 1, 2020

k8s-ci-robot assigned DirectXMan12 Apr 1, 2020

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Apr 1, 2020

skaslev mentioned this pull request Apr 1, 2020

Incorporate schema into CRDs knative/serving#912

Closed

yoichiwo7 mentioned this pull request Apr 6, 2020

markers to skip fields that does not have json tags for crd generation #391

Closed

DirectXMan12 suggested changes Apr 18, 2020

View reviewed changes

DirectXMan12 mentioned this pull request Apr 20, 2020

controller-gen doesn't work with k8s.io/apiextensions-apiserver #291

Closed

abhinavdahiya mentioned this pull request Apr 28, 2020

Bug 1830270: cmd: add explain subcommand openshift/installer#3515

Merged

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 5, 2020

skaslev requested a review from DirectXMan12 May 5, 2020 10:38

skaslev deleted the custom-json branch May 19, 2020 11:59

DirectXMan12 mentioned this pull request Jul 16, 2020

Escape hatch for CRD validation kubernetes-sigs/kubebuilder#301

Closed

jlewi mentioned this pull request Jul 16, 2020

[Model Sync Controller] Remove post processing of the kubebuilder CRD kubeflow/code-intelligence#172

Open

christarazi added a commit to christarazi/controller-tools that referenced this pull request Jul 17, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

05b7bd0

This reverts commit 820a4a2, reversing changes made to 67819ee.

christarazi added a commit to christarazi/controller-tools that referenced this pull request Jul 17, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

03c74a8

This reverts commit 820a4a2, reversing changes made to 67819ee.

christarazi added a commit to christarazi/controller-tools that referenced this pull request Jul 17, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

5c51fa4

This reverts commit 820a4a2, reversing changes made to 67819ee.

christarazi added a commit to christarazi/controller-tools that referenced this pull request Jul 17, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

6c2a40c

This reverts commit 820a4a2, reversing changes made to 67819ee.

christarazi added a commit to christarazi/controller-tools that referenced this pull request Jul 17, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

475b26a

This reverts commit 820a4a2, reversing changes made to 67819ee.

christarazi mentioned this pull request Jul 28, 2020

Custom JSON marshaling causes schema validation generation to be skipped #466

Closed

christarazi mentioned this pull request Jul 31, 2020

Network policies for the host endpoint cilium/cilium#11507

Merged

christarazi added a commit to christarazi/controller-tools that referenced this pull request Aug 14, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

7956d02

This reverts commit 820a4a2, reversing changes made to 67819ee.

christarazi added a commit to christarazi/controller-tools that referenced this pull request Sep 4, 2020

Revert "Merge pull request kubernetes-sigs#427 from skaslev/custom-json"

89aa3e5

This reverts commit 820a4a2, reversing changes made to 67819ee.

muvaf mentioned this pull request Oct 20, 2020

Have controllers deal in v1 CRDs crossplane/crossplane#1861

Merged

2 tasks

alvaroaleman mentioned this pull request Aug 27, 2021

🐛 Allow setting the type for structs that implement json.Marshaler #601

Merged

Conversation

skaslev commented Apr 1, 2020

Uh oh!

k8s-ci-robot commented Apr 1, 2020

Uh oh!

k8s-ci-robot commented Apr 1, 2020

Uh oh!

skaslev commented Apr 1, 2020

Uh oh!

skaslev commented Apr 1, 2020

Uh oh!

k8s-ci-robot commented Apr 1, 2020

Uh oh!

skaslev commented Apr 1, 2020

Uh oh!

DirectXMan12 left a comment

Choose a reason for hiding this comment

Uh oh!

DirectXMan12 commented Apr 18, 2020

Uh oh!

yuzisun commented Apr 26, 2020

Uh oh!

DirectXMan12 commented Apr 27, 2020

Uh oh!

skaslev commented May 5, 2020

Uh oh!

vincepri commented Oct 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vincepri commented Oct 5, 2020

Uh oh!

alvaroaleman commented Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vincepri commented Oct 5, 2020 •

edited

Loading

alvaroaleman commented Oct 12, 2020 •

edited

Loading