[BUG] Can't apply a resource if it's too big

[ryanzhang-oss]

Describe the bug

The three way merge annotation hits the annotation length limit

'Failed to apply manifest: Secret "sh.helm.release.v1.ryan-redis.v1"
          is invalid: metadata.annotations: Too long: must have at most 262144 bytes'

Environment

Please provide the following:

• Hub cluster details
• Member cluster details

To Reproduce

Follow the steps of the redis install guide

Expected behavior

The secret should be omitted as it's created by the helm. This bug actually makes it work.

Screenshots

 status:
    conditions:
    - lastTransitionTime: "2022-09-16T18:57:06Z"
      message: Successfully scheduled resources for placement
      observedGeneration: 1
      reason: ScheduleSucceeded
      status: "True"
      type: Scheduled
    - lastTransitionTime: "2022-09-16T21:14:48Z"
      message: there are failed to apply manifests, please check the `failedResourcePlacements`
        status
      observedGeneration: 1
      reason: ApplyFailed
      status: "False"
      type: Applied
    failedPlacements:
    - clusterName: member1
      condition:
        lastTransitionTime: "2022-09-16T21:14:47Z"
        message: 'Failed to apply manifest: Secret "sh.helm.release.v1.ryan-redis.v1"
          is invalid: metadata.annotations: Too long: must have at most 262144 bytes'
        reason: appliedManifestFailed
        status: "False"
        type: Applied
      kind: Secret
      name: sh.helm.release.v1.ryan-redis.v1
      namespace: redis-test
      version: v1
    - clusterName: member2
      condition:
        lastTransitionTime: "2022-09-16T21:14:46Z"
        message: 'Failed to apply manifest: Secret "sh.helm.release.v1.ryan-redis.v1"
          is invalid: metadata.annotations: Too long: must have at most 262144 bytes'
        reason: appliedManifestFailed
        status: "False"
        type: Applied
      kind: Secret
      name: sh.helm.release.v1.ryan-redis.v1
      namespace: redis-test
      version: v1

Additional context

Add any other context about the problem here.

[Arvindthiru]

Potential approaches:

• kubectl apply - Hard limit of 256 KB for all annotations combined
• compression - try different algorithms and document results
• external storage - running as a pod (did not explore since config map is a better k8s native option)
• config map - use config map in place of the large annotation

Compression:

Go library: https://pkg.go.dev/compress@go1.19.3

The go compress package has 4 sub packages which provides functionality to compress and decompress, and among the 4 the flate package the deflate algorithm is used by gzip & zlib which are different formats for compressed files and lzw seems to be a dictionary based compression algorithm.

Simple test to compress base 64 encoded string of the secret created by redis helm chart, the default compression rate seems to produce the best result for each package

After compressing the size is still large the packages on average seem to reduce the size by 80-120KB (Original size 364 KB). This may be because I'm using base 64 encoded string of the secret which is difficult to compress cause it introduces randomness in the encoded string.

Pros:

• Compress the annotation value to be less the 256KB.

Cons:

• Unreliable compression rates since it depends on the file that needs to compressed.
• Need to tweak many options to get the best compression rates.

Config map:

Even though the config map's max limit is 1 MB trying to kubectl apply(client side apply) a config map which holds the base 64 encoded secret as a value causes a similar error but now it's due to last applied configuration annotation used by k8s then I tried a server side apply of the same config map which succeeded,

But when creating the config map from the controller we wouldn't encounter any issue because client side apply won't be used so no need to use server side apply.

Pros:

• k8s native method to store large values

Cons:

• Has a maximum limit of 1MB

Result:

The config map approach seems to be the best bet as it provide a k8s native approach to store large values which could not be stored as an annotation. But a key thing to keep in mind is that the config map has a max limit of 1MB if the manifest size somehow so large that it exceeds this limit we may run into a similar error.

If we consistently run into this the 1MB limit issue for the config map after we make the proposed change in the future we may have to look into using a third party storage solution running as a pod on the hub cluster, but doing this introduces an extra dependency for us to manage.

Questions:

How is the create in applyUnstructured method in apply controller running into annotation limit ? (Is it using client-side apply)

Answer: It's not using client side apply but we are doing something similar to kubectl by tracking the last applied configuration to perform the three way merge patch when updating manifests.

Useful links:

• Use of last applied configuration in k8s https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-to-create-objects
• Using server side apply https://kubernetes.io/docs/reference/using-api/server-side-apply/#upgrading-from-client-side-apply-to-server-side-apply
• https://stackoverflow.com/questions/20762094/how-are-zlib-gzip-and-zip-related-what-do-they-have-in-common-and-how-are-they?rq=1