-
Notifications
You must be signed in to change notification settings - Fork 742
design: add wasm extension supports OCI image code source #3313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
4e6eea0
desing docs for wasm oci support
zhaohuabing 1181ca4
fix lint
zhaohuabing eaff3fa
Update site/content/en/contributions/design/wasm-extension.md
zhaohuabing 492c33e
minor wording
zhaohuabing 2ac9e8f
authn consideration
zhaohuabing b727e9f
address comments
zhaohuabing 491ba0b
minor wording
zhaohuabing e48ad93
minor wording
zhaohuabing a0cc42c
minor wording
zhaohuabing 4f2ad71
minor wording
zhaohuabing 5bca6cc
restrict access to priave images
zhaohuabing 7069a9f
Merge branch 'main' into wasm-design
zhaohuabing 6d76462
minor change
zhaohuabing 6e72cd6
move image to /img
zhaohuabing e5a9e4e
Merge branch 'main' into wasm-design
zirain f70cbda
Merge branch 'main' into wasm-design
guydc 0ba441c
Merge branch 'main' into wasm-design
zirain File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| --- | ||
| title: "Wasm OCI Image Support" | ||
| --- | ||
|
|
||
| ## Motivation | ||
| Envoy Gateway (EG) should support Wasm OCI image as a remote wasm code source. | ||
| This feature will allow users to deploy Wasm modules from OCI registries, such as Docker Hub, | ||
| Google Container Registry, and Amazon Elastic Container Registry, to Envoy proxies managed by EG. | ||
| Deploying Wasm modules from OCI registries has several benefits: | ||
| * **Versioning**: Users can use the tag feature of the OCI image to manage the version of the Wasm module. | ||
| * **Security**: Users can use private registries to store the Wasm module. | ||
| * **Distribution**: Users can use the existing distribution mechanism of the OCI registry to distribute the Wasm module. | ||
|
|
||
| ## Goals | ||
| * Define the system components needed to support Wasm OCI images as remote Wasm code sources. | ||
|
|
||
| ## Architecture | ||
|
|
||
|  | ||
|
|
||
| ### Control Plane Wasm File Cache | ||
|
|
||
| Envoy lacks native OCI image support, therefore, EG needs to download Wasm modules from their original OIC registries, | ||
| cache them locally in the file system, and serve them to Envoy over HTTP. | ||
|
|
||
| #### HTTP Code Source | ||
|
|
||
| For HTTP code source, we have two options: serve Wasm modules directly from their | ||
| original HTTP URLs, or cache them in EG (as with OCI images). Caching both the HTTP Wasm modules and OCI | ||
| images inside EG can make UI consistent, for example, sha256sum can be calculated on the EG side and made | ||
| optional in the API. This will also make the Envoy proxy side more efficient as it won’t have to download the | ||
| Wasm module from the original URL every time, which can be slow. | ||
|
|
||
| #### Resource Consumption | ||
|
|
||
| Memory: Since we cache Wasm modules in the file system, we can optimize the memory usage | ||
| of the cache and the HTTP server to avoid introducing too much memory consumption by this feature. | ||
| For example, when receiving a Wasm pulling request form the Envoy, we can open the Wasm file and write | ||
| the file content directly to response, and then close the file. There won’t be significant memory consumption involved. | ||
|
|
||
| Disk: Though it's possible to mount a volume to the container for the Wasm file cache, the current implementation just | ||
| stores the Wasm files in the EG container’s file system. The disk space consumed by the cache is limited to 1GB by default, | ||
| it can be made configurable in the future. | ||
|
|
||
| #### Caching Mechanism | ||
|
|
||
| Cached files will be evicted based on LRU(Last recently used)algorithm. | ||
| If the image’s tag is latest, then they will be updated periodically. The cache clean and update periods | ||
| will be configurable. | ||
|
|
||
| #### Restrict Access to Private Images | ||
|
|
||
| * **Client Authn with mTLS:** To prevent unauthorized proxies from accessing the Wasm modules, the communication between | ||
| the Envoy and EG will be secured using mTLS. | ||
| * **User Authn with Registry Credentials:** To prevent unauthorized users from accessing the Wasm modules, the user who | ||
| creates the EEP must have the appropriate permissions to access the OCI registry. For example, if two users create EEPs | ||
| in different namespaces (ns1, ns2) accessing the same OCI image, each must also create a unique secret with registry | ||
| credentials (secret1 for user1 in ns1, secret2 for user2 in ns2) and provide it in the EEP configuration. EG will validate | ||
| the provided secret against the OCI registry before serving the Wasm module to the target HTTPRoute/Gateway of that EEP. | ||
| * **Unguessable Download URLs:** It's possible that users who have no permission to access a private OCI image could create | ||
| an EnvoyPatchPolicy to bypass the EG permission check. For example, a user could create an EPP, inject a Wasm filter, | ||
| and put the download URL of a private OCI image in the Wasm filter configuration. To prevent this, we need to make the | ||
| download URL unguessable. The download URL will be generated by EG and will be a random string that is impossible to | ||
| guess. If a user can get the config dump the Envoy Proxy, they can still get the download URL. However, they won’t be | ||
| able to do so without the permission to access the config dump of Envoy Proxy, which is a more restricted permission (usually Admin role). | ||
|
|
||
| ## Alternative Considered | ||
|
|
||
| ### Inline Bytes | ||
|
|
||
| EG downloads the Wasm modules, caches them in memory, and pushes the Wasm code through xDS as inline bytes. | ||
| This could inflate the xDS, potentially causing memory issues for EG and Envoy. | ||
|
|
||
| ### Data Plane Agent | ||
| Mount Wasm modules in the local file system of the Envoy container. We’ll need an agent deployed in the | ||
| same pod as the Envoy for this. It would be too expensive to implement this as we’ll need to intercept | ||
| the xDS at the agent. | ||
|
|
||
| ### Standalone Wasm HTTP Server | ||
| Deploying the Wasm HTTP server as a standalone service. While this has no obvious benefits, it increases | ||
| operational costs. | ||
|
|
||
| ### Wait for Envoy OCI image support | ||
| We could wait indefinitely for Envoy to support OCI imag as a remote wasm code source. | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.