feat(sync): add plugin system for extensible post-deployment sync steps#513
feat(sync): add plugin system for extensible post-deployment sync steps#513
Conversation
…erface - Add `type: plugin` as a new sync step type in canister.yaml (path/url/sha256/dirs fields) - New `crates/icp-sync-plugin` crate: sandbox path enforcement implemented and tested; runtime stub pending wasmtime Component Model implementation - Wire SyncStep::Plugin through manifest adapter, syncer, deploy and sync commands - Define plugin interface in sync-plugin/sync-plugin.wit (WIT / Component Model) - Add design.md and plan.md in sync-plugin/ - Add POC plugin skeleton in sync-plugin/poc/ - Update JSON schemas Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… plugin Replace the stub in `crates/icp-sync-plugin/src/runtime.rs` with a full wasmtime component model host implementation. The host provides four import functions to the plugin (canister-call, read-file, list-dir, log) and calls the plugin's exported exec() function. Also flesh out the proof-of-concept guest plugin in sync-plugin/poc/ with wit-bindgen bindings and a seed-data uploader that demonstrates the full host↔guest contract end-to-end. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add examples/icp-sync-plugin with a Rust CDK canister that stores (name, content) pairs seeded by the sync plugin from seed-data/ files - Add Candid interface (demo.did) and ic-wasm step to embed it - Link WASI P2 in the wasmtime host so wasm32-wasip2 plugins work - Walk the full error cause chain in sync failure output for better error messages; include wasm path in ReadWasm error - Update POC plugin to pass (filename, content) to the canister's seed() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the custom `read-file` / `list-dir` / `stat` WIT imports with WASI preopens of the manifest's `dirs` entries, so plugins traverse them with standard `std::fs`. Add a new `files` manifest field whose contents the host reads and passes inline via `sync-exec-input`. Update the example canister (`set_config`, `register`, `show`) and POC plugin to exercise both paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
canister-call now exchanges raw Candid-encoded bytes in both directions. The host forwards arg bytes to ic-agent and returns the response bytes unchanged; plugins are responsible for encoding arguments and decoding responses. The POC plugin is updated accordingly and trimmed to just set_config + register. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Highest wasmtime version compatible with Rust 1.90.0 (42+ requires 1.91.0). Adapts to API breakage: WasiView::ctx now returns WasiCtxView, add_to_linker_sync moved to the p2 module, and component bindgen requires a HasData marker. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidate target/ ignore rule into root .gitignore so nested Rust workspaces don't each need their own gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Pipe Drops LineBuf/PluginStdio/PluginOutputStream and the host-side log() import in favour of MemoryOutputPipe: plugin stdout/stderr are captured after exec() returns and forwarded to the progress channel. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Keeps the WIT file alongside its host-side implementation rather than in a separate top-level directory. Update all path references in build.rs and bindgen! / wit_bindgen::generate! invocations accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix the sync-plugin.wit link to its new location in the crate. Update the stdio section: output is now buffered until exec() returns (stdout then stderr), removing stale line-buffering / 64 KiB details. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…workspace Combined the sync plugin PoC (sync-plugin/poc/) and the example canister into a single Cargo workspace under examples/icp-sync-plugin/. The canister lives in canister/ and the plugin in plugin/. Updated icp.yaml and WIT paths accordingly; removed sync-plugin/poc from the root workspace excludes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes the stale top-level sync-plugin/ directory (design.md, plan.md) and SANDBOX.md, replacing them with a DESIGN.md that reflects the current wasmtime/WASI-based implementation and a TODO.md with remaining work items. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…canister-call Threads args.proxy from icp deploy through sync_many → Params so plugin syncers can route update calls via the proxy canister. Extends the WIT interface with a direct: bool field on canister-call-request; when true the host bypasses the proxy and calls the target canister directly even if --proxy was set. The assets and script syncers are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…y principal - Add identity-principal and proxy-canister-id to sync-exec-input in the WIT interface so plugins can act on the caller's identity and proxy configuration. - Replace set_config with set_uploader(Principal): controller-gated update that stores the uploader; register is now restricted to that principal. - Plugin calls set_uploader via proxy (direct: false) and register directly (direct: true), demonstrating both routing modes in a single sync run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers project structure (canister, plugin, seed-data), the role of each component, and a walkthrough of how the direct flag is exercised across the two canister calls made during a sync run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wasmtime 43 requires Rust 1.91.0 (MSRV bump) and changed its error type from anyhow::Error to wasmtime::Error. Update snafu source fields in icp-sync-plugin accordingly and drop the now-unused anyhow dependency. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract shared wasm resolution logic (HTTP fetch, checksum verification, cache read/write) from prebuilt build and plugin sync into a single private canister::wasm module. Rename package cache abstractions from canister/prebuilt-specific names (CanisterCache, canisters_dir, canister_sha, read_cached_prebuilt, cache_prebuilt) to generic wasm equivalents, and move the on-disk subdirectory from "canisters/" to "wasms/" to reflect that plugin wasms are now cached there too. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…execution semantics Adds a wasm32-wasip2 test fixture crate that implements the sync-plugin WIT world with behaviour controlled via the `environment` field, and a build.rs step that compiles it into OUT_DIR and exposes the path via TEST_PLUGIN_WASM. Five tests cover: missing WASM (LoadComponent), missing preopened dir (PreopenDir), plugin Ok/Err returns, and stdout capture through the stdio channel. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… path Covers the full round-trip: compile canister + plugin from examples/icp-sync-plugin/ at test time (no committed binaries), deploy to a local managed network, run icp sync, and verify the canister state via a query call. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Update build_adapter_display_failing_prebuilt_output test to match current error messages. Add wasm32-unknown-unknown and wasm32-wasip2 targets to rust-toolchain.toml so sync_tests pass in CI and locally. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new plugin sync-step type to the icp manifest format, allowing icp sync to execute a sandboxed wasm32-wasip2 component (wasmtime/WASI) as an extensible post-deployment step. This integrates with existing sync/build flows (including caching and remote wasm fetching) and exposes --proxy to control call routing during sync/plugin execution.
Changes:
- Introduce
pluginsync steps (manifest + schema + CLI dispatch) and a newicp-sync-pluginhost runtime crate using wasmtime component model + WASI sandboxing. - Factor wasm source resolution (local/remote + optional sha256 + caching) into a shared
crates/icp/src/canister/wasm.rsmodule and migrate prebuilt build adapter to use it. - Add tests/fixtures and an end-to-end example exercising plugin execution and canister calls.
Reviewed changes
Copilot reviewed 45 out of 49 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| rust-toolchain.toml | Bumps Rust toolchain version. |
| examples/icp-sync-plugin/seed-data/fruit-01.txt | Example seed-data fixture file. |
| examples/icp-sync-plugin/seed-data/fruit-02.txt | Example seed-data fixture file. |
| examples/icp-sync-plugin/seed-data/fruit-03.txt | Example seed-data fixture file. |
| examples/icp-sync-plugin/plugin/src/lib.rs | Example plugin implementation using WIT bindings + canister calls. |
| examples/icp-sync-plugin/plugin/build.rs | Rebuild trigger for WIT changes in example plugin. |
| examples/icp-sync-plugin/plugin/Cargo.toml | Example plugin crate definition (wasm32-wasip2 cdylib). |
| examples/icp-sync-plugin/icp.yaml | Example project manifest wiring build + plugin sync step. |
| examples/icp-sync-plugin/demo.did | Example canister interface for the demo. |
| examples/icp-sync-plugin/config.txt | Example inline file input. |
| examples/icp-sync-plugin/canister/src/lib.rs | Example canister implementation used by the plugin demo. |
| examples/icp-sync-plugin/canister/Cargo.toml | Example canister crate definition. |
| examples/icp-sync-plugin/README.md | Example documentation explaining routing modes and flow. |
| examples/icp-sync-plugin/Cargo.toml | Example workspace manifest. |
| examples/icp-sync-plugin/Cargo.lock | Lockfile for the example workspace. |
| docs/schemas/icp-yaml-schema.json | Regenerated schema including plugin sync step fields. |
| docs/schemas/canister-yaml-schema.json | Regenerated schema including plugin sync step fields. |
| docs/reference/cli.md | Regenerated CLI docs including icp sync --proxy. |
| crates/icp/src/package.rs | Renames wasm cache layout (wasms/…/module.wasm) and cache helpers. |
| crates/icp/src/manifest/canister.rs | Adds SyncStep::Plugin and display formatting + tests. |
| crates/icp/src/manifest/adapter/prebuilt.rs | Updates prebuilt adapter docs to reflect generic wasm-source usage. |
| crates/icp/src/manifest/adapter/plugin.rs | Adds manifest adapter struct for plugin step (source, sha256, dirs, files) + unit tests. |
| crates/icp/src/manifest/adapter/mod.rs | Exposes new plugin manifest adapter module. |
| crates/icp/src/canister/wasm.rs | New shared wasm fetch/verify/cache resolver for local/remote sources. |
| crates/icp/src/canister/sync/plugin.rs | Implements plugin-step execution: resolve wasm path, read inline files, run plugin. |
| crates/icp/src/canister/sync/mod.rs | Wires plugin step into sync dispatcher and plumbs pkg cache/proxy/environment. |
| crates/icp/src/canister/mod.rs | Exposes new internal wasm module. |
| crates/icp/src/canister/build/prebuilt.rs | Refactors prebuilt build adapter to use shared wasm resolver. |
| crates/icp/Cargo.toml | Adds dependency on icp-sync-plugin. |
| crates/icp-sync-plugin/tests/fixtures/test-plugin/src/lib.rs | Minimal test plugin fixture source (WIT guest). |
| crates/icp-sync-plugin/tests/fixtures/test-plugin/build.rs | Rebuild trigger for fixture WIT changes. |
| crates/icp-sync-plugin/tests/fixtures/test-plugin/Cargo.toml | Fixture crate manifest for compiling test plugin wasm. |
| crates/icp-sync-plugin/tests/fixtures/test-plugin/Cargo.lock | Fixture lockfile. |
| crates/icp-sync-plugin/sync-plugin.wit | Defines WIT contract between host/runtime and plugins. |
| crates/icp-sync-plugin/src/runtime.rs | Implements wasmtime component runtime, WASI sandbox, and host canister_call import. |
| crates/icp-sync-plugin/src/lib.rs | Exposes run_plugin API and error type. |
| crates/icp-sync-plugin/build.rs | Build script that compiles the wasm fixture and sets TEST_PLUGIN_WASM. |
| crates/icp-sync-plugin/TODO.md | Tracks planned follow-ups (timeout, more tests). |
| crates/icp-sync-plugin/DESIGN.md | Design rationale, sandbox table, and authoring guidance. |
| crates/icp-sync-plugin/Cargo.toml | New crate manifest wiring wasmtime/wasmtime-wasi dependencies. |
| crates/icp-cli/tests/sync_tests.rs | Adds PocketIC E2E test that builds example canister+plugin and validates sync results. |
| crates/icp-cli/src/operations/sync.rs | Plumbs environment/proxy/pkg cache into sync execution and improves error cause logging. |
| crates/icp-cli/src/commands/sync.rs | Adds --proxy flag to icp sync and passes through environment/proxy/cache. |
| crates/icp-cli/src/commands/deploy.rs | Passes environment/proxy/cache through deploy-triggered sync. |
| Cargo.toml | Excludes example workspace; adds icp-sync-plugin + wasmtime deps to workspace. |
| CHANGELOG.md | Documents new plugin sync step and icp sync --proxy. |
| .gitignore | Updates target ignore pattern. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
On Windows, joining an absolute Unix path (e.g. /foo) with a base dir prepends the current drive letter (e.g. C:/foo), causing error messages to differ across platforms. Use the original path from the manifest (s.path) in the log and error context instead of the OS-resolved path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bugs: - Fix temp file leak: cleanup now runs on both success and error paths in plugin::sync, not only on success. - Remove needless clone: ¶ms.environment.clone() -> ¶ms.environment (params is already &Params). Security: - Validate `files` manifest entries in plugin::sync: reject absolute paths and '..' components before joining with the canister directory. - Validate `dirs` entries in run_plugin: same check before preopening directories into the WASI sandbox. Non-blocking: - Cap MemoryOutputPipe at 1 MiB per stream instead of usize::MAX. - Add --locked to build.rs fixture build to prevent network access. - Make build_sync_plugin_example() return Option and skip the e2e test gracefully when wasm32-wasip2 is not installed. Docs / stale text: - Replace "Extism sandbox" with "wasmtime WASI sandbox" in canister.rs doc comment; regenerate JSON schemas. - Update --proxy clap help text and changelog: it routes sync plugin calls to the target canister, not management canister calls; regenerate cli.md. - Update run_plugin signature in DESIGN.md (add proxy + identity_principal). - Clarify sync-plugin.wit: direct=false only proxies update calls; query calls always go directly to the target canister. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…chain wasm32-wasip2 is declared in rust-toolchain.toml so it is always available locally and in CI. Remove all the optional-skip paths: - build.rs: assert fixture build succeeds instead of silently skipping - runtime.rs unit tests: use env! instead of option_env! guards - sync_tests.rs: revert build_sync_plugin_example() to return (PathBuf, PathBuf) directly and assert the build succeeds Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 45 out of 49 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Enforce at parse time that a plugin step using `url:` must supply `sha256:`. Previously the doc comment said it was required but nothing prevented omitting it, which silently allowed unverified downloads. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The sha256 enforcement from the previous commit means remote plugin sources always have a checksum, so they always resolve through the content-addressed cache. The temp-file branch (which used wasm content bytes as a name, creating a race under concurrent sync) is now dead code — remove it along with the WriteTempWasm error variant and the post-run cleanup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
plugin_stdout_forwarded_through_stdio_channel calls run_plugin, which uses block_in_place internally. Wrapping the call in block_in_place and using a multi-thread runtime matches the production callsite and prevents a silent panic if canister_call (which needs Handle::current()) is ever exercised in this test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A closed or full stdio channel should not abort a wasm fetch or build. Drop the Log error variants from WasmError and PrebuiltError and replace .context(LogSnafu)? with let _ = tx.send(...).await, matching the pattern already used for stdout forwarding in the plugin runtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes the implicit update default from the host and requires plugins to always specify call-type explicitly, making intent clear at the call site. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add `cycles: u64` to the `canister-call-request` WIT record so plugins can attach cycles to proxied update calls, replacing the hardcoded zero. The field is documented as a no-op for direct calls and query calls. Update the example plugin and DESIGN.md snippet accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds sync_plugin_routes_through_proxy, which deploys through the local proxy canister (making it a controller) and verifies that the plugin's set_uploader call is correctly routed through the proxy while register calls go directly. Also removes the redundant explicit sync call from sync_plugin_registers_seed_data since deploy already runs sync steps. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Unverified remote wasm bytes were previously written to the cache unconditionally, allowing integrity-unverified content to persist on disk and the cache key (a recomputed digest) to never match a future lookup (which requires a known sha256). Now the cache write is gated on sha256 being provided, consistent with the cache read path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stack depth (512 KiB), compute time (60s via epoch interruption), and a memory-bounds comment are added per PR review. Canister call latency is excluded from the compute budget by crediting elapsed ticks back after each host call returns. Limits are documented in DESIGN.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ilt and plugin wasm::resolve() now returns a PathBuf instead of bytes, covering both use cases with consistent behavior: local sources verify sha256 if present and return the original path; remote sources with sha256 check the cache first, remote without sha256 always download and cache by the computed sha256. prebuilt uses fs::copy() (cross-filesystem safe, handles WSL) instead of writing bytes. plugin calls the same resolve() as prebuilt, removing the separate resolve_path()/cached_path() functions and the redundant LockCache error variant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use console::strip_ansi_codes to prevent terminal injection via plugin stdout/stderr or return messages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
What this PR does
Adds a
pluginsync step type to canister manifests. A plugin is a WebAssembly component (wasm32-wasip2) that runs inside a wasmtime WASI sandbox duringicp sync. It can call update/query methods on the canister being synced, and read files from directories declared in the manifest. Plugins are specified as a localpath:or a remoteurl:with optionalsha256:verification.Also adds
--proxytoicp sync, forwarded through to the plugin runtime.See
crates/icp-sync-plugin/DESIGN.mdfor the full design rationale, WIT interface spec, and sandbox capability table.Reviewer guide
Read carefully — core logic
crates/icp-sync-plugin/sync-plugin.witcrates/icp-sync-plugin/src/runtime.rscanister_callhost implcrates/icp/src/manifest/adapter/plugin.rspluginstep fieldscrates/icp/src/manifest/canister.rsPluginStepstruct and integration with the existing step enumcrates/icp/src/canister/sync/plugin.rsrun_plugindispatchcrates/icp-cli/src/operations/sync.rsscript/assetsstepsMechanical / low scrutiny
crates/icp/src/canister/wasm.rsprebuilt.rs— same logic, new shared locationcrates/icp/src/canister/build/prebuilt.rswasm.rsdocs/schemas/canister-yaml-schema.json,icp-yaml-schema.jsondocs/reference/cli.mdrust-toolchain.tomlCargo.toml,.gitignoretarget/dirs ignoredTests added
crates/icp-sync-plugin/src/runtime.rs(5 unit tests)crates/icp-sync-plugin/tests/fixtures/test-plugin/wasm32-wasip2component used as a fixture by the unit tests abovecrates/icp-cli/tests/sync_tests.rs::sync_plugin_registers_seed_dataExample
examples/icp-sync-plugin/is a self-contained workspace (canister + plugin + seed data) that exercises the full flow, including both proxy and direct call routing. See itsREADME.mdfor how to run it manually.🤖 Generated with Claude Code