oci: Add fast path for oci: transport using ocidir crate by cgwalters · Pull Request #233 · composefs/composefs-rs

cgwalters · 2026-02-13T17:57:09Z

When importing from local OCI layout directories (oci: transport), read
the layout directly instead of going through skopeo; this path
is faster.

But another motivation here is that right now the skopeo proxy
always peels manifest lists, but I want to add proper support
for storing those. Fetching from local oci helps us write
unit/integration tests for that without blocking on patching
skopeo.

cgwalters · 2026-02-26T19:27:48Z

OK this now depends on bootc-dev/ocidir-rs#57 which cleans this up a lot.

jeckersb · 2026-03-06T22:48:02Z

Looks like this needs rebased after #248 landed

cgwalters · 2026-03-06T23:16:58Z

Looks like this needs rebased after #248 landed

Done

jeckersb · 2026-03-10T18:47:02Z

Conflicts again, and while I was looking at doing the rebase it's not totally clear to me what needs to happen since this intersects with the recent tar changes

jeckersb · 2026-03-10T19:12:50Z

More concretely, I think we're losing the parts that added with_capacity for buffered reads if I just naively take the chunk from the branch.

cgwalters · 2026-03-10T21:05:27Z

More concretely, I think we're losing the parts that added with_capacity for buffered reads if I just naively take the chunk from the branch.

I think with this new resolution it's right - we're using the bufread changes in decompression (and when doing OCI artifact storage) but the tar parsing code is using the Bytes abstraction.

cgwalters · 2026-03-11T15:16:37Z

While not blocking this PR directly it'd be helpful to land bootc-dev/ocidir-rs#64 since I think as we head towards #224 we will need good support for referrers.

cgwalters · 2026-03-17T14:24:21Z

Was just noticing: This makes composefs-oci depend on cap-std, and since it's part of the same binary we may as well just use it in more of the composefs crate.

Johan-Liebert1 · 2026-04-13T09:26:23Z

This needs another rebase

cgwalters · 2026-04-27T15:17:45Z

This also wants bootc-dev/ocidir-rs#73

Johan-Liebert1

I get this error when trying cfsctl oci pull <image> locally

Caused by:
    0: Opening stream ''
    1: Creating new splitstream reader
    2: Invalid splitstream content type

Would be great to have contexts on functions as it's pretty tedious to trace this error

Johan-Liebert1 · 2026-04-28T08:09:27Z

The above error seems to be originating from create_filesystem function.

Extract decompression, tar import, blob storage, and media type checking from skopeo.rs and oci_image.rs into a reusable layer module. This prepares for adding a direct OCI layout import path that needs the same functionality without going through the skopeo proxy. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

ocidir 0.7.1 introduces open_image_this_platform() for resolving manifest lists, needed for the OCI layout fast path added in the next commit. Move both ocidir and cap-std-ext to workspace deps so versions stay in sync across composefs-oci and integration-tests. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

Use containers_image_proxy::ImageReference to parse the image reference once in pull_image() and pass it through to ImageOp::new(), which now takes &ImageReference instead of re-parsing the transport from the raw string. This also lets us use open_image_ref() instead of open_image(). This prepares for transport-based dispatch (e.g. fast-pathing oci: references) without manual string prefix matching. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

For local OCI layout directories (oci: transport), read the layout directly using the ocidir crate instead of going through the containers-image-proxy / skopeo subprocess. This avoids subprocess spawning, IPC overhead, and proxy protocol parsing for local imports. The new oci_layout module handles manifest list resolution for the current platform via ocidir's open_image_this_platform(), imports layers in parallel using the shared layer module, and produces identical splitstream output to the proxy path. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

Ensure all import paths add named stream refs in the order that layers appear in the OCI image config (diff_ids), rather than in whatever order the import happens to process them (e.g. sorted by size for parallel fetching, or non-deterministic HashMap iteration). The skopeo, oci_layout, and write_config paths now iterate the config diff_ids array and look up layer verities by key, returning an error if any layer verity is missing. The write_manifest signature changes from HashMap to an ordered slice so callers control the order structurally. Assisted-by: OpenCode (claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>

When a diff_id key is not found in the refs map, include the available keys in the error message to make debugging easier. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

…_image The oci: fast path was added after the ensure_writable() guard in pull_image(), so a read-only repo would get an unhelpful "No such file or directory" error instead of the expected "not writable" error. Move the check to the very top of pull_image() so it applies uniformly to all transports. Fixes the privileged_pull_readonly_repo integration test. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

Update containers-image-proxy to 0.10, which uses oci-spec 0.9.0 instead of 0.8.x. Bump ocidir to 0.7.2, which adds the fallback to read the manifest config blob when an index entry has no explicit platform annotation (the correct OCI-spec behavior, matching what container runtimes do). Bump cstorage's oci-spec dep from 0.8 to 0.9 to match. With both deps on oci-spec 0.9 the types unify, so the composefs-rs workaround in resolve_manifest() that manually replicated ocidir's missing logic can be removed. Also adapt to ImageProxyConfig being #[non_exhaustive] in 0.10, using Default::default() + field assignment instead of a struct literal. Assisted-by: OpenCode (claude-sonnet-4-6@default) Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters force-pushed the fastpath-oci-import branch 2 times, most recently from 168de92 to 256bd97 Compare February 26, 2026 19:26

cgwalters force-pushed the fastpath-oci-import branch from 256bd97 to 33304b6 Compare February 27, 2026 01:43

pkpbynum mentioned this pull request Mar 3, 2026

Support passing --insecure-policy bootc-dev/containers-image-proxy-rs#127

Merged

cgwalters force-pushed the fastpath-oci-import branch from 33304b6 to cec8563 Compare March 5, 2026 16:49

cgwalters marked this pull request as ready for review March 5, 2026 16:51

cgwalters closed this Mar 5, 2026

cgwalters reopened this Mar 5, 2026

cgwalters force-pushed the fastpath-oci-import branch from f1be28a to cc11746 Compare March 6, 2026 02:08

cgwalters enabled auto-merge (rebase) March 6, 2026 12:25

cgwalters requested review from Johan-Liebert1 and jeckersb March 6, 2026 12:27

cgwalters force-pushed the fastpath-oci-import branch from cc11746 to 2af7a01 Compare March 6, 2026 23:14

cgwalters force-pushed the fastpath-oci-import branch from 2af7a01 to bf7932e Compare March 10, 2026 20:05

cgwalters force-pushed the fastpath-oci-import branch from bf7932e to 34f2b19 Compare March 31, 2026 13:49

cgwalters force-pushed the fastpath-oci-import branch from 34f2b19 to be05dbb Compare April 25, 2026 18:31

Johan-Liebert1 requested changes Apr 28, 2026

View reviewed changes

cgwalters force-pushed the fastpath-oci-import branch from be05dbb to 55aa4f4 Compare May 1, 2026 22:28

cgwalters added 7 commits May 1, 2026 18:28

cgwalters force-pushed the fastpath-oci-import branch from 55aa4f4 to 5c62103 Compare May 1, 2026 22:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oci: Add fast path for oci: transport using ocidir crate#233

oci: Add fast path for oci: transport using ocidir crate#233
cgwalters wants to merge 8 commits intocomposefs:mainfrom
cgwalters:fastpath-oci-import

cgwalters commented Feb 13, 2026

Uh oh!

cgwalters commented Feb 26, 2026

Uh oh!

jeckersb commented Mar 6, 2026

Uh oh!

cgwalters commented Mar 6, 2026

Uh oh!

jeckersb commented Mar 10, 2026

Uh oh!

jeckersb commented Mar 10, 2026

Uh oh!

cgwalters commented Mar 10, 2026

Uh oh!

cgwalters commented Mar 11, 2026

Uh oh!

cgwalters commented Mar 17, 2026

Uh oh!

Johan-Liebert1 commented Apr 13, 2026

Uh oh!

cgwalters commented Apr 27, 2026

Uh oh!

Johan-Liebert1 left a comment

Uh oh!

Johan-Liebert1 commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

cgwalters commented Feb 13, 2026

Uh oh!

cgwalters commented Feb 26, 2026

Uh oh!

jeckersb commented Mar 6, 2026

Uh oh!

cgwalters commented Mar 6, 2026

Uh oh!

jeckersb commented Mar 10, 2026

Uh oh!

jeckersb commented Mar 10, 2026

Uh oh!

cgwalters commented Mar 10, 2026

Uh oh!

cgwalters commented Mar 11, 2026

Uh oh!

cgwalters commented Mar 17, 2026

Uh oh!

Johan-Liebert1 commented Apr 13, 2026

Uh oh!

cgwalters commented Apr 27, 2026

Uh oh!

Johan-Liebert1 left a comment

Choose a reason for hiding this comment

Uh oh!

Johan-Liebert1 commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants