fix: auto-heal corrupted OCI local store by forcing re-pull

[Pnkcaht]

---

What I did

• Fixed the OCI source loading logic to treat the local content store as a cache, not a source of truth.
• Added an automatic self-healing mechanism when the local OCI store is corrupted or inconsistent.
• Implemented a clear fallback flow:
  1. Try to load from local store.
  2. Attempt a normal pull (digest-based, no forced re-download).
  3. Force a re-pull when corruption is detected.
• Ensured the agent can recover transparently without requiring manual cache cleanup.

Related issue

Fixes #1448

What was the bug

When an OCI artifact was already present in the local content store, the system assumed it was valid and tried to load it unconditionally.

If the local store became corrupted or partially written (for example, interrupted downloads, invalid tar layers, or missing metadata), the following happened:

• The loader failed with ErrStoreCorrupted.
• A normal remote.Pull was not sufficient, because the reference already existed locally.
• The system never attempted a forced re-download, even though the local data was unusable.
• This resulted in a permanent failure state unless the user manually deleted the local cache.

In practice, a broken local cache could brick the OCI source resolution entirely.

---

Explain With Diagrams

OCI source resolution — old behavior (bug)

Description

In the previous implementation, OCI-based agent configurations were loaded from the local content store without any reliable recovery mechanism.

When an agent was requested, the OCI source attempted to read the artifact directly from the local content.Store. If any file inside the store was missing or inconsistent (for example, a missing reference file, tarball, or metadata), the store returned ErrStoreCorrupted.

At this point, the error was treated as fatal.

What went wrong

Once ErrStoreCorrupted was returned:

• No automatic re-fetch from the remote OCI registry was triggered
• The system did not attempt to repair or recreate the missing files
• Subsequent runs kept failing with the same error
• The agent became unusable until the user manually deleted the local store

This created a persistent failure mode where a transient or partial disk issue resulted in a permanently broken agent cache. The system had no mechanism to invalidate or repair a broken local reference.

Impact

From the user’s perspective, this surfaced as intermittent but unrecoverable errors such as:

• “reference <name>:latest not found”
• “failed to load agent from store”

Even though the remote artifact was valid and accessible, the local corruption prevented recovery.

New behavior (self-healing OCI store flow)

Description

With the new implementation, the local OCI content store is treated as a recoverable cache rather than a source of truth.

When an agent is requested from an OCI reference, the system follows a multi-step fallback strategy that guarantees recovery from partial or inconsistent local state.

Step-by-step flow

1. Local load attempt

   • The system first tries to load the agent from the local content store.
   • If all required files are present and consistent, the agent is returned immediately.

2. Normal OCI pull (safe revalidation)

   • If the local load fails, a standard OCI pull is attempted.
   • This pull verifies and updates missing content without forcing a full re-download.
   • After the pull, the local store is retried.

3. Corruption detection

   • If the failure is identified as ErrStoreCorrupted, the store is considered inconsistent.
   • This includes missing tarballs, reference links, or metadata files.

4. Forced re-pull (store repair)

   • When corruption is detected, a forced OCI pull is executed.
   • This fully re-fetches the artifact and reconstructs the local store state.

5. Final retry

   • After the forced pull, the system retries loading the agent from the store.
   • If successful, execution continues transparently.

Key guarantees

• The system never remains stuck in a corrupted local state
• Missing or partially written files are automatically repaired
• Users do not need to manually clean or reset the local store
• Remote OCI artifacts are always treated as the authoritative source

Outcome

This change converts a hard failure scenario into a self-healing process, ensuring that agent execution remains reliable even in the presence of local cache corruption.

Failure scenarios and recovery boundaries

Description

This diagram highlights the different failure scenarios that can occur when loading agents from OCI artifacts and clearly defines where recovery is possible and where it must stop.

The goal is to avoid infinite retries while still guaranteeing automatic repair whenever feasible.

Failure scenarios covered

1. Missing reference link

   • The reference file under store/refs/ does not exist.
   • This usually happens due to partial writes, interrupted pulls, or manual filesystem changes.
   • Classified as store corruption.

2. Missing or unreadable tarball

   • The <digest>.tar file is missing or cannot be parsed.
   • Indicates an incomplete or corrupted local artifact.
   • Classified as store corruption.

3. Invalid image structure

   • The tarball exists but cannot be loaded as a valid OCI image.
   • This includes errors while loading layers or manifests.
   • Classified as store corruption.

4. Empty or unreadable layers

   • The image loads, but contains no layers or unreadable content.
   • Indicates a logically inconsistent artifact.
   • Classified as store corruption.

5. Remote pull failure

   • Network errors, authentication issues, or registry unavailability.
   • Not considered corruption by itself.
   • Recovery depends on whether a valid local copy exists.

Recovery boundaries

• If local corruption is detected:

  • A forced OCI pull is always attempted.
  • The local store is fully rebuilt from the remote source.

• If remote pull fails and no valid local copy exists:

  • The operation fails explicitly.
  • An error is returned to the user with clear context.

• If remote pull fails but a valid local copy exists:

  • The system continues using the cached version.
  • A warning is logged for observability.

Safety guarantees

• No infinite retry loops
• Forced pulls happen at most once per request
• Corruption detection is explicit and centralized
• Clear separation between recoverable and unrecoverable failures

Outcome

This model ensures that the system aggressively heals local state when possible, while still failing fast and transparently when recovery is genuinely impossible.

---

Notes

• The local OCI content store is now treated as a best-effort cache.
• Corruption detection is explicit and based on content.ErrStoreCorrupted.
• Forced re-pull is only triggered when corruption or inconsistency is detected, avoiding unnecessary downloads.
• This change improves resilience without altering the happy-path performance.

Important

No automated tests were added as part of this change.
The fix was validated through manual builds and targeted reasoning over failure scenarios.

OS / System

• OS: Linux Debian 13

[krissetto]

can't be merged until the linter is pleased and a maintainer re-approves

[Pnkcaht]

can't be merged until the linter is pleased and a maintainer re-approves

I will solve

[Pnkcaht]

Resolved guys @krissetto @simonferquel

Task lint

pnkcaht@pnkcaht:~/Documents/CagentDocker/cagent$ task lint
task: [lint] golangci-lint run
0 issues.
task: [lint] go mod tidy --diff >/dev/null || (echo "go.mod/go.sum files are not tidy" && exit 1)

[krissetto]

This is a breaking change to this function and introduces new bugs.

This early return means we no longer check for updates to the OCI reference on the registry, which was intended behavior.

It also causes the --pull-interval <mins> feature of cagent api, which is intended to auto-update the image periodically if updates are available, to break. This function now never does that check unless there are store corruption issues.

Let's make sure any change in this PR only fixes potential corruption issues, without changing the underlying logical behavior.

[Pnkcaht]

done