Reimplement the data backend of the Storage component so it can become scalable horizontally

[matthyx]

[yugal07]

Why the current backend can't scale out

StorageImpl (pkg/registry/file/storage.go) holds three pieces of node-local state. All three break when running more than one replica:

Piece	Where	Why it breaks N>1
Metadata	sqlitemigration.Pool over a local SQLite file (sqlite.go)	File-locked, single-writer; replicas can't share
Payloads	afero.Fs rooted at a local dir	Each replica has its own disk
Watch fan-out	WatchDispatcher (watch.go), in-process channels	Replica A never sees replica B's writes — silently breaks every controller that watches these CRDs

The sqlite+afero design exists because etcd couldn't handle the payload sizes (large CVE manifests, application profiles, network neighborhoods). Any replacement must preserve that property . "move back to etcd" is not an option.

Proposed plan , two phases

Phase 1: extract a Backend interface (refactor only, no behavior change)

Introduce an interface that captures the three responsibilities above:

type Backend interface {
    Metadata() MetadataStore  // CRUD on metadata rows
    Payloads() PayloadStore   // read/write opaque object payloads
    Watch()    WatchBus       // publish/subscribe across replicas
}

The existing sqlite + afero + in-process dispatcher becomes LocalBackend and remains the default. StorageImpl stops talking to *sqlitemigration.Pool and afero.Fs directly and goes through Backend.

This phase is mergeable on its own, ships zero user-visible change, and is the prerequisite for any remote implementation. It's also the right place to land test coverage that pins current behavior before swapping internals.

Phase 2: add a RemoteBackend for HA installs (separate PR, behind config)

Selected via Helm values / config flag; LocalBackend stays the default for single-node installs. Candidate stacks for maintainers to weigh in on:

• Postgres + S3-compatible object store + Postgres LISTEN/NOTIFY (or NATS) for watch fan-out. Most flexible, three concerns cleanly split, two new infra deps for the operator.
• CockroachDB / FoundationDB for metadata + same object store for payloads. One distributed DB instead of two services, heavier single dep.
• Postgres for both metadata and payloads (bytea / large objects) , fewer moving parts, but payload size pushes Postgres harder than it likes.

My weak preference is option 1 (Postgres + S3 + NATS) , it matches how each subsystem is independently scaled in practice and lets payload storage be swapped for any S3-compatible service the user already runs.

Few Clarifications before implementation

1. Is a pluggable backend acceptable, or do you want a single replacement with no LocalBackend left behind?
2. Which Phase 2 stack is acceptable as a hard dependency for HA Kubescape installs?
3. Migration path from existing on-disk sqlite+afero data to a remote backend , required for v1, or a follow-up?
4. Watch semantics under multi-replica: is at-least-once delivery acceptable, or do we need exactly-once?

If this is open for contribution can I take on this? @matthyx

[matthyx]

@yugal07 I would not touch this... the whole reason we switched from etcd to SQLite + files was to reduce memory usage.

If you start using Postgres, S3, or these distributed datastores, we could even just deploy a new etcd and rely on the existing implementation inside sample-apiserver.

I think this will remain here until a hyperscaler asks specifically for a scalable storage because their clusters are too big for us...

[yugal07]

Ahh I see. No issues, Thanks