feat: content-addressable storage (CAS) for scientific data files

[rorybyrne]

Summary

Replace the current path-based file storage with content-addressable storage (CAS) where files are stored by SHA256 hash and referenced by depositions, records, and manifests via hash rather than filesystem path.

Motivation

The current architecture stores files at conventional filesystem paths (depositions/{id}/files/) and moves them between lifecycle stages using rename() / shutil.copy(). This breaks on non-POSIX storage backends (S3 mountpoint gives EPERM on metadata-preserving copies — see the reference-based fix that eliminates copies as step 1).

CAS eliminates file copies entirely (store once, reference many times), enables deduplication (critical for scientific data where reference genomes/structures are shared across thousands of depositions), and provides integrity verification (the hash IS the address — corruption is impossible without detection).

Architecture

S3 Bucket / Local Filesystem:
  /blobs/
    sha256:abc123...    # Raw files stored by content hash
    sha256:def456...
  /lake/                # Future: datalake projection
    records/            # Parquet/Iceberg metadata tables
    features/           # Validator-extracted features

Record manifest with CAS references:

{
  "srn": "urn:osa:node:rec:123@1",
  "files": [
    {"name": "structure.cif", "hash": "sha256:abc123", "size": 4096},
    {"name": "metadata.json", "hash": "sha256:def456", "size": 512}
  ]
}

Why CAS Matters for OSA

• Deduplication: Same reference genome across many instances/depositions stored once
• Federation: Content-addressed = location-independent. Mirror records between nodes without re-copying blobs
• Datalake-ready: CAS hashes are stable identifiers for datalake metadata tables (Iceberg/Parquet). Foundation for analytics over published scientific data
• Integrity: SHA256 hash verifies scientific data hasn't been corrupted — important for reproducibility

Evolution Path

1. Reference-based storage (prerequisite — eliminates copies, introduces FileReference type)
2. Content-addressable storage (this issue — FileReference gains content hash, blobs stored by hash)
3. Datalake projection (Export domain projects manifests → Iceberg tables)
4. Incremental lake (changefeed → streaming Iceberg updates)

Scope

• FileReference type gains content_hash: str field (SHA256 computed on ingest)
• New BlobStore port: put(content) → hash, get(hash) → stream, exists(hash) → bool
• FilesystemBlobAdapter (local): stores at {base}/blobs/{hash}
• S3BlobAdapter: stores at s3://{bucket}/blobs/{hash}
• Deposition file upload computes hash, stores blob, records reference
• Source ingest computes hash after container writes files
• Garbage collection: periodic sweep of unreferenced blobs
• Migration: backfill hashes for existing files (non-breaking, additive)

Depends On

• Reference-based file storage (FileReference type, no-copy source→deposition flow)

[rorybyrne]

Design Decisions from #103 Planning (2026-03-24)

These decisions were made while planning #103 (source-agnostic Record) and inform the CAS design:

CAS Eliminates the File-Move Atomicity Problem

#103 considered moving files to a canonical records/{record_srn}/ directory at publication time. This was rejected because:

• Moving files at publish time creates a distributed transaction across DB and filesystem with no rollback strategy
• S3 has no atomic rename — "move" is copy+delete per key, which can take minutes for large files
• If the move fails after the DB insert, you have a record with no files; if it succeeds but the DB rolls back, you have orphaned files

CAS solves this entirely: files are stored by content hash, "publishing" is just writing a manifest (metadata-only DB operation). No bytes move.

Record's staging_dir Becomes a Content Manifest

In #103, records store staging_dir: str pointing to where files live in the source-specific staging area. When CAS lands, this field evolves into a content-addressed manifest — e.g., files: [{name: str, hash: str, size: int}]. No throwaway work from #103.

Hook File Access via Pre-Staging

Hooks run as sandboxed k8s Jobs with no network (by default). With CAS, files are content-addressed blobs in a flat bucket — you can't scope a FUSE mount to "just these 3 blobs."

Decision: pre-stage files into a scratch directory before the hook runs. The hook runner copies the record's files from CAS into a temporary per-job directory, mounts it as $OSA_IN/files/. The hook sees the same filesystem contract as today. After the hook completes, scratch is cleaned up.

This was chosen over:

• Giving hooks an S3 client scoped by IAM policy → requires network access, operationally complex
• Giving hooks access to all blobs → any hook can read any blob, non-starter for multi-tenant

The hook contract does not change when CAS is adopted. Only the runner's setup logic changes.

Scoped Network Access for K8s Jobs

Both sources and hooks are operator-authored via the OSA SDK. The trust model is the same. No-network is the default. Scoped network access (OSA API only, via K8s NetworkPolicy) is opt-in per definition — needed for ontology resolution during source extraction.