feat: ingest pipeline backpressure and transient failure handling#121

Merged

rorybyrne merged 13 commits intomainfrom

120-feat-ingest-pipeline

Apr 15, 2026

Commits on Apr 5, 2026

feat: add database migration for deliver_after column and batches_failed tracking
rorybyrne
committed
refactor: replace ingest run SRN with simple ID throughout system
rorybyrne
committed
refactor: simplify ingest completion check logic
rorybyrne
committed
feat: add backpressure control and container log capture
rorybyrne
committed
feat: increase MAX_PENDING_BATCHES from 1 to 4 to improve ingestion throughput
rorybyrne
committed
feat: add max retries configuration and improve backpressure logging
rorybyrne
committed
refactor: replace batch count backpressure with cluster capacity check
rorybyrne
committed
fix: include tests directory in linting and formatting commands
rorybyrne
committed

Commits on Apr 10, 2026

feat: enhance k8s job failure handling and cleanup
rorybyrne
committed

Commits on Apr 12, 2026

Commits on Apr 15, 2026