Skip to content

feat: file upload to object storage and then async file verification #954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
heldtogether wants to merge 9 commits into
base: main
Choose a base branch
from
Open

feat: file upload to object storage and then async file verification #954

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f265e47
feat: add blob storage for file uploads and async validation
heldtogether Apr 7, 2026
a427be9
chore: lint
heldtogether Apr 7, 2026
2c92de2
feat: move fusillade-scoped db writes to fusillade
heldtogether Apr 7, 2026
9e0260d
feat: download files from object storage
heldtogether Apr 7, 2026
7d0940c
chore: fix merge conflicts
heldtogether Apr 7, 2026
0752310
test: increase unit test coverage
heldtogether Apr 7, 2026
5cc3aba
feat: use aws creds chain if present
heldtogether Apr 7, 2026
9737cfe
fix: s3 and s3-compatible
heldtogether Apr 7, 2026
5c642a9
feat: surface processing errors to user in dashboard
heldtogether Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
981 changes: 897 additions & 84 deletions Cargo.lock
View file
Open in desktop

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,22 @@ batches:
# Higher values improve throughput for large files but use more memory
# Reduce value if you encounter memory spike issues with many concurrent uploads
batch_insert_size: 5000
# Storage backend for upload path:
# - postgres: legacy (validate + insert synchronously during upload)
# - object_store: write raw JSONL to blob storage then ingest asynchronously
storage_backend: postgres
# Required when storage_backend=object_store
# object_store:
# provider: s3_compatible
# endpoint: "http://localhost:9000"
# bucket: "dwctl-batches"
# region: "us-east-1"
# access_key_id: "minioadmin"
# secret_access_key: "minioadmin"
# path_style: true
# prefix: "uploads/"
# connect_timeout_ms: 5000
# request_timeout_ms: 120000

# Resource limits configuration
# Controls file size, request count, and concurrency limits to prevent resource exhaustion
Expand Down
5 changes: 5 additions & 0 deletions dashboard/src/api/control-layer/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1042,6 +1042,11 @@ export interface FileObject {
| "vision"
| "user_data"
| "evals";
status:
| "uploaded"
| "processed"
| "error";
status_details: string;
/** Email of the individual who created this file */
created_by_email?: string;
/** "Personal" or org name */
Expand Down
81 changes: 49 additions & 32 deletions dashboard/src/components/features/batches/FilesTable/columns.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,8 @@ export const createFileColumns = (
size: 200, // Constrain the actions column width
cell: ({ row }) => {
const file = row.original;
const viewDisabled = file.status === "error";
const disabledReason = file.status_details || "This file cannot be viewed.";
Comment on lines +216 to +217
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The UI only disables View/Download actions when file.status === "error", but in the new async-ingest flow files can be in "uploaded" (pending/processing). In that state the backend won’t have templates/results available yet, so these actions will likely show empty content or confusing errors. Consider also disabling (or showing a distinct “processing” tooltip/state) when status !== "processed".

Suggested change
const viewDisabled = file.status === "error";
const disabledReason = file.status_details || "This file cannot be viewed.";
const viewDisabled = file.status !== "processed";
const disabledReason =
file.status === "error"
? file.status_details || "This file cannot be viewed."
: "This file is still processing and cannot be viewed yet.";

Copilot uses AI. Check for mistakes.
// Disabled for now - expiration not yet enforced on backend
// const isExpired =
// file.expires_at && new Date(file.expires_at * 1000) < new Date();
Expand Down Expand Up @@ -240,53 +242,68 @@ export const createFileColumns = (
)}
<Tooltip delayDuration={500}>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0 text-gray-700 hover:bg-gray-100 hover:text-gray-900"
onClick={(e) => {
e.stopPropagation();
actions.onView(file);
}}
>
<List className="h-4 w-4" />
</Button>
</TooltipTrigger>
<TooltipContent>View Requests</TooltipContent>
</Tooltip>
{file.purpose === "batch" && (
<Tooltip delayDuration={500}>
<TooltipTrigger asChild>
<span className="inline-flex">
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0 text-gray-700 hover:bg-gray-100 hover:text-gray-900"
onClick={(e) => {
e.stopPropagation();
actions.onViewBatches(file);
actions.onView(file);
}}
disabled={viewDisabled}
>
<Layers className="h-4 w-4" />
<List className="h-4 w-4" />
</Button>
</span>
</TooltipTrigger>
<TooltipContent>
{viewDisabled ? disabledReason : "View Requests"}
</TooltipContent>
</Tooltip>
{file.purpose === "batch" && (
<Tooltip delayDuration={500}>
<TooltipTrigger asChild>
<span className="inline-flex">
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0 text-gray-700 hover:bg-gray-100 hover:text-gray-900"
onClick={(e) => {
e.stopPropagation();
actions.onViewBatches(file);
}}
disabled={viewDisabled}
>
<Layers className="h-4 w-4" />
</Button>
</span>
</TooltipTrigger>
<TooltipContent>View Batches</TooltipContent>
<TooltipContent>
{viewDisabled ? disabledReason : "View Batches"}
</TooltipContent>
</Tooltip>
)}
<Tooltip delayDuration={500}>
<TooltipTrigger asChild>
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0 text-gray-700 hover:bg-gray-100 hover:text-gray-900"
onClick={(e) => {
e.stopPropagation();
actions.onDownloadCode(file);
}}
>
<Download className="h-4 w-4" />
</Button>
<span className="inline-flex">
<Button
variant="ghost"
size="sm"
className="h-7 w-7 p-0 text-gray-700 hover:bg-gray-100 hover:text-gray-900"
onClick={(e) => {
e.stopPropagation();
actions.onDownloadCode(file);
}}
disabled={viewDisabled}
>
<Download className="h-4 w-4" />
</Button>
</span>
</TooltipTrigger>
<TooltipContent>Download File</TooltipContent>
<TooltipContent>
{viewDisabled ? disabledReason : "Download File"}
</TooltipContent>
</Tooltip>
<Tooltip delayDuration={500}>
<TooltipTrigger asChild>
Expand Down
5 changes: 4 additions & 1 deletion dwctl/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ embedded-db = ["dep:postgresql_embedded"]

[dependencies]
axum = { version = "0.8", features = ["multipart"] }
fusillade = { version = "14.2.1" }
fusillade = { version = "14.2.1", path = "../../fusillade" }
Copy link
Copy Markdown
Author

@heldtogether heldtogether Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this needs bumping when https://github.com/doublewordai/fusillade/pull/207/changes is merged in.

Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fusillade is switched to a path dependency at ../../fusillade, which is outside this repository/workspace. That will break reproducible builds/CI unless the environment happens to have that directory present. Prefer using the crates.io version, adding fusillade as a workspace member within this repo, or using a git dependency pinned to a commit/tag.

Suggested change
fusillade = { version = "14.2.1", path = "../../fusillade" }
fusillade = "14.2.1"

Copilot uses AI. Check for mistakes.
tokio = { version = "1.0", features = ["full"] }
tokio-stream = { version = "0.1", features = ["sync"] }
tokio-util = "0.7"
Expand Down Expand Up @@ -146,6 +146,9 @@ multer = "3.1.0"
ctor = "0.6"
hmac = "0.12.1"
sha2 = "0.10.9"
aws-config = "1.8.10"
aws-sdk-s3 = "1.115.0"
aws-credential-types = "1.2.9"

[dev-dependencies]
axum-test = { version = "18.4.1" }
Expand Down
10 changes: 10 additions & 0 deletions dwctl/migrations/090_add_file_ingest_jobs.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
CREATE TABLE IF NOT EXISTS file_ingest_jobs (
file_id UUID PRIMARY KEY,
object_key TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN ('pending', 'processing', 'processed', 'failed')),
error_message TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_file_ingest_jobs_status ON file_ingest_jobs(status);
Loading