fix: pass artifact paths to rsync via --files-from

[Paul-Goulpie]

fix: pass artifact paths to rsync via --files-from

Problem

When copying artifacts out, each path listed under artifacts.paths
was appended directly as a rsync CLI argument. With a large number of
files (or glob patterns expanding to many files), the total size of
the argument list could exceed ARG_MAX (~2 MB on Linux), causing
rsync to fail silently (swallowed by || true) and producing no
artifacts.

Additionally, glob patterns such as artifact_* were passed
single-quoted to rsync, preventing bash from expanding them — so
rsync received a literal artifact_* and matched nothing.

Solution

• Collect artifact paths into a temporary file using a bash for
  loop (a shell built-in, not subject to ARG_MAX)
• The for loop correctly expands glob patterns before writing
  individual resolved paths to the temp file
• Pass the temp file to rsync via --files-from so the argument
  list remains constant (3 args) regardless of the number of files
• Remove the temp file after rsync completes

Test

Added tests/test-cases/artifacts-many-paths: creates 9 000 files
with 255-character names (~2.3 MB of paths, exceeding ARG_MAX),
exports them via a glob pattern, and asserts in the consume stage
that all 9 000 files were transferred correctly.

---

Summary by cubic

Fix artifact export by passing paths to rsync via --files-from to avoid ARG_MAX errors and ensure glob patterns expand correctly. Uses a temp file and a bash for loop to list files safely.

• Bug Fixes
  • Write expanded artifact paths to a temp file via a bash for loop (not subject to ARG_MAX).
  • Use rsync --files-from to read paths; remove the temp file after.
  • Add an integration test with 9,000 long-named files; assert artifacts are exported and fully consumed under --shell-isolation.

^{Written for commit d81fcfd. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/job.ts">

<violation number="1" location="src/job.ts:1430">
P2: Unquoted `echo \$_gcl_f` can split or glob filenames when generating the rsync `--files-from` list, breaking artifact copying for paths with spaces or leading `-` characters. Quote the variable (or use printf) when writing entries.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}