Improve pipelines setup command with validation retry loops

[jonesphillip]

Add validation retry loops to pipelines setup command:

• Validation errors now prompt for retry instead of failing the entire setup.
• Added a "simple" mode for sink config that uses sensible defaults.
• Buckets are created automatically if they don't exist, and Data Catalog is enabled automatically if not already active.
• Added tests for validation and retry flows.

In addition to new tests, conducted these manual tests:

• Pipeline name, bucket name, field name validation with retry
• R2 sink with simple and advanced modes, automatic credential generation
• Automatic bucket creation and Data Catalog enablement
• Data Catalog flow with token validation
• Schema from JSON file, SQL from file, custom SQL with validation errors

---

• Tests
  • Tests included/updated
  • Automated tests not possible - manual testing has been completed as follows:
  • Additional testing not necessary because:
• Public documentation
  • Cloudflare docs PR(s):
  • Documentation not necessary because: Improves existing interactive command

---

[changeset-bot]

🦋 Changeset detected

Latest commit: 39170a7

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[claude]

Claude finished @jonesphillip's task —— View job

---

Changeset Review

• Read changeset guidelines
• Review changeset file
• Validate against criteria
• Provide feedback

---

✅ All changesets look good

The changeset .changeset/fancy-gifts-shave.md meets all the required criteria:

1. Version Type: ✅ Minor is appropriate for new features (retry loops, simple mode, automatic resource creation)
2. Changelog Quality: ✅ Clear, meaningful description with bullet points explaining each improvement. The description effectively communicates both what changed and why it matters to users.
3. Markdown Headers: ✅ No h1/h2/h3 headers used
4. Analytics: ✅ Not applicable (no new analytics collection)
5. Experimental features: ✅ Not applicable (this is a stable feature improvement)

The changeset description is well-written and addresses the earlier feedback from @ascorbic - it now provides sufficient detail about the changes including validation retry prompts, simple mode, automatic bucket creation, and automatic Data Catalog enablement, along with the user benefit.

[pkg-pr-new]

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@12401

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@12401

miniflare

npm i https://pkg.pr.new/miniflare@12401

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@12401

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@12401

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@12401

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@12401

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@12401

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@12401

wrangler

npm i https://pkg.pr.new/wrangler@12401

commit: 39170a7

[devin-ai-integration]

Devin Review found 2 potential issues.

View 7 additional findings in Devin Review.

[ascorbic]

Just a couple of small things raised in the automated reviews. Then it should be good to go.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 11 additional findings in Devin Review.

[devin-ai-integration]

🔴 Orphaned stream and sink without guidance when SQL setup throws during pipeline creation retry

When pipeline creation fails and the user opts to retry with different SQL, setupSQLTransformationWithValidation is called at packages/wrangler/src/pipelines/cli/setup.ts:1229. If the user then fails SQL validation and declines the retry within that function (or cancels SQL file loading, etc.), a UserError is thrown that propagates out of createPipelineIfNeeded unhandled — bypassing the helpful guidance messages at lines 1216-1225 that tell the user about their already-created stream and sink resources.

Root Cause and Impact

The setupSQLTransformationWithValidation call at line 1229 is not wrapped in a try/catch. Any UserError it throws (e.g., "SQL validation failed and setup cannot continue..." from line 1047, "SQL file loading cancelled" from loadSqlFromFile, or "SQL query cannot be empty" from line 996) propagates directly up to the handler's catch block at line 365, which simply re-throws it.

This means the user sees only the error message (e.g., "SQL validation failed...") but never sees the critical guidance:

You can create the pipeline later with: wrangler pipelines create
Your stream "..._stream" and sink "..._sink" are ready.

The stream and sink have already been created at this point (by reviewAndCreateStreamSink), so the user is left with orphaned resources and no instructions on how to complete the setup.

Impact: Users who hit SQL issues during a pipeline creation retry will have orphaned cloud resources (stream and sink) with no guidance on how to finish setting up their pipeline or clean up.

Prompt for agents

In the createPipelineIfNeeded function (packages/wrangler/src/pipelines/cli/setup.ts), wrap the call to setupSQLTransformationWithValidation at line 1229 in a try/catch block. When it throws (e.g. UserError from SQL validation failure or cancellation), display the same helpful guidance that is shown when the user explicitly declines retry (lines 1216-1225), informing them that their stream and sink were created successfully and that they can create the pipeline later with wrangler pipelines create. Then return gracefully instead of letting the error propagate. For example:

try {
  await setupSQLTransformationWithValidation(config, setupConfig);
} catch (sqlError) {
  logger.log(chalk.dim("\n  Stream and sink were created, but pipeline creation failed."));
  logger.log(chalk.dim("\n  You can create the pipeline later with: wrangler pipelines create"));
  logger.log(chalk.dim(`  Your stream "${setupConfig.streamName}" and sink "${setupConfig.sinkName}" are ready.`));
  return;
}

---

Was this helpful? React with 👍 or 👎 to provide feedback.