feat: decode inline Arrow IPC + warehouse-compat fallback#329
Open
jamesbroadhead wants to merge 1 commit intostack/arrow-2-format-renamefrom
Open
feat: decode inline Arrow IPC + warehouse-compat fallback#329jamesbroadhead wants to merge 1 commit intostack/arrow-2-format-renamefrom
jamesbroadhead wants to merge 1 commit intostack/arrow-2-format-renamefrom
Conversation
Serverless warehouses return ARROW_STREAM + INLINE results as base64 Arrow IPC in result.attachment rather than result.data_array. The previous code path discarded inline data for any ARROW_STREAM response (designed for EXTERNAL_LINKS), so these warehouses silently returned empty results. This commit makes the analytics plugin work across classic and serverless warehouses by handling both dispositions for ARROW_STREAM, decoding inline Arrow IPC attachments server-side, and falling back to JSON_ARRAY when a warehouse rejects ARROW_STREAM + INLINE. Changes - Inline Arrow IPC decoding (new arrow-schema.ts) via apache-arrow's tableFromIPC, producing the same row-object shape as JSON_ARRAY regardless of warehouse backend. apache-arrow@21.1.0 added as a server dep. - Format fallback: ARROW_STREAM + INLINE requests automatically fall back to JSON_ARRAY if a classic warehouse rejects them. Explicit format requests are respected without fallback. - Zod-validated SSE wire protocol for /api/analytics/query (shared schema between server and client; malformed payloads surface a clear error instead of silent undefined). - Default remains JSON_ARRAY for compatibility. Stack: layer 3 of 3 carved from #256. - #327 — coverage backfill (layer 1) - #328 — AnalyticsFormat rename to API enum names (layer 2) - (this PR) — the actual fix Fixes #242 Co-authored-by: Isaac
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Serverless warehouses return
ARROW_STREAM+INLINEresults as base64 Arrow IPC inresult.attachmentrather thanresult.data_array. The previous code path discarded inline data for anyARROW_STREAMresponse (designed forEXTERNAL_LINKS), so these warehouses silently returned empty results.This PR makes the analytics plugin work across classic and serverless warehouses by handling both dispositions for
ARROW_STREAM, decoding inline Arrow IPC attachments server-side, and falling back toJSON_ARRAYwhen a warehouse rejectsARROW_STREAM+INLINE.Stack
AnalyticsFormatrename to API enum names (layer 2, mechanical rename)Based on #328. Merge order: #327 → #328 → this PR.
Changes
connectors/sql-warehouse/arrow-schema.ts+client.ts): detectsresult.attachmentand decodes viaapache-arrow'stableFromIPC, producing the same row-object shape asJSON_ARRAYregardless of warehouse backend.apache-arrow@21.1.0added as a server dependency.ARROW_STREAM+INLINErequests automatically fall back toJSON_ARRAYif a classic warehouse rejects them. Explicit format requests are respected without fallback.shared/sse/analytics.ts): single source of truth between server (AnalyticsPlugin._handleQueryRoute) and client (useAnalyticsQuery). Malformed payloads surface a clear error instead of silentundefinedallocation.JSON_ARRAYfor compatibility.Tests
Per-area additions:
connectors/sql-warehouse/tests/arrow-schema.test.ts(new, 514 lines)connectors/sql-warehouse/tests/client.test.ts(new, 382 lines) — uses real base64 Arrow IPC captured from a live serverless warehouseappkit-ui/src/react/hooks/__tests__/use-analytics-query.test.ts(new, 143 lines)shared/src/sse/analytics.test.ts(new, 87 lines)plugins/analytics/tests/analytics.test.ts(+449)Test plan
ARROW_STREAM+INLINEattachments — verifyuseAnalyticsQueryreturns rowsJSON_ARRAYdefault works andARROW_STREAM+EXTERNAL_LINKSstill worksARROW_STREAM+INLINEFixes #242. Replaces #256.
This pull request was AI-assisted by Isaac.