Skip to content

fix: convert BigInt to strings in Arrow IPC row.toJSON()#270

Closed
jamesbroadhead wants to merge 1 commit intodatabricks:mainfrom
jamesbroadhead:fix/bigint-arrow-serialization
Closed

fix: convert BigInt to strings in Arrow IPC row.toJSON()#270
jamesbroadhead wants to merge 1 commit intodatabricks:mainfrom
jamesbroadhead:fix/bigint-arrow-serialization

Conversation

@jamesbroadhead
Copy link
Copy Markdown
Contributor

Summary

  • row.toJSON() from apache-arrow preserves BigInt for Int64 columns. When this data is later serialized with JSON.stringify() for SSE streaming, it throws TypeError: Do not know how to serialize a BigInt.
  • Convert BigInt values to strings in _transformArrowAttachment during row transformation, before the data reaches JSON serialization.
  • This ensures Arrow IPC deserialization works correctly for serverless warehouses that return inline results as Arrow IPC in result.attachment.

Test plan

  • Verify that queries returning Int64 columns via Arrow IPC attachment no longer throw TypeError: Do not know how to serialize a BigInt during SSE streaming
  • Verify that non-BigInt columns are unaffected by the change
  • Verify that the string representation of BigInt values is correct and usable downstream

This pull request was AI-assisted by Isaac.

Co-authored-by: Isaac
Signed-off-by: James Broadhead <jamesbroadhead@gmail.com>
@jamesbroadhead
Copy link
Copy Markdown
Contributor Author

Superseded by #315 — recreated against the upstream repo so CI workflows can run (fork PRs don't get secrets, hence the broken Lint/Unit/Playground/Docs checks here).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant