Skip to content

Conversation

@singhpk234
Copy link
Contributor

@singhpk234 singhpk234 commented Dec 30, 2025

About the change

This change attempts to avoid object store call to recreate the TableMetadata from the object store in case we are making a serialized table copy of the table, this could be seen as an optimization where rather than going to the obj store we keep the serialized copy of the table metadata and then just store it and send it via wire for example (spark driver sending to executor) or can also be looked as a feature where if the catalog has redacted some sensitive info, all the serializable table respects it and reads that rather than going to the object store (which it may not have access to) for example:

Presently its gauraded only for RESTTable and for the case when the location is null

I understand that spec wording the following:

Per REST spec the metadata location is nullable in certain cases

metadata-location:
type: string
description: May be null if the table is staged as part of a transaction
nullable: true

The spec says the following but that is SHOULD and not MUST :

The table metadata JSON is returned in the `metadata` field. The corresponding file location of table metadata should be returned in the `metadata-location` field, unless the metadata is not yet committed. For example, a create transaction may return metadata that is staged but not committed.

just wanted to float this to folks on what they think of this feature.

@github-actions github-actions bot added the core label Dec 30, 2025
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the rest-catalog-serializable-table-optimization branch from 8baa35a to 87130b3 Compare December 30, 2025 01:36
@singhpk234 singhpk234 marked this pull request as ready for review January 1, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants