add hive catalog table properties docs and refactor #2941
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request adds comprehensive documentation explaining the interaction between Iceberg table properties and HMS (Hive Metastore) table properties, along with a minor refactoring to improve variable naming clarity.
Changes:
- Added detailed documentation comments explaining how Iceberg and HMS properties are synchronized during table commits
- Refactored variable names to be more explicit (
new_parameters→new_iceberg_properties,removed_keys→deleted_iceberg_properties,merged_params→existing_hms_parameters) - Updated inline comments to better describe the merge operation
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
pyiceberg/catalog/hive.py
Outdated
| # | ||
| # While it is possible to modify HMS table properties through this API, it is not recommended: | ||
| # - New/Updated HMS table properties will also be stored in Iceberg metadata (even though it's HMS-specific) | ||
| # - HMS properties cannot be deleted since they are not visible to Iceberg |
There was a problem hiding this comment.
Can we be more explicit here like
HMS-native properties (set outside Iceberg) cannot be deleted since they are not visible to Iceberg
because in cases like below, it can actually delete HMS properties?
# Set via Iceberg
table.updateProperties().set("hive.table_prop", "true").commit()
# → Goes to BOTH Iceberg metadata AND HMS
# → Now it's "visible" to Iceberg (tracked in Iceberg metadata)
# Unset via Iceberg
table.updateProperties().remove("hive.table_prop").commit()
# → deleted_iceberg_properties = {"hive.table_prop"}
# → Removed from BOTH Iceberg metadata AND HMS ✅
Fokko
left a comment
Fokko
left a comment
There was a problem hiding this comment.
Thanks for clarifying this @kevinjqliu 👍
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| merged_params.pop(key, None) | ||
| merged_params.update(new_parameters) | ||
| hive_table.parameters = merged_params | ||
| deleted_iceberg_properties = current_table.properties.keys() - updated_staged_table.properties.keys() |
There was a problem hiding this comment.
nit: this is non blocking but seems like this logic is technically still "constructing parameters". wdyt of moving this into constructing parameters and adding these comments as a python docstring?
4 participants
Rationale for this change
Follow up to #2927
Add notes on updating iceberg table properties and hive table properties when using the hive catalog (HMS)
Minor refactor on variable naming to be more explicit
Are these changes tested?
Are there any user-facing changes?