You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 14, 2024. It is now read-only.
When incremental refresh is called on hive-partitioned data, if partitoin columns are part of index columns (indexed/included), they are not picked by the refresh call and are filled with nulls.
To Reproduce
create hive partitioned data. e.g. df.write.partitionBy("c1").parquet...
create index where c1 is used in either indexed or included columns
append new data to source
hs.refresh("index", "incremental")
check index data. 'c1' column will be filled with 'nulls' for all appended data.
Expected behavior
'c1' column should contain proper partition values instead of nulls