[core] Fix stats filter by pushdown after schema evolution #5825
Conversation
yuzelin
changed the title
|
@JingsongLi can we delete |
|
SchemaEvolutionTableTestBase and related tests are removed. |
now which tests can cover the cases here? |
I will check all the tests and find tests that can be kept. |
| Collections.singletonList(filter), | ||
| isKeyStats, | ||
| nullSafe)); | ||
| return devolved.isEmpty() ? null : devolved.get(0); |
There was a problem hiding this comment.
Here should consider and filter, should deal with multiple predicates situation.
|
|
||
| // Stats filter is unsafe if it should be devolved and the devolving is unsafe | ||
| // null safe: it's safe if the predicate field is added later and filter it on the old schema | ||
| public boolean statsFilterUnsafe(ManifestEntry entry, Predicate statsFilter) { |
There was a problem hiding this comment.
Should not return bool, we should return predicate, here should consider and filter, should deal with multiple predicates situation.
| this.bucketSelectConverter = bucketSelectConverter; | ||
| this.fieldKeyStatsConverters = | ||
| new SimpleStatsEvolutions( | ||
| sid -> keyValueFieldsExtractor.keyFields(scanTableSchema(sid)), |
There was a problem hiding this comment.
We should not introduce isKeyFilter. We should remove _KEY_ for keyFields here.
|
+1 |
2 participants
Purpose
Reproduce steps:
The reason is stats type has been modified (similar to #4705 ).
Tests
Note:
SchemaEvolutionTest#testAddField for
nullSafearg.API and Format
Documentation