Skip to content

Fix Parquet Reader when ingestion need to read columns in filter#16874

Merged
clintropolis merged 1 commit intoapache:masterfrom
maytasm:fix-parquet-reader-filter-column
Aug 14, 2024
Merged

Fix Parquet Reader when ingestion need to read columns in filter#16874
clintropolis merged 1 commit intoapache:masterfrom
maytasm:fix-parquet-reader-filter-column

Conversation

@maytasm
Copy link
Copy Markdown
Contributor

@maytasm maytasm commented Aug 9, 2024

Fix Parquet Reader when ingestion need to read columns in filter

Description

When columns are required in the filter (of the TransformSpec) but are not specified in any of the metricSpec, dimensionSpec, transform (of the TransformSpec), flattenSpec, timestampSpec, then it will not be read for Parquet data files. Since the columns are not read, the filter will end up not matching any rows. This fix makes sure that required columns for the filter are read from the Parquet data files (so that filters can work as expected!).

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@clintropolis clintropolis merged commit c2ddff3 into apache:master Aug 14, 2024
writer-jill added a commit to writer-jill/druid that referenced this pull request Sep 26, 2024
@kfaraz kfaraz added this to the 31.0.0 milestone Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants