Skip to content

Exact Cardinality Count extension #18021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FrankChen021 merged 71 commits into from
Jul 23, 2025
Merged

Exact Cardinality Count extension #18021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
3b2de51
Add Bitmap64 extension
Jan 27, 2025
7b3f455
Update max intermediate size
GWphua Mar 17, 2025
52c3c48
Changes to Bitmap64 after build
GWphua May 20, 2025
9c8d669
Replace cardinality count name
GWphua May 20, 2025
2aec5b6
Unit tests for bitmap64
GWphua May 20, 2025
bb28202
Unit tests for bitmap aggregator
GWphua May 20, 2025
8aeb38d
tidy up for tests and Counter
GWphua May 20, 2025
703c7a9
Fix checkstyle
GWphua May 20, 2025
cafa112
Tidy README
GWphua May 20, 2025
53d30b3
Tidy review suggestions
GWphua May 21, 2025
9b28766
Fix checkstyle
GWphua May 21, 2025
cb52b82
Clean up dependencies in pom file
GWphua May 21, 2025
1db8df4
Docs for extension
GWphua May 21, 2025
b7b5a04
Fix Spelling check
GWphua May 21, 2025
83dfef9
Add wikipedia datasource walkthrough
GWphua May 21, 2025
9310334
Add SQL Test
GWphua May 22, 2025
39b7ce5
Fix Cannot find class error
GWphua May 22, 2025
f552150
Fix Assert cannot be found
GWphua May 22, 2025
ddfc4e5
Fix test case
GWphua May 22, 2025
074b2ba
Fix SqlAggregator test cases
GWphua May 23, 2025
e2a1afe
Fix checkstyle
GWphua May 23, 2025
683a06a
Fix registerSerde problem
GWphua May 23, 2025
df0725e
Add JavaDocs for interface
GWphua May 23, 2025
3c67580
Fix null check not passing
GWphua May 23, 2025
033db62
Add dockerfiles for DruidExactCardinalityIT
GWphua May 27, 2025
2285590
Add integration tests for Druid Exact Cardinality
GWphua May 27, 2025
608d93c
Merge branch 'apache:master' into integration-test
GWphua May 27, 2025
2a8164a
Checkstyle
GWphua May 27, 2025
b897b86
Change groupId to contrib
GWphua May 28, 2025
76d5921
Change docker file to keep things running
GWphua May 28, 2025
2bde0d1
Set fullDatasourceName after initialization to prevent NPE with config
GWphua May 28, 2025
6ea8f13
Fix adding contrib pom changes to distribution/pom.xml
GWphua May 28, 2025
ba4716f
Setup IT for contrib packages
GWphua May 28, 2025
c45827b
Add DruidExactCardinality into IT
GWphua May 28, 2025
74b0d19
Add case to check cardinality ignores duplicates
GWphua May 29, 2025
58c5c00
Add case to check cardinality works on rolled-up columns:
GWphua May 29, 2025
1dea8f0
Fix fullDatasourceName can be a local variable
GWphua May 30, 2025
029de04
Bitmap64 exactcount update
GWphua Jun 4, 2025
3c63fc4
Add ExactCount extension into github actions
GWphua Jun 4, 2025
05187e5
Fix resources/cluster docker file
GWphua Jun 4, 2025
5f8fe81
Add additional line to eof
GWphua Jun 4, 2025
5a19eff
Change docs to specify columns
GWphua Jun 4, 2025
2d5a7f5
Eliminate unnecessary byte array copying
GWphua Jun 4, 2025
45c48f6
Change Bitmap64 function to filter out numeric types
GWphua Jun 4, 2025
67a059e
Prevent copying of array in Output stream
GWphua Jun 4, 2025
33a623b
Fix checkstyle
GWphua Jun 4, 2025
9b16a5b
Fix UnsupportedOperationException
GWphua Jun 5, 2025
89413d5
Fix SQL type allowable
GWphua Jun 5, 2025
d583ce7
Remove unnecessary build config
GWphua Jun 5, 2025
1b48891
Add unit test for String Column
GWphua Jun 16, 2025
e97d808
Type checking for String in ExactCount
GWphua Jun 16, 2025
21719bf
Implement unit tests to confirm String column check
GWphua Jun 16, 2025
74fcc25
Update difference between exact count + distinct count
GWphua Jun 17, 2025
c89d458
Fix spelling mistake
GWphua Jun 17, 2025
5153349
Merge remote-tracking branch 'origin/master' into bitmap64-extension
GWphua Jul 10, 2025
42cd526
Address review feedback P1 - Rework ExposedByteArrayOutputStream
GWphua Jul 10, 2025
f458342
Address unclear exception message when decoding using base64
GWphua Jul 10, 2025
51bb045
Comments for 1KiB max intermediate size
GWphua Jul 10, 2025
d488bcf
Allow Bitmap64 to take directly from data input
GWphua Jul 10, 2025
1a6c113
Use underlying byte array
GWphua Jul 10, 2025
d5d5b15
Checkstyle
GWphua Jul 10, 2025
6ce90c2
Fix array writing
GWphua Jul 11, 2025
c934ce1
Rename to druid-bitmap-exact-count
GWphua Jul 11, 2025
96e5887
Missing rename
GWphua Jul 11, 2025
1dc14ed
Merge remote-tracking branch 'origin/master' into bitmap64-extension
GWphua Jul 11, 2025
e6173b6
Update to snapshot 35
GWphua Jul 11, 2025
26fa914
Renamed to druid-exact-count-bitmap
GWphua Jul 11, 2025
ec7e184
Merge branch 'master' into bitmap64-extension
GWphua Jul 17, 2025
fb0131a
Druid exact count bitmap
GWphua Jul 17, 2025
f2f28e1
Merge remote-tracking branch 'origin/master' into bitmap64-extension
GWphua Jul 22, 2025
494f7c5
Merge remote-tracking branch 'origin/master' into bitmap64-extension
GWphua Jul 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/revised-its.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ jobs:
fail-fast: false
matrix:
jdk: [17]
it: [HighAvailability, MultiStageQuery, Catalog, BatchIndex, MultiStageQueryWithMM, InputSource, InputFormat, Security, Query]
it: [HighAvailability, MultiStageQuery, Catalog, BatchIndex, MultiStageQueryWithMM, InputSource, InputFormat, Security, Query, DruidExactCountBitmap]
indexer: [middleManager]
uses: ./.github/workflows/reusable-revised-its.yml
if: ${{ needs.changes.outputs.core == 'true' || needs.changes.outputs.common-extensions == 'true' }}
Expand Down
2 changes: 2 additions & 0 deletions distribution/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,6 +462,8 @@
<argument>org.apache.druid.extensions.contrib:grpc-query</argument>
<argument>-c</argument>
<argument>org.apache.druid.extensions.contrib:druid-ranger-security</argument>
<argument>-c</argument>
<argument>org.apache.druid.extensions.contrib:druid-exact-count-bitmap</argument>
</arguments>
</configuration>
</execution>
Expand Down
1 change: 1 addition & 0 deletions docs/configuration/extensions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ All of these community extensions can be downloaded using [pull-deps](../operati
|druid-ddsketch|Support for DDSketch approximate quantiles based on [DDSketch](https://github.com/datadog/sketches-java) | [link](../development/extensions-contrib/ddsketch-quantiles.md)|
|druid-deltalake-extensions|Support for ingesting Delta Lake tables.|[link](../development/extensions-contrib/delta-lake.md)|
|druid-distinctcount|DistinctCount aggregator|[link](../development/extensions-contrib/distinctcount.md)|
|druid-exact-count-bitmap|Support for exact cardinality counting using Roaring Bitmap over a Long column.|[link](../development/extensions-contrib/druid-exact-count-bitmap.md)|
|druid-iceberg-extensions|Support for ingesting Iceberg tables.|[link](../development/extensions-contrib/iceberg.md)|
|druid-redis-cache|A cache implementation for Druid based on Redis.|[link](../development/extensions-contrib/redis-cache.md)|
|druid-time-min-max|Min/Max aggregator for timestamp.|[link](../development/extensions-contrib/time-min-max.md)|
Expand Down
Loading
Loading