OBSDATA-1354: Druid 25 with confluent patches#136
Merged
Mohammad Ghazanfar (m-ghazanfar) merged 80 commits into25.0.0-confluent-rc1from Jun 6, 2023
Merged
Conversation
The extension packaging included both shaded and unshaded dependencies in the classpath. Shading should not be necessary in this case. Also excludes guava dependencies, which are already provided by Druid and don't need to be added to the extensions jars.
* METRICS-1302: Added prefix support for resource labels. * Addressed review comments. * Added and moved configs to ingestion spec, optimized code. * Addressed review comments * Updated metric dimesnion and other review comments * Flipped ternary operator * Moved from NullHandling to StringUtils. * Removed unnecessary HashMap. * Removed verbosity for instance variables.
* Added getters for configs, labels for distribution metric. * Addressed review comments * Removed extra brackets in JsonProperty.
Align protobuf dependencies to use the main pom one
- fix millisecond resolution being dropped when converting timestamps - remove unnecessary conversion of ByteBuffer to ByteString - make test code a little more concise
- remove the need to parse timestamps into their own column - reduce the number of times we copy maps of labels - pre-size hashmaps and arrays when possible - use loops instead of streams in critical sections Combined these changes improve parsing performance by about 15% - added benchmark for reference
…tFormat (#26) InputRowParsers have been deprecated in favor or InputFormat. This implements the InputFormat version of the OpenCensus Protobuf parser, and deprecates the existing InputRowParser implementation. - the existing InputRowParser behavior is unchanged. - the InputFormat behaves like the InputRowParser, except for the default resource prefix which now defaults to "resource." instead of empty. - both implementations internally delegate to OpenCensusProtobufReader, which is covered by the existing InputRowParser tests.
…roduction of SettableByteEntity (#113) * OBSDATA-483: Adapt opencensus extension to the introduction of SettableByteEntity * OBSDATA-483: Adapt opentelemetry extension to the introduction of SettableByteEntity * OBSDATA-483: Decide which reader to instantiate on read between opencensus and opentelemetry * OBSDATA-483: Add logger config in opencensus tests * OBSDATA-483: Fix issue with opening the byte entity * OBSDATA-483: Instantiate the right iterator in every read request * OBSDATA-483: Add comments * OBSDATA-483: Address Xavier's comments * OBSDATA-483: Remove unused member fields * OBSDATA-483: Rename enum * OBSDATA-483: Fix trace log to actually print the argument * OBSDATA-483: Keep passing the underlying byte buffer and move its position explicitly * OBSDATA-483: Fix checkstyle issues * OBSDATA-483: Add back handling of InvalidProtocolBufferException * OBSDATA-483: Extend the semaphore workflow execution time to 2 hours * Revert "OBSDATA-483: Extend the semaphore workflow execution time to 2 hours" * OBSDATA-483: Don't close iterator in sample
Co-authored-by: Confluent Jenkins Bot <jenkins@confluent.io>
…125) * Upgrade proto version * Fix names and tests - Upgrade version * Fix open census tests * Fix test name
* bumping version of java to 17 for semaphore test run * bumping java version to 17 as per https://github.com/confluentinc/druid/pull/127/files * After speaking with Xavier, made these changes
) (#131) Co-authored-by: Tejaswini Bandlamudi <96047043+tejaswini-imply@users.noreply.github.com>
Member
Sadananda Aithal (saithal-confluent)
left a comment
Sadananda Aithal (saithal-confluent)
left a comment
There was a problem hiding this comment.
Left a few comments. Please check
Mohammad Ghazanfar (m-ghazanfar)
requested a review
from Sadananda Aithal (saithal-confluent)
Sadananda Aithal (saithal-confluent)
approved these changes
Jun 2, 2023
Sadananda Aithal (saithal-confluent)
approved these changes
Jun 2, 2023
| && tar -zxf ./distribution/target/apache-druid-${VERSION}-bin.tar.gz -C /opt \ | ||
| && mv /opt/apache-druid-${VERSION} /opt/druid | ||
| RUN tar -zxf /src/apache-druid-${VERSION}-bin.tar.gz -C /opt \ | ||
| && ln -s /opt/apache-druid-${VERSION} /opt/druid |
Member
There was a problem hiding this comment.
there were quite a few changes upstream to the docker build (e.g. apache#11299) If you've tested the build and it work then we can do it as a follow-up but, since it looks like this was done prior to 24.x and we missed it, but this might be a good time to do it and avoid further drift since it might bite us later.
There was a problem hiding this comment.
Hey Xavier Léauté (@xvrl) !
I have tested the build - I will submit a follow up PR for this.
Mohammad Ghazanfar (m-ghazanfar)
deleted the
OBSDATA-1354-druid-25-with-cflt-patches
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Apply Confluent patches on top of Druid 25.
Druid 25 release notes can be found here
Why suffix the branch wit
-rc1?In case I messup, I don't have to work with IT to delete the
25.0.0-confluentbranch.If 25.0.0-confluent-rc1 works out, I will just create a
25.0.0-confluentbranch from that one. If not, I'll give it another trying using-rc2This section quotes Xavier Léauté (@xvrl) from slack,
Some features of interest to us
Caution
Task management and segment discovery will default to http by default https://github.com/apache/druid/releases/tag/druid-25.0.0#25-upgrade-segment-discovery
Additional commits
The lint would fail. A similar fix has been made on druid 26. See here. I didn't cherry-pick the corresponding commit from apache/druid because it was very large and changes a lot of files.
These changes were created by running the following commands,
Testing
mvn clean package -DskipTests -T1C -X