In order to enable innovation in segment format (see Segment Format and Ser/De tag) I think some system should be developed, so that when somebody want to improve something in the segment format, very little work is required, except actual segment format (and consequently query engines) changes. Currently there is a prohibitive barrier, the mark of this is that almost no improvements were done in segment format in the last few years.
I'm not sure even about the list of requirements, but something that might be included:
Something that could be done, again, very unsure about specifics:
- Druid version is synchronized with segment version?
- Global indexing of serializers, to be independent of class names
- New segment format features become "incubating" and they are enabled by one flag/property (just one for all features), and automatically "promoted" in the next druid version. It should be automatic, without requiring extra PRs.
Please suggest both requirements and ideas.
@nishantmonu51 @gianm @drcrallen @himanshug
In order to enable innovation in segment format (see
Segment Format and Ser/Detag) I think some system should be developed, so that when somebody want to improve something in the segment format, very little work is required, except actual segment format (and consequently query engines) changes. Currently there is a prohibitive barrier, the mark of this is that almost no improvements were done in segment format in the last few years.I'm not sure even about the list of requirements, but something that might be included:
Something that could be done, again, very unsure about specifics:
Please suggest both requirements and ideas.
@nishantmonu51 @gianm @drcrallen @himanshug