rework cursor creation

[clintropolis]

Description

This PR reworks Cursor and VectorCursor creation from CursorFactory (StorageAdapter) to be more efficient and push additional details about the query into the process with a newly introduced CursorBuildSpec. A new CursorHolder interface has been added that is constructed by another new interface, CursorHolderFactory with a method makeCursorHolder, which accepts the CursorBuildSpec and replaces the arguments of CursorFactory.makeCursors and CursorFactory.makeVectorCursors.

The primary goal here is to make it easy to push these details down to cursor creation for an upcoming feature under development, projections, which are effectively a type of materialized views that live within segments and will be created at ingestion time. More details on this coming later once I finish writing the proposal, but basically we need to know stuff like what grouping columns, filters, aggregations, etc, are involved in a query in order to be able to automatically select the appropriate projection available within the segment, or using the regular default segment rows if no matching projections are available.

In addition to the primary goal, this refactor also allows cursor creation to be more efficient. Since CursorHolder is created from the CursorBuildSpec, the canVectorize, asCursor, and asVectorCursor methods can all share and re-use the same resources.

Finally, since this PR is introducing new cursor building interfaces, it is also taking the opportunity to downplay the prominence of time ordering, particularly query granularity and its role in cursors. CursorFactory.makeCursors previously returned a Sequence<Cursor>, corresponding to query granularity buckets. It has been transitioned to no longer do this in CursorHolder.asCursor, instead returning a single Cursor, equivalent to using ALL granularity with the previous method but without being wrapped in a Sequence. Query granularity in engines is now handled using a new CursorGranularizer which is equivalent to how the vectorized engine works, simplifying query engines which do not care about query granularity, and making things easier for engines in the future to have segments which are not sorted by time.

StorageAdapter is a @PublicApi, and extends CursorFactory, so a default implementation of CursorHolderFactory that is backed by the old canVectorize, makeCursors, and makeVectorCursor methods is provided. That said, all of the built-in query engines and tests have been migrated to use makeCursorHolder, all CursorFactory implementations have been migrated to CursorHolderFactory, and the old methods of CursorFactory have been marked as @Deprecated, and implementations of CursorFactory have been completely removed.

summary of changes

• Added CursorBuildSpec which captures all of the 'interesting' stuff that goes into producing a cursor as a replacement for the method arguments of CursorFactory.canVectorize, CursorFactory.makeCursor, and CursorFactory.makeVectorCursor
• added new interface CursorHolder and new interface CursorHolderFactory as a replacement for CursorFactory, with method makeCursorHolder, which takes a CursorBuildSpec as an argument and replaces CursorFactory.canVectorize, CursorFactory.makeCursor, and CursorFactory.makeVectorCursor
• CursorFactory.makeCursors previously returned a Sequence<Cursor> corresponding to the query granularity buckets, with a separate Cursor per bucket. CursorHolder.asCursor instead returns a single Cursor (equivalent to 'ALL' granularity), and a new CursorGranularizer has been added for query engines to iterate over the cursor and divide into granularity buckets. This makes the non-vectorized engine behave the same way as the vectorized query engine (with its VectorCursorGranularizer), and simplifies a lot of stuff that has to read segments particularly if it does not care about bucketing the results into granularities.
• Deprecated CursorFactory, CursorFactory.canVectorize, CursorFactory.makeCursors, and CursorFactory.makeVectorCursor
• updated all StorageAdapter implementations to implement makeCursorHolder, transitioned direct CursorFactory implementations to instead implement CursorMakerFactory. StorageAdapter being a CursorMakerFactory is intended to be a transitional thing, ideally will not be released in favor of moving CursorMakerFactory to be fetched directly from Segment, however this PR was already large enough so this will be done in a follow-up.
• updated all query engines to use makeCursorHolder, granularity based engines to use CursorGranularizer.

Release note

(for developers)

Use notes from #16985 instead

There is a change to the StorageAdapter interface, which is a 'public' api for extension writers to add additional types of segments to participate in Druids query engines. StorageAdapter extends the CursorFactory interface, whose methods canVectorize, makeCursors, and makeVectorCursor have been deprecated. This interface is replaced with a new CursorHolderFactory, which StorageAdapter implements, with method makeCursorHolder, which accepts a CursorBuildSpec and returns a new interface CursorHolder which defines no argument versions of canVectorize, asCursor, and asVectorCursor.

A default implementation of makeCursorHolder is provided that uses the existing deprecated methods, so no immediate action is needed for StorageAdapter implementors on upgrade, but implementors should plan in the future to migrate to implementing CursorHolderFactory directly. Custom query engines can no longer use these deprecated methods, as they have been removed from all StorageAdapter implementations. Use CursorHolder instead. CursorHolder.asCursor is equivalent to calling CursorFactory.getCursors with ALL granularity. Any custom engines which need to bucket the cursor by query granularity should instead use CursorGranularizer, which can be used to produce a set of intervals and advance the cursor within those intervals.

(for users)
Due to some internal refactoring, TopN queries no longer can use the "pooled" algorithms internally for query granularities other than 'ALL', unless the columns value cardinality is smaller than the internally computed "numValuesPerPass", which is based on processing buffer size and projected aggregator size. This is not completely obvious from a users perspective, so the only indicator might be a performance difference for TopN queries with other query granularities. SQL will only ever plan to a TopN query with 'ALL' granularity so it is not affected.

---

Key changed/added classes in this PR

• CursorHolderFactory
• CursorHolder
• CursorBuildSpec
• CursorGranularizer
• Cursor
• Offset
• CursorFactory (deprecated)
• GroupingEngine (to use CursorHolder and CursorGranularizer)
• TimeseriesQueryEngine (to use CursorHolder and CursorGranularizer)
• TopNQueryEngine (to use CursorHolder and CursorGranularizer)
• All other built-in query engines to use CursorHolder
• All tests that use CursorFactory methods now use CursorHolder

---

This PR has:

• been self-reviewed.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• been tested in a test Druid cluster.

[asdf2014]

This method needs to be removed as it is never used to pass the intellij-inspections check in Github Action