Add support for selective loading of broadcast datasources in the task layer

[abhishekrb19]

Motivation

Currently, servers and tasks download all broadcast datasources. While the downloading of bootstrap segments is generally fast, as it happens in parallel during startup, it does consume storage space. This is especially relevant for tasks like kill tasks and MSQ worker tasks, which either don't need to load any broadcast datasources or only selectively load those that are required. This optimization will save task storage space and speed up the task startup time a bit.

Description

The design for loading broadcast datasources follows a similar approach to what was implemented for loading lookups in #16328.

Broadcast datasources can be specified in SQL queries through JOIN and FROM clauses, or obtained from other sources such as lookups.To this effect, we have introduced a BroadcastDatasourceLoadingSpec. Finding the set of broadcast datasources during SQL planning will be done in a follow-up, which will apply only to MSQ tasks, so they load only required broadcast datasources. This PR primarily focuses on the skeletal changes around BroadcastDatasourceLoadingSpec and integrating it from the Task interface via CliPeon to SegmentBootstrapper.

Changes

• Added BroadcastDatasourceLoadingSpec class and Task.getBroadcastDatasourceLoadingSpec() tasks to control how broadcast datasources are loaded.
  • The supported broadcast datasource loading modes are ALL, NONE and ONLY_REQUIRED.
  • By default, all tasks will load all broadcast datasources.
  • The KillTask and MSQControllerTask override getBroadcastDatasourceLoadingSpec() to NONE.
• The CliPeon initializes the broadcast module by parsing the command line option --loadBroadcastDatasourceMode.
• The CLIPeon option --loadBroadcastSegments have been marked as deprecated in favor of --loadBroadcastDatasourceMode. This can be removed in a future release.
• The PodTemplateTaskAdapter and K8sTaskAdapter support wiring for the option LOAD_BROADCAST_DATASOURCE_MODE, similar to existing LOAD_BROADCAST_SEGMENTS.

In summary:

• Data servers load all broadcast datasources.
• By default, all tasks load all broadcast datasources unless an override exists.
• MSQ Controller and Kill tasks do not load any broadcast datasources.
• In a future patch, MSQ ingest and query tasks will selectively load segments pertaining to the broadcast datasources determined as part of query planning. This will be addressed in a follow-up.

Release note:

Tasks control the loading of broadcast datasources via BroadcastDatasourceLoadingSpec getBroadcastDatasourceLoadingSpec(). By default, tasks download all broadcast datasources, unless there's an override as with kill and MSQ controller task.

The CLIPeon command line option --loadBroadcastSegments is deprecated in favor of --loadBroadcastDatasourceMode.

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• been tested in a test Druid cluster.

[abhishekrb19]

The failing UTs are unrelated: other module test failed with OOM (ongoing investigation with heap dumps from #17029) and flaky CDS test.