Skip to content

new environment vars for external druid and kafka + jupyter template#14592

Merged
techdocsmith merged 4 commits intoapache:masterfrom
sergioferragut:jupyter-template
Jul 25, 2023
Merged

new environment vars for external druid and kafka + jupyter template#14592
techdocsmith merged 4 commits intoapache:masterfrom
sergioferragut:jupyter-template

Conversation

@sergioferragut
Copy link
Copy Markdown
Contributor

@sergioferragut sergioferragut commented Jul 16, 2023

Description

The goal of this PR is to parameterize the hostnames for the Kafka and Druid instances such that the jupyter lab environment and its set of notebooks can run on against a variety of deployments. It also standardizes how notebooks are created such that they make use of these environment variables when connecting to Kafka or Druid.

Adjusted the docker compose files to use defaults for
DRUID_VERSION -> defaults to 26.0
DRUID_HOST -> defaults to "router" which works with Druid deployed with the docker compose
KAFKA_HOST -> defaults to "kafka" which works with Kafka deployed with docker compose

Also added a jupyter notebook template @`examples/ that shows how to initialize both kafka and druid host variables in order to make the notebooks work with this parameterized hosts approach.

Using these variables users can use the notebooks on other instances of Druid and/or Kafka.
Examples:

  • If you run both Kafka and Druid standalone in the local system and only run jupyter through docker compose:
    KAFKA_HOST=host.docker.internal DRUID_HOST=host.docker.internal docker compose --profile jupyter up -d
  • If you run Druid standalone in the local system and run kafka and jupyter through docker compose:
    DRUID_HOST=host.docker.internal docker compose --profile kafka-jupyter up -d
  • Self-contained startup with all services running:
    docker compose --profile all-services up -d

This PR has:

  • [ x ] been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • [ x ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • [ x ] been tested in a test Druid cluster.

Copy link
Copy Markdown
Contributor

@techdocsmith techdocsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit

Copy link
Copy Markdown
Contributor

@techdocsmith techdocsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@techdocsmith techdocsmith merged commit 6b229f5 into apache:master Jul 25, 2023
@sergioferragut sergioferragut deleted the jupyter-template branch July 25, 2023 19:26
@LakshSingla LakshSingla added this to the 28.0 milestone Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants