Skip to content

Azure data lake input source docs #11153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sthetland merged 10 commits into from
Jun 25, 2021
Merged

Azure data lake input source docs #11153

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f66c154
Mention Azure Data Lake
Apr 22, 2021
2258797
Make consistent with other entries
Apr 22, 2021
918a2b4
Update docs/ingestion/native-batch.md
Apr 23, 2021
680fd19
Update docs/ingestion/native-batch.md
Apr 23, 2021
d3eb264
Another reference added to bullet
Apr 23, 2021
fcf76ca
Update docs/ingestion/native-batch.md
Apr 27, 2021
c9af93c
Update docs/ingestion/native-batch.md
Apr 27, 2021
cab1e4a
Merge branch 'master' of github.com:apache/druid into azure-data-lake…
Apr 28, 2021
65a59e5
Merge branch 'azure-data-lake-input-source' of https://github.com/sth…
Jun 25, 2021
6af4806
Merge branch 'master' into azure-data-lake-input-source
Jun 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions docs/ingestion/native-batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ The supported splittable input formats for now are:

- [`s3`](#s3-input-source) reads data from AWS S3 storage.
- [`gs`](#google-cloud-storage-input-source) reads data from Google Cloud Storage.
- [`azure`](#azure-input-source) reads data from Azure Blob Storage.
- [`azure`](#azure-input-source) reads data from Azure Blob Storage and Azure Data Lake.
- [`hdfs`](#hdfs-input-source) reads data from HDFS storage.
- [`http`](#http-input-source) reads data from HTTP servers.
- [`local`](#local-input-source) reads data from local storage.
Expand Down Expand Up @@ -1046,10 +1046,8 @@ Google Cloud Storage object:

> You need to include the [`druid-azure-extensions`](../development/extensions-core/azure.md) as an extension to use the Azure input source.

The Azure input source is to support reading objects directly from Azure Blob store. Objects can be
specified as list of Azure Blob store URI strings. The Azure input source is splittable and can be used
by the [Parallel task](#parallel-task), where each worker task of `index_parallel` will read
a single object.
The Azure input source reads objects directly from Azure Blob store or Azure Data Lake sources. You can
specify objects as a list of file URI strings or prefixes. You can split the Azure input source for use with [Parallel task](#parallel-task) indexing and each worker task reads one chunk of the split data.

Sample specs:

Expand Down Expand Up @@ -1108,17 +1106,17 @@ Sample specs:
|property|description|default|required?|
|--------|-----------|-------|---------|
|type|This should be `azure`.|None|yes|
|uris|JSON array of URIs where Azure Blob objects to be ingested are located. Should be in form "azure://\<container>/\<path-to-file\>"|None|`uris` or `prefixes` or `objects` must be set|
|prefixes|JSON array of URI prefixes for the locations of Azure Blob objects to be ingested. Should be in the form "azure://\<container>/\<prefix\>". Empty objects starting with one of the given prefixes will be skipped.|None|`uris` or `prefixes` or `objects` must be set|
|objects|JSON array of Azure Blob objects to be ingested.|None|`uris` or `prefixes` or `objects` must be set|
|uris|JSON array of URIs where the Azure objects to be ingested are located, in the form "azure://\<container>/\<path-to-file\>"|None|`uris` or `prefixes` or `objects` must be set|
|prefixes|JSON array of URI prefixes for the locations of Azure objects to ingest, in the form "azure://\<container>/\<prefix\>". Empty objects starting with one of the given prefixes are skipped.|None|`uris` or `prefixes` or `objects` must be set|
|objects|JSON array of Azure objects to ingest.|None|`uris` or `prefixes` or `objects` must be set|

Note that the Azure input source will skip all empty objects only when `prefixes` is specified.
Note that the Azure input source skips all empty objects only when `prefixes` is specified.

Azure Blob object:
The `objects` property is:

|property|description|default|required?|
|--------|-----------|-------|---------|
|bucket|Name of the Azure Blob Storage container|None|yes|
|bucket|Name of the Azure Blob Storage or Azure Data Lake container|None|yes|
|path|The path where data is located.|None|yes|

### HDFS Input Source
Expand Down