From ec5d8f7d30a63e385a5458bd52064894bcd5f48a Mon Sep 17 00:00:00 2001 From: Nico Berlee Date: Tue, 16 Dec 2025 10:38:02 +0100 Subject: [PATCH] out_azure_blob: document path templating (refs fluent/fluent-bit#11178) Signed-off-by: Nico Berlee --- pipeline/outputs/azure_blob.md | 107 +++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 39 deletions(-) diff --git a/pipeline/outputs/azure_blob.md b/pipeline/outputs/azure_blob.md index 843af4575..8a83f85f8 100644 --- a/pipeline/outputs/azure_blob.md +++ b/pipeline/outputs/azure_blob.md @@ -20,45 +20,74 @@ Ensure you have an Azure Storage account. [Azure Blob Storage Tutorial (video)]( Fluent Bit exposes the following configuration properties. -| Key | Description | Default | -| :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------- | -| `account_name` | Azure Storage account name. | _none_ | -| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` | -| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `true` | -| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` | -| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` | -| `blob_uri_length` | Set the length of the generated blob URI used when creating and uploading objects to Azure Blob Storage. | `64` | -| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` | -| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` | -| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` | -| `compress` | Sets payload compression in network transfer. Supported values: `gzip`, `zstd`. | _none_ | -| `compress_blob` | Enables compression in the final `blockblob` file. When enabled without `compress`, it uses GZIP; if `compress` is also set, it inherits that codec. This option isn't compatible when `blob_type` = `appendblob`. Fluent Bit returns a configuration error and fails to start. | `false` | -| `configuration_endpoint_bearer_token` | Bearer token for the configuration endpoint. | _none_ | -| `configuration_endpoint_password` | Basic authentication password for the configuration endpoint. | _none_ | -| `configuration_endpoint_url` | Configuration endpoint URL. | _none_ | -| `configuration_endpoint_username` | Basic authentication username for the configuration endpoint. | _none_ | -| `container_name` | Name of the container that will contain the blobs. | _none_ | -| `database_file` | Absolute path to a database file used to store blob file contexts. | _none_ | -| `date_key` | Key name used to store the record timestamp. | `@timestamp` | -| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` | -| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `false` | -| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ | -| `file_delivery_attempt_limit` | Maximum number of delivery attempts for a file. | `1` | -| `io_timeout` | HTTP IO timeout. | `60s` | -| `part_delivery_attempt_limit` | Maximum number of delivery attempts for a file part. | `1` | -| `part_size` | Size of each part when uploading blob files. | `25M` | -| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. For example, setting `path` to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. | _none_ | -| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ | -| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` | -| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ | -| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` | -| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` | -| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` | -| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` | -| `upload_part_freshness_limit` | Maximum lifespan of an uncommitted file part. | `6D` | -| `upload_parts_timeout` | Timeout for uploading parts of a blob file. | `10M` | -| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and haven't reached `upload_file_size` limit yet. | `30m` | -| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | +| Key | Description | Default | +| :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------- | +| `account_name` | Azure Storage account name. | _none_ | +| `auth_type` | Specify the type to authenticate against the service. Supported values: `key`, `sas`. | `key` | +| `auto_create_container` | If `container_name` doesn't exist in the remote service, enabling this option handles the exception and auto-creates the container. | `true` | +| `azure_blob_buffer_key` | Set the Azure Blob buffer key which needs to be specified when using multiple instances of Azure Blob output plugin and buffering is enabled. | `key` | +| `blob_type` | Specify the desired blob type. Supported values: `appendblob`, `blockblob`. | `appendblob` | +| `blob_uri_length` | Set the length of the generated blob URI used when creating and uploading objects to Azure Blob Storage. | `64` | +| `buffer_dir` | Specifies the location of directory where the buffered data will be stored. | `/tmp/fluent-bit/azure-blob/` | +| `buffer_file_delete_early` | Whether to delete the buffered file early after successful blob creation. | `false` | +| `buffering_enabled` | Enable buffering into disk before ingesting into Azure Blob. | `false` | +| `compress` | Sets payload compression in network transfer. Supported values: `gzip`, `zstd`. | _none_ | +| `compress_blob` | Enables compression in the final `blockblob` file. When enabled without `compress`, it uses GZIP; if `compress` is also set, it inherits that codec. This option isn't compatible when `blob_type` = `appendblob`. Fluent Bit returns a configuration error and fails to start. | `false` | +| `configuration_endpoint_bearer_token` | Bearer token for the configuration endpoint. | _none_ | +| `configuration_endpoint_password` | Basic authentication password for the configuration endpoint. | _none_ | +| `configuration_endpoint_url` | Configuration endpoint URL. | _none_ | +| `configuration_endpoint_username` | Basic authentication username for the configuration endpoint. | _none_ | +| `container_name` | Name of the container that will contain the blobs. | _none_ | +| `database_file` | Absolute path to a database file used to store blob file contexts. | _none_ | +| `date_key` | Key name used to store the record timestamp. | `@timestamp` | +| `delete_on_max_upload_error` | Whether to delete the buffer file on maximum upload errors. | `false` | +| `emulator_mode` | To send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option to format the requests in the expected format. | `false` | +| `endpoint` | When using an emulator, this option lets you specify the absolute HTTP address of such service. For example, `http://127.0.0.1:10000`. | _none_ | +| `file_delivery_attempt_limit` | Maximum number of delivery attempts for a file. | `1` | +| `io_timeout` | HTTP IO timeout. | `60s` | +| `part_delivery_attempt_limit` | Maximum number of delivery attempts for a file part. | `1` | +| `part_size` | Size of each part when uploading blob files. | `25M` | +| `path` | Optional. The path to store your blobs. If your blob name is `myblob`, specify subdirectories for storage using `path`. This option also supports [path templating](#path-templating). For example, setting `path` to `/logs/$TAG/%Y/%m/%d/$UUID` stores blobs under a dynamic prefix. | _none_ | +| `sas_token` | Specify the Azure Storage shared access signatures to authenticate against the service. This configuration property is mandatory when `auth_type` is `sas`. | _none_ | +| `scheduler_max_retries` | Maximum number of retries for the scheduler send blob. | `3` | +| `shared_key` | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory when `auth_type` is `key`. | _none_ | +| `store_dir_limit_size` | Set the max size of the buffer directory. | `8G` | +| `tls` | Enable or disable TLS encryption. Azure service requires this to be set to `on`. | `off` | +| `unify_tag` | Whether to create a single buffer file when buffering mode is enabled. | `false` | +| `upload_file_size` | Specifies the size of files to be uploaded in MB. | `200M` | +| `upload_part_freshness_limit` | Maximum lifespan of an uncommitted file part. | `6D` | +| `upload_parts_timeout` | Timeout for uploading parts of a blob file. | `10M` | +| `upload_timeout` | Optional. Specify a timeout for uploads. Fluent Bit will start ingesting buffer files which have been created more than `x` minutes and haven't reached `upload_file_size` limit yet. | `30m` | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `0` | + +### Path templating + +When `path` is set, Fluent Bit resolves the value as a template (similar to the Amazon S3 output) before each upload. The resolved prefix is persisted alongside buffered files, so retries and restarts keep writing to the same Azure path. Leading and trailing slashes are removed automatically to avoid duplicate separators. + +| Token / syntax | Example output | Notes | +| :---------------------------------------------------------------- | :----------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `strftime` directives such as `%Y`, `%m`, `%d`, `%H`, `%M`, `%S` | `2025/12/16/05/42/00` | Uses the event timestamp when available; falls back to the current Fluent Bit time. | +| `%3N` | `123` | Millisecond portion of the event timestamp, zero-padded to three digits. | +| `%9N`, `%L` | `123456789` | Nanosecond portion of the event timestamp, zero-padded to nine digits. `%L` is an alias for `%9N`. | +| `$UUID` | `a1b2c3d4` | Eight-character random suffix generated once per blob so all references for that blob share the same value. | +| `$TAG` | `var.log.containers.app` | Expands to the full tag that matched the output. | +| `$TAG[n]` | `app` | Expands to the nth component (zero-based) of the tag split on dots. This uses Fluent Bit [record accessor syntax](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md); see its [templating limitations](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md#limitations-of-record_accessor-templating). | + +#### Example + +```yaml +pipeline: + outputs: + - name: azure_blob + match: '*' + account_name: YOUR_ACCOUNT_NAME + shared_key: YOUR_SHARED_KEY + container_name: logs + path: "/$TAG[0]/$TAG[4]/%Y/%m/%d/%H/%3N/$UUID" + buffering_enabled: true +``` + +If a chunk arrives with the tag `kube.var.log.containers.app-default`, this configuration creates blobs under `kube/app-default/2025/12/16/05/042/abcd1234/...`. ## Get started