From 5dac612e6d3ea46ac469b33ce5d51e39a403fbca Mon Sep 17 00:00:00 2001 From: mats Date: Tue, 7 Apr 2026 10:40:33 +0900 Subject: [PATCH 1/3] docs: outputs: add ZeroBus output plugin documentation - Introduced new ZeroBus output plugin for sending logs to Databricks via the ZeroBus streaming ingestion interface. - Updated SUMMARY.md to include ZeroBus in the list of output plugins. - Provided detailed configuration parameters, usage examples, and record format transformations for the ZeroBus plugin. Signed-off-by: mats --- SUMMARY.md | 1 + pipeline/outputs/zerobus.md | 155 ++++++++++++++++++++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 pipeline/outputs/zerobus.md diff --git a/SUMMARY.md b/SUMMARY.md index c1512f972..ff42dd524 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -221,6 +221,7 @@ * [UDP](pipeline/outputs/udp.md) * [Vivo Exporter](pipeline/outputs/vivo-exporter.md) * [WebSocket](pipeline/outputs/websocket.md) + * [ZeroBus](pipeline/outputs/zerobus.md) ## Stream processing diff --git a/pipeline/outputs/zerobus.md b/pipeline/outputs/zerobus.md new file mode 100644 index 000000000..14f7b38a5 --- /dev/null +++ b/pipeline/outputs/zerobus.md @@ -0,0 +1,155 @@ +--- +description: Send logs to Databricks via ZeroBus +--- + +# ZeroBus + +{% hint style="info" %} +**Supported event types:** `logs` +{% endhint %} + +The _ZeroBus_ output plugin lets you ingest log records into a [Databricks](https://www.databricks.com/) table through the ZeroBus streaming ingestion interface. Records are converted to JSON and sent via the ZeroBus SDK using gRPC. + +Before you begin, you need a Databricks workspace with a Unity Catalog table configured for ZeroBus ingestion, and an OAuth2 service principal (client ID and client secret) with appropriate permissions. + +## Configuration parameters + +| Key | Description | Default | +| :--- | :--- | :--- | +| `endpoint` | ZeroBus gRPC endpoint URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | +| `workspace_url` | Databricks workspace URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | +| `table_name` | Fully qualified Unity Catalog table name in `catalog.schema.table` format. | _none_ | +| `client_id` | OAuth2 client ID for authentication. | _none_ | +| `client_secret` | OAuth2 client secret for authentication. | _none_ | +| `add_tag` | If enabled, the Fluent Bit tag is added as a `_tag` field in each record. | `true` | +| `time_key` | Key name for the injected timestamp. The timestamp is formatted as RFC 3339 with nanosecond precision. Set to an empty string to disable timestamp injection. | `_time` | +| `log_key` | Comma-separated list of record keys to include in the output. When unset, all keys are included. | _none_ | +| `raw_log_key` | If set, the full original record (before filtering by `log_key`) is stored as a JSON string under this key name. | _none_ | + +## Get started + +To send log records to Databricks via ZeroBus, configure the plugin with your ZeroBus endpoint, workspace URL, table name, and OAuth2 credentials. + +### Configuration file + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: tail + tag: app.logs + path: /var/log/app/*.log + + outputs: + - name: zerobus + match: '*' + endpoint: https://.zerobus..cloud.databricks.com + workspace_url: https://.cloud.databricks.com + table_name: catalog.schema.logs + client_id: + client_secret: +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name tail + Tag app.logs + Path /var/log/app/*.log + +[OUTPUT] + Name zerobus + Match * + Endpoint https://.zerobus..cloud.databricks.com + Workspace_Url https://.cloud.databricks.com + Table_Name catalog.schema.logs + Client_Id + Client_Secret +``` + +{% endtab %} +{% endtabs %} + +### Record format + +Each log record is converted to a JSON object before ingestion. The plugin applies the following transformations in order: + +1. If `raw_log_key` is set, the full original record is captured as a JSON string before any filtering. +2. If `log_key` is set, only the specified keys are included in the output record. +3. If `raw_log_key` is set, the captured JSON string is injected under the configured key (unless a key with that name already exists). +4. If `time_key` is set, a timestamp in RFC 3339 format with nanosecond precision (for example, `2024-01-15T10:30:00.123456789Z`) is injected (unless a key with that name already exists). +5. If `add_tag` is enabled, the Fluent Bit tag is injected as `_tag` (unless a key with that name already exists). + +For example, given the following input record: + +```json +{"level": "info", "message": "request completed", "status": 200} +``` + +The default configuration produces: + +```json +{ + "level": "info", + "message": "request completed", + "status": 200, + "_time": "2024-01-15T10:30:00.123456789Z", + "_tag": "app.logs" +} +``` + +### Filtering keys + +Use `log_key` to select specific fields from the record. Combined with `raw_log_key`, you can send a filtered record while preserving the original data: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + outputs: + - name: zerobus + match: '*' + endpoint: https://.zerobus..cloud.databricks.com + workspace_url: https://.cloud.databricks.com + table_name: catalog.schema.logs + client_id: + client_secret: + log_key: level,message + raw_log_key: _raw +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[OUTPUT] + Name zerobus + Match * + Endpoint https://.zerobus..cloud.databricks.com + Workspace_Url https://.cloud.databricks.com + Table_Name catalog.schema.logs + Client_Id + Client_Secret + Log_Key level,message + Raw_Log_Key _raw +``` + +{% endtab %} +{% endtabs %} + +This produces: + +```json +{ + "level": "info", + "message": "request completed", + "_raw": "{\"level\":\"info\",\"message\":\"request completed\",\"status\":200}", + "_time": "2024-01-15T10:30:00.123456789Z", + "_tag": "app.logs" +} +``` From 1c4b72f02d9f21df2c7daae5ebec4a1ceae5cc14 Mon Sep 17 00:00:00 2001 From: mats Date: Thu, 9 Apr 2026 17:23:56 +0900 Subject: [PATCH 2/3] docs: outputs: correct spelling of Zerobus in documentation - Updated instances of "ZeroBus" to "Zerobus" for consistency throughout the documentation. - Ensured accurate representation of the Zerobus output plugin and its configuration parameters. Signed-off-by: mats --- pipeline/outputs/zerobus.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pipeline/outputs/zerobus.md b/pipeline/outputs/zerobus.md index 14f7b38a5..c6e55eb94 100644 --- a/pipeline/outputs/zerobus.md +++ b/pipeline/outputs/zerobus.md @@ -1,22 +1,22 @@ --- -description: Send logs to Databricks via ZeroBus +description: Send logs to Databricks via Zerobus --- -# ZeroBus +# Zerobus {% hint style="info" %} **Supported event types:** `logs` {% endhint %} -The _ZeroBus_ output plugin lets you ingest log records into a [Databricks](https://www.databricks.com/) table through the ZeroBus streaming ingestion interface. Records are converted to JSON and sent via the ZeroBus SDK using gRPC. +The _Zerobus_ output plugin lets you ingest log records into a [Databricks](https://www.databricks.com/) table through the Zerobus streaming ingestion interface. Records are converted to JSON and sent via the Zerobus SDK using gRPC. -Before you begin, you need a Databricks workspace with a Unity Catalog table configured for ZeroBus ingestion, and an OAuth2 service principal (client ID and client secret) with appropriate permissions. +Before you begin, you need a Databricks workspace with a Unity Catalog table configured for Zerobus ingestion, and an OAuth2 service principal (client ID and client secret) with appropriate permissions. ## Configuration parameters | Key | Description | Default | | :--- | :--- | :--- | -| `endpoint` | ZeroBus gRPC endpoint URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | +| `endpoint` | Zerobus gRPC endpoint URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | | `workspace_url` | Databricks workspace URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | | `table_name` | Fully qualified Unity Catalog table name in `catalog.schema.table` format. | _none_ | | `client_id` | OAuth2 client ID for authentication. | _none_ | @@ -28,7 +28,7 @@ Before you begin, you need a Databricks workspace with a Unity Catalog table con ## Get started -To send log records to Databricks via ZeroBus, configure the plugin with your ZeroBus endpoint, workspace URL, table name, and OAuth2 credentials. +To send log records to Databricks via Zerobus, configure the plugin with your Zerobus endpoint, workspace URL, table name, and OAuth2 credentials. ### Configuration file From 8fd9703f413fa746d448f4b76c4cb0de1a89d6fd Mon Sep 17 00:00:00 2001 From: mats Date: Thu, 9 Apr 2026 17:28:04 +0900 Subject: [PATCH 3/3] docs: outputs: update Zerobus documentation for clarity and consistency - Revised the description to use "through" instead of "via" for improved readability. - Clarified the OAuth2 terminology to "OAuth 2.0" for consistency. - Ensured consistent language throughout the documentation regarding Zerobus and its configuration parameters. Signed-off-by: mats --- pipeline/outputs/zerobus.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pipeline/outputs/zerobus.md b/pipeline/outputs/zerobus.md index c6e55eb94..385d0ef55 100644 --- a/pipeline/outputs/zerobus.md +++ b/pipeline/outputs/zerobus.md @@ -1,5 +1,5 @@ --- -description: Send logs to Databricks via Zerobus +description: Send logs to Databricks through Zerobus --- # Zerobus @@ -8,9 +8,9 @@ description: Send logs to Databricks via Zerobus **Supported event types:** `logs` {% endhint %} -The _Zerobus_ output plugin lets you ingest log records into a [Databricks](https://www.databricks.com/) table through the Zerobus streaming ingestion interface. Records are converted to JSON and sent via the Zerobus SDK using gRPC. +The _Zerobus_ output plugin lets you ingest log records into a [Databricks](https://www.databricks.com/) table through the Zerobus streaming ingestion interface. Records are converted to JSON and sent by using the Zerobus SDK using gRPC. -Before you begin, you need a Databricks workspace with a Unity Catalog table configured for Zerobus ingestion, and an OAuth2 service principal (client ID and client secret) with appropriate permissions. +Before you begin, you need a Databricks workspace with a Unity Catalog table configured for Zerobus ingestion, and an OAuth 2.0 service principal (client ID and client secret) with appropriate permissions. ## Configuration parameters @@ -19,8 +19,8 @@ Before you begin, you need a Databricks workspace with a Unity Catalog table con | `endpoint` | Zerobus gRPC endpoint URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | | `workspace_url` | Databricks workspace URL. If no scheme is provided, `https://` is automatically prepended. | _none_ | | `table_name` | Fully qualified Unity Catalog table name in `catalog.schema.table` format. | _none_ | -| `client_id` | OAuth2 client ID for authentication. | _none_ | -| `client_secret` | OAuth2 client secret for authentication. | _none_ | +| `client_id` | OAuth 2.0 client ID for authentication. | _none_ | +| `client_secret` | OAuth 2.0 client secret for authentication. | _none_ | | `add_tag` | If enabled, the Fluent Bit tag is added as a `_tag` field in each record. | `true` | | `time_key` | Key name for the injected timestamp. The timestamp is formatted as RFC 3339 with nanosecond precision. Set to an empty string to disable timestamp injection. | `_time` | | `log_key` | Comma-separated list of record keys to include in the output. When unset, all keys are included. | _none_ | @@ -28,7 +28,7 @@ Before you begin, you need a Databricks workspace with a Unity Catalog table con ## Get started -To send log records to Databricks via Zerobus, configure the plugin with your Zerobus endpoint, workspace URL, table name, and OAuth2 credentials. +To send log records to Databricks through Zerobus, configure the plugin with your Zerobus endpoint, workspace URL, table name, and OAuth 2.0 credentials. ### Configuration file