From 69148f614070cbb2223cd4f83124fe630ae8f0f2 Mon Sep 17 00:00:00 2001 From: Pierre Rognant Date: Thu, 10 Dec 2020 18:38:14 +0100 Subject: [PATCH 1/4] Update docker logs collection with AD label supporting file tailing --- content/en/agent/amazon_ecs/logs.md | 11 +++++++++++ content/en/agent/docker/log.md | 27 ++++++++++++++++++++++++++- 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/content/en/agent/amazon_ecs/logs.md b/content/en/agent/amazon_ecs/logs.md index 0cb05afa193..60cbd31d79b 100644 --- a/content/en/agent/amazon_ecs/logs.md +++ b/content/en/agent/amazon_ecs/logs.md @@ -111,6 +111,8 @@ To collect all logs written by running applications in your ECS containers and s ### Custom log collection +#### Using configuration file + If your container writes any logs to files, follow the [Custom Log Collection documentation][1] to tail files for logs. To gather logs from your `` application stored in `/.log` create a `.d/conf.yaml` file at the root of your [Agent's configuration directory][2] with the following content: @@ -125,6 +127,13 @@ logs: **Note**: Container metadata is not retrieved with custom log collection, therefore the Agent does not automatically assign container tags to logs. Use [custom tags][3] to create container tags. +#### Using container label + +It is possible to enable file tailing by using container label so the logs collected will get the tags of the container on which the label was set. See this [example][4] that details the exact label to use. + +**Note**: The file path are always relative to the Agent. So this require extra configuration for involved ECS tasks to share a directory between the container writing to the file and the Agent container. Please refere to the [AWS documentation][5] for additional details on volume management with ECS. + + ## Activate log integrations The `source` attribute is used to identify the integration to use for each container. Override it directly in your containers labels to start using [log integrations][1]. Read Datadog's [Autodiscovery guide for logs][2] to learn more about this process. @@ -136,3 +145,5 @@ The `source` attribute is used to identify the integration to use for each conta [1]: /agent/logs/?tab=tailfiles#custom-log-collection [2]: /agent/logs/#custom-log-collection [3]: /getting_started/tagging/assigning_tags/?tab=noncontainerizedenvironments#methods-for-assigning-tags +[4]: /agent/docker/log/?tab=logcollectionfromfile#examples +[5]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html diff --git a/content/en/agent/docker/log.md b/content/en/agent/docker/log.md index 79a70ecf76b..1a04f2c54e5 100644 --- a/content/en/agent/docker/log.md +++ b/content/en/agent/docker/log.md @@ -36,7 +36,7 @@ Configuring log collection depends on your current environment. Choose one of th - If you cannot deploy the containerized Agent and your container writes **all** logs to `stdout`/`stderr`, follow the [host Agent](?tab=hostagent#installation) installation to enable containerized logging within your Agent configuration file. -- If your container writes logs to files (only partially writes logs to `stdout`/`stderr` and writes logs to files OR fully writes logs to files), follow the [host Agent with custom log collection](?tab=hostagentwithcustomlogging#installation) installation. +- If your container writes logs to files (only partially writes logs to `stdout`/`stderr` and writes logs to files OR fully writes logs to files), follow the [host Agent with custom log collection](?tab=hostagentwithcustomlogging#installation) installation or follow the [containerized Agent](?tab=containerized-agent#installation) installation and check the [log collection from file with Autodiscovery configuration example](?tab=logcollectionfromfile#examples). ## Installation @@ -228,6 +228,31 @@ See the [multi-line processing rule documentation][1] to get more pattern exampl [1]: /agent/logs/advanced_log_collection/#multi-line-aggregation +{{% /tab %}} +{{% tab "Log collection from file" %}} + +The Agent can also directly collect logs from file based on a container Autodiscovery label. To collect logs from a file associated to a container use the `com.datadoghq.ad.logs` label as shown below on your containers to get `/logs/app.log` to be collected: + +```yaml +labels: + com.datadoghq.ad.logs: '[{"type":"file", "source": "sample_app", "service": "sample_service", "path": "/logs/app/prod.log"}]' +``` + +Log collected from that file will be tagged with the container metadata. Log collection is linked to the container life cycle, as soon as the container stops, log collection from that file also stops. + + +**Important notes**: + +- The path is **relative** to the Agent, thus the directory containing the file should be shared between the container running the application logging to the file and the Agent container. For example the container may mount `/logs` and each container logging to file may mount a volume like `/logs/app` where the log file will be written. + +- When using this kind of label on a container, its `stderr`/`stdout` logs are not collected automatically. If collection from both `stderr`/`stdout` and a file are needed it should be explicity enabled by using a label like: +```yaml +labels: + com.datadoghq.ad.logs: '[{"type":"file", "source": "java", "service": "app", "path": "/logs/app/prod.log"}, {"type": "docker", "source": "app_container", "service": "app"}]' +``` + +- When using this kind of combination, `source` and `service` have no default value, and therefore they should be explicitely set in the Autodiscovery label. + {{% /tab %}} {{< /tabs >}} From 9fc46c9c59dbd5dae74bfe7aa9098e6a3e1eed8c Mon Sep 17 00:00:00 2001 From: Pierre Rognant Date: Thu, 10 Dec 2020 18:48:28 +0100 Subject: [PATCH 2/4] add version note --- content/en/agent/amazon_ecs/logs.md | 3 +-- content/en/agent/docker/log.md | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/content/en/agent/amazon_ecs/logs.md b/content/en/agent/amazon_ecs/logs.md index 60cbd31d79b..1dbb832459d 100644 --- a/content/en/agent/amazon_ecs/logs.md +++ b/content/en/agent/amazon_ecs/logs.md @@ -129,11 +129,10 @@ logs: #### Using container label -It is possible to enable file tailing by using container label so the logs collected will get the tags of the container on which the label was set. See this [example][4] that details the exact label to use. +With Agent version >= 7.25.0/6.25.0 it is possible to enable file tailing by using container label so the logs collected will get the tags of the container on which the label was set. See this [example][4] that details the exact label to use. **Note**: The file path are always relative to the Agent. So this require extra configuration for involved ECS tasks to share a directory between the container writing to the file and the Agent container. Please refere to the [AWS documentation][5] for additional details on volume management with ECS. - ## Activate log integrations The `source` attribute is used to identify the integration to use for each container. Override it directly in your containers labels to start using [log integrations][1]. Read Datadog's [Autodiscovery guide for logs][2] to learn more about this process. diff --git a/content/en/agent/docker/log.md b/content/en/agent/docker/log.md index 1a04f2c54e5..f3313c9ca66 100644 --- a/content/en/agent/docker/log.md +++ b/content/en/agent/docker/log.md @@ -231,7 +231,7 @@ See the [multi-line processing rule documentation][1] to get more pattern exampl {{% /tab %}} {{% tab "Log collection from file" %}} -The Agent can also directly collect logs from file based on a container Autodiscovery label. To collect logs from a file associated to a container use the `com.datadoghq.ad.logs` label as shown below on your containers to get `/logs/app.log` to be collected: +The Agent (version >= 7.25) can also directly collect logs from file based on a container Autodiscovery label. To collect logs from a file associated to a container use the `com.datadoghq.ad.logs` label as shown below on your containers to get `/logs/app.log` to be collected: ```yaml labels: From d2dc1d5d7e05fe57bfa79f75e57024c1f714cbd3 Mon Sep 17 00:00:00 2001 From: Pierre Rognant <59561183+prognant@users.noreply.github.com> Date: Fri, 11 Dec 2020 10:36:21 +0100 Subject: [PATCH 3/4] Address reviews Co-authored-by: ruthnaebeck <19349244+ruthnaebeck@users.noreply.github.com> --- content/en/agent/amazon_ecs/logs.md | 8 ++++---- content/en/agent/docker/log.md | 12 ++++++------ 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/content/en/agent/amazon_ecs/logs.md b/content/en/agent/amazon_ecs/logs.md index 1dbb832459d..046a8f73ce1 100644 --- a/content/en/agent/amazon_ecs/logs.md +++ b/content/en/agent/amazon_ecs/logs.md @@ -111,7 +111,7 @@ To collect all logs written by running applications in your ECS containers and s ### Custom log collection -#### Using configuration file +#### Configuration file If your container writes any logs to files, follow the [Custom Log Collection documentation][1] to tail files for logs. @@ -127,11 +127,11 @@ logs: **Note**: Container metadata is not retrieved with custom log collection, therefore the Agent does not automatically assign container tags to logs. Use [custom tags][3] to create container tags. -#### Using container label +#### Container label -With Agent version >= 7.25.0/6.25.0 it is possible to enable file tailing by using container label so the logs collected will get the tags of the container on which the label was set. See this [example][4] that details the exact label to use. +With Agent v7.25.0+/6.25.0+, it is possible to enable file tailing by using a container label so the logs collected receive the tags of the container on which the label was set. See this [example][4] that details the exact label to use. -**Note**: The file path are always relative to the Agent. So this require extra configuration for involved ECS tasks to share a directory between the container writing to the file and the Agent container. Please refere to the [AWS documentation][5] for additional details on volume management with ECS. +**Note**: The file paths are always relative to the Agent. So, this requires extra configuration for involved ECS tasks to share a directory between the container writing to the file and the Agent container. Refer to the [AWS documentation][5] for additional details on volume management with ECS. ## Activate log integrations diff --git a/content/en/agent/docker/log.md b/content/en/agent/docker/log.md index f3313c9ca66..80421a89245 100644 --- a/content/en/agent/docker/log.md +++ b/content/en/agent/docker/log.md @@ -229,7 +229,7 @@ See the [multi-line processing rule documentation][1] to get more pattern exampl [1]: /agent/logs/advanced_log_collection/#multi-line-aggregation {{% /tab %}} -{{% tab "Log collection from file" %}} +{{% tab "From file" %}} The Agent (version >= 7.25) can also directly collect logs from file based on a container Autodiscovery label. To collect logs from a file associated to a container use the `com.datadoghq.ad.logs` label as shown below on your containers to get `/logs/app.log` to be collected: @@ -238,20 +238,20 @@ labels: com.datadoghq.ad.logs: '[{"type":"file", "source": "sample_app", "service": "sample_service", "path": "/logs/app/prod.log"}]' ``` -Log collected from that file will be tagged with the container metadata. Log collection is linked to the container life cycle, as soon as the container stops, log collection from that file also stops. +Logs collected from a file are tagged with the container metadata. Log collection is linked to the container life cycle, as soon as the container stops, log collection from that file stops. -**Important notes**: +**Notes**: -- The path is **relative** to the Agent, thus the directory containing the file should be shared between the container running the application logging to the file and the Agent container. For example the container may mount `/logs` and each container logging to file may mount a volume like `/logs/app` where the log file will be written. +- The file path is **relative** to the Agent, so the directory containing the file should be shared between the container running the application and the Agent container. For example, if the container mounts `/logs` each container logging to file may mount a volume such as `/logs/app` where the log file is written. -- When using this kind of label on a container, its `stderr`/`stdout` logs are not collected automatically. If collection from both `stderr`/`stdout` and a file are needed it should be explicity enabled by using a label like: +- When using this kind of label on a container, its `stderr`/`stdout` logs are not collected automatically. If collection from both `stderr`/`stdout` and a file are needed it should be explicity enabled by using a label, for example: ```yaml labels: com.datadoghq.ad.logs: '[{"type":"file", "source": "java", "service": "app", "path": "/logs/app/prod.log"}, {"type": "docker", "source": "app_container", "service": "app"}]' ``` -- When using this kind of combination, `source` and `service` have no default value, and therefore they should be explicitely set in the Autodiscovery label. +- When using this kind of combination, `source` and `service` have no default value and should be explicitly set in the Autodiscovery label. {{% /tab %}} {{< /tabs >}} From c06ab035d5c2f9f88bcc703ddec62c1cd1015a43 Mon Sep 17 00:00:00 2001 From: Pierre Rognant Date: Fri, 11 Dec 2020 10:43:58 +0100 Subject: [PATCH 4/4] Address reviews (2) --- content/en/agent/docker/log.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/agent/docker/log.md b/content/en/agent/docker/log.md index 80421a89245..30964575bb6 100644 --- a/content/en/agent/docker/log.md +++ b/content/en/agent/docker/log.md @@ -231,7 +231,7 @@ See the [multi-line processing rule documentation][1] to get more pattern exampl {{% /tab %}} {{% tab "From file" %}} -The Agent (version >= 7.25) can also directly collect logs from file based on a container Autodiscovery label. To collect logs from a file associated to a container use the `com.datadoghq.ad.logs` label as shown below on your containers to get `/logs/app.log` to be collected: +The Agent v7.25.0+/6.25.0+ can directly collect logs from a file based on a container Autodiscovery label. To collect these logs, use the `com.datadoghq.ad.logs` label as shown below on your containers to collect `/logs/app.log`: ```yaml labels: