Skip to content

[logs/ad] Support file tailing config from k8s pod annotation#7176

Merged
prognant merged 6 commits intomasterfrom
prognant/ac-720-file-tailing-from-k8s-AD
Jan 20, 2021
Merged

[logs/ad] Support file tailing config from k8s pod annotation#7176
prognant merged 6 commits intomasterfrom
prognant/ac-720-file-tailing-from-k8s-AD

Conversation

@prognant
Copy link
Copy Markdown
Contributor

@prognant prognant commented Jan 14, 2021

What does this PR do?

Make #6883 slightly more generic by supporting annotation from both kubernetes (All runtime) & docker.

Motivation

Feature consistency.

Additional Notes

File path shall be relative to where the agent is running.
Add a slight modification to properly handle multiple logs config in kubernetes launcher, In that extend the following annotation will work :

  • ad.datadoghq.com/logapp.logs: '[{"type":"file","path":"/tmp/share/test.log"}]', will collect log from /tmp/share/test.log and the container log if container_collect_all is set to true
  • ad.datadoghq.com/logapp.logs: '[{"type":"file","path":"/tmp/share/test.log"},{"source":"my_source", "service":"my_service"}]', will collect log from /tmp/share/test.log and the container log with a custom source/service value
  • ad.datadoghq.com/logapp.logs: '[{"type":"file","path":"/tmp/share/test.log"},{"source":"my_source", "service":"my_service", "type": "<CONTAINER_TYPE>"}]' (where <CONTAINER_TYPE> is the container runtime : cri-o, docker or containerd) will collect log from /tmp/share/test.log and the container log with a custom source/service value if type is set to the correct container runtime.

Note: source/service are set independently for each config.

Describe your test plan

Use the following pod annotation for testing.

apiVersion: v1
kind: Pod
metadata:
  name: logapp
  annotations:
    ad.datadoghq.com/logapp.logs: '[{"type":"file","path":"/tmp/share/test.log"}]'
spec:
  containers:
  - name: logapp
    image: mingrammer/flog:latest
    imagePullPolicy: Always
    command: ["/bin/flog"]
    args: ["-l", "-d", "2"]

Tests:

  • Different container runtime with k8s
  • Plain docker
  • Assess that there are no side effects, especially regarding AD combination like ad.datadoghq.com/<container>.logs: '[{"type":"file","path":"/tmp/share/test.log"},{"source":"my_source", "service":"my_service"}]' leads to properly tagged log from the container and that the file tailer actually starts.

@prognant prognant added the [deprecated] team/agent-core Deprecated. Use metrics-logs / shared-components labels instead.. label Jan 14, 2021
@prognant prognant added this to the 7.26.0 milestone Jan 14, 2021
@prognant prognant requested review from a team as code owners January 14, 2021 13:27
Copy link
Copy Markdown
Contributor

@ogaca-dd ogaca-dd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just having a question.

Comment thread pkg/logs/input/kubernetes/launcher.go Outdated
@prognant prognant merged commit e204649 into master Jan 20, 2021
@prognant prognant deleted the prognant/ac-720-file-tailing-from-k8s-AD branch January 20, 2021 13:39
sergei-deliveroo pushed a commit to deliveroo/datadog-agent that referenced this pull request Mar 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[deprecated] team/agent-core Deprecated. Use metrics-logs / shared-components labels instead.. [deprecated] team/containers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants