[PAR] trace private action executions by julesmcrt · Pull Request #49526 · DataDog/datadog-agent

julesmcrt · 2026-04-17T13:35:18Z

What does this PR do?

Adds tracing in the Private Action Runner so each task dispatch emits a span on service: private-action-runner. The rshell bundle swaps the context service to rshell before invoking the interpreter, so rshell-emitted child spans land on service: rshell.

Span layout

action.run                          (service: private-action-runner)
  resource: <action FQN>
  tag:      task_id: <uuid>
  └── <rshell spans from DataDog/rshell#186>   (service: rshell)

Changes

WorkflowRunner.RunTask wraps every dispatch in an action.run span. Resource = action FQN, task_id as a tag.
rshell.runCommand sets service: rshell on the context before invoking interp.Runner.Run so rshell's public callbacks (OnCommandDispatch, OnPipelineStart, …) attribute child spans correctly.
PAR component bootstraps a fleettelemetry.Telemetry flusher at startup using api_key + configured site so spans ship to the instrumentation telemetry intake.
New pkg/privateactionrunner/observability/traces.go holds the service and operation name constants.

Motivation

Give PAR and rshell runtime visibility in APM. Enables debugging blocked commands, rshell script performance, and cross-service latency breakdown without log-mining.

Depends on

Tracer base changes: base of this PR (jules.macret/Q/lightweight-tracer-improvements)
rshell callbacks: feat(telemetry): integrate Datadog Installer lightweight tracer rshell#186

Describe how you validated your changes

Unit tests pass (dda inv test --targets=./pkg/privateactionrunner/...)
Deployed to a local Kind cluster; verified end-to-end in org2 (US1 prod) APM — action.run span with service: private-action-runner and rshell child spans with service: rshell

Additional notes

Blocked by the two dependencies above. Merge base first, then this PR.

dd-octo-sts · 2026-04-17T13:46:20Z

Go Package Import Differences

Baseline: d0fef68
Comparison: 914e439

binary	os	arch	change
privateactionrunner	linux	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/fleet/installer/telemetry
privateactionrunner	linux	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/fleet/installer/telemetry
privateactionrunner	windows	amd64	+10, -0 +github.com/DataDog/datadog-agent/pkg/fleet/installer/telemetry +github.com/go-ole/go-ole +github.com/go-ole/go-ole/oleutil +github.com/shirou/gopsutil/v4/common +github.com/shirou/gopsutil/v4/cpu +github.com/shirou/gopsutil/v4/host +github.com/shirou/gopsutil/v4/mem +github.com/shirou/gopsutil/v4/net +github.com/shirou/gopsutil/v4/process +github.com/yusufpapurcu/wmi
privateactionrunner	darwin	amd64	+1, -0 +github.com/DataDog/datadog-agent/pkg/fleet/installer/telemetry
privateactionrunner	darwin	arm64	+1, -0 +github.com/DataDog/datadog-agent/pkg/fleet/installer/telemetry

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eb30791506

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

dd-gplassard

A few nits but looks good, thanks !

Should be merged with #49488 to propagate the trace from the backend (can be done as a followup)

dd-gplassard · 2026-04-17T14:48:21Z

+	// priority and, if it signals reject (<= 0), the backend drops them.
+	// Standard values: -1 = USER_REJECT, 0 = AUTO_REJECT, 1 = AUTO_KEEP,
+	// 2 = USER_KEEP. Omit to use the tracer's default (USER_KEEP).
+	ApmSamplingPriority *int `json:"apmSamplingPriority,omitempty"`


feels like this should be managed at the AP level no ? (we have some metadata attributes on our task)

Good point. I kept it on RunCommandInputs for this PR because we do not yet have a task-level metadata field on the types.Task envelope (only Attributes.Inputs, JobId, Client, etc. — see types/task.go). Happy to follow up once the backend/task envelope can carry arbitrary metadata; at that point the sampling-priority flag becomes generic across all actions. Tracking as a follow-up.

dd-gplassard · 2026-04-17T14:51:23Z

+		span.SetService(rshellSpanService)
+		span.SetResourceName("execution")
+		defer func() { span.Finish(nil) }()
+	}


I think we could wrap action.Run and have it for all actions ?
https://github.com/DataDog/datadog-agent/pull/49526/changes#diff-9448fa3edd4678d5fd48c3f46775321ca7d2448d8660ac1f31c03904fdb19109L116

Agreed that wrapping action.Run at the workflow-runner level would avoid duplicating span boilerplate in each action. I kept the rshell.script wrap inside the rshell handler for now because it serves as the service-boundary span: it's the span that flips the APM service from private-action-runner (the task wrap in WorkflowRunner.RunTask) to rshell, and every child command/pipeline/for/if span inherits that chain. If we wrap at action.Run in the runner, we need the action to declare its desired service name (e.g. via a new Action.Service() string method) so the runner knows what to stamp. Reasonable next step — happy to do it as a follow-up.

dd-gplassard · 2026-04-17T14:52:43Z

+// parSpanService is the Datadog APM service name used for spans that
+// represent private-action-runner work itself (task dispatch, etc.).
+// rshell-specific spans override this to "rshell" via [Span.SetService].
+const parSpanService = "private-action-runner"


can we put this in observability ? 🙏

The constant now lives in the telemetry def package as partelemetry.ParServiceName (and RshellServiceName for the rshell override). I put it there rather than observability because comp/privateactionrunner/telemetry/impl/telemetry.go also needs it and comp/ cannot import pkg/ (depguard). Single source of truth for both sides, which also resolves the "duplicated" comment below.

julesmcrt · 2026-04-17T15:34:44Z

Addressed the feedback from @chatgpt-codex-connector and @dd-gplassard in 8fa1291:

Fixed

Codex P1 (noop span panic): setTag now lazy-initializes Meta/Metrics maps in pkg/fleet/installer/telemetry/span.go, so the zero-value span returned by the noop component no longer panics on SetTag.
var err on deferred span finish: RunTask uses a named return err and span.Finish(err).
Use observability constants: task span tags are now observability.TaskIDTagName / observability.ActionFqnTagName.
Duplicated service constant: extracted to partelemetry.ParServiceName / partelemetry.RshellServiceName in the telemetry def package (one source of truth for both comp/ and pkg/ callers; comp/ can't import pkg/observability due to depguard).
Team tag added to def component.
CI: added the three missing BUILD.bazel files, renamed package to telemetryimpl, regenerated comp/README.md and .github/CODEOWNERS via dda inv components.lint-components, and plumbed the new partelemetry.Component arg through the cluster-agent callsite in cmd/cluster-agent/subcommands/start/command.go (passing nil — the cluster agent does not enable PAR telemetry today).

Follow-ups (replied inline)

Moving ApmSamplingPriority from RunCommandInputs to a task-level metadata field: needs a new Attributes.Metadata on the task envelope first. Happy to do it as a follow-up.
Wrapping action.Run at the workflow-runner level: good idea, but wants an Action.Service() string contract since rshell needs to flip the APM service — follow-up.

lint_linux-x64 may still fail until rshell v0.0.11 ships (pinned version does not yet expose the interp.On* callbacks); everything else should be green.

chatgpt-codex-connector · 2026-04-17T15:36:29Z

Summary

I reviewed the branch state and verified that the feedback listed in the trigger is already present in HEAD (8fa1291), so there were no additional code changes for me to apply.
The no-op span panic fix is in place: setTag now lazy-initializes Meta and Metrics, which makes zero-value spans safe for tag writes. pkg/fleet/installer/telemetry/span.goL144-L154
RunTask now uses a named err return with deferred span.Finish(err), and task tags use observability constants (TaskIDTagName, ActionFqnTagName). pkg/privateactionrunner/runners/workflow_runner.goL93-L105
PAR/rshell service-name constants and team tag are centralized in telemetry def (ParServiceName, RshellServiceName, team: action-platform). comp/privateactionrunner/telemetry/def/component.goL18-L30
The cluster-agent callsite passes nil for PAR telemetry (as described in the trigger follow-up). cmd/cluster-agent/subcommands/start/command.goL748-L767

Testing

✅ git rev-parse --abbrev-ref HEAD && git rev-parse HEAD && git status --short
✅ rg -n "TaskIDTagName|ActionFqnTagName|ParServiceName|RshellServiceName|Finish\\(err\\)|setTag" pkg/privateactionrunner/runners/workflow_runner.go pkg/fleet/installer/telemetry/span.go comp/privateactionrunner/telemetry/def/component.go

No new commit was created (working tree was already clean), and I did not call make_pr because there were no new changes to propose.

View task →

dd-gplassard · 2026-04-17T17:27:56Z

 /comp/notableevents @DataDog/windows-products
 /comp/offlinereporter @DataDog/agent-metric-pipelines
 /comp/privateactionrunner @DataDog/action-platform
+/comp/privateactionrunner/telemetry @DataDog/action-platform


should be implicit from the previous line no ?

dd-gplassard · 2026-04-17T17:32:36Z

+		// Stamp error type only — don't pass err to Finish, which would
+		// record error.message / error.stack / error.details. Action
+		// errors can embed customer data (rshell parse errors quote the
+		// offending script fragment; HTTP action errors can carry URLs
+		// and response bodies). The task_id + error type is enough to
+		// correlate back to the action log for details.
+		defer func() {
+			if err != nil {
+				span.SetTag("error", 1)
+				span.SetTag("error.type", fmt.Sprintf("%T", err))
+			}
+			span.Finish(nil)
+		}()


feels like we could create a errors.New making sure we construct it from "safe" values and keep idiomatic usage

jeremy-hanna

👍 for agent-runtime owned files

dd-octo-sts · 2026-04-18T02:01:26Z

Files inventory check summary

File checks results against ancestor d0fef680:

Results for datadog-agent_7.80.0~devel.git.58.914e439.pipeline.108551498-1_amd64.deb:

No change detected

cit-pr-commenter-54b7da · 2026-04-18T02:23:05Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: b4ddad9e-34a0-46f5-8ed0-b6d37e1b9ed5

Baseline: d0fef68
Comparison: 914e439
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	-3.51	[-6.48, -0.53]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	tcp_syslog_to_blackhole	ingress throughput	+0.80	[+0.61, +0.99]	1	Logs
➖	ddot_metrics	memory utilization	+0.53	[+0.34, +0.72]	1	Logs
➖	quality_gate_idle	memory utilization	+0.09	[+0.04, +0.14]	1	Logs bounds checks dashboard
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.07	[-0.36, +0.51]	1	Logs
➖	docker_containers_memory	memory utilization	+0.05	[-0.03, +0.13]	1	Logs
➖	ddot_logs	memory utilization	+0.02	[-0.05, +0.08]	1	Logs
➖	otlp_ingest_metrics	memory utilization	+0.02	[-0.14, +0.18]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.01	[-0.40, +0.42]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	+0.00	[-0.22, +0.22]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.12, +0.11]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.00	[-0.51, +0.51]	1	Logs
➖	uds_dogstatsd_to_api_v3	ingress throughput	-0.00	[-0.20, +0.20]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.21, +0.19]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.04	[-0.20, +0.11]	1	Logs
➖	otlp_ingest_logs	memory utilization	-0.07	[-0.17, +0.03]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	-0.10	[-0.15, -0.04]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.25	[-0.29, -0.22]	1	Logs bounds checks dashboard
➖	ddot_metrics_sum_delta	memory utilization	-0.61	[-0.79, -0.44]	1	Logs
➖	file_tree	memory utilization	-0.71	[-0.77, -0.65]	1	Logs
➖	ddot_metrics_sum_cumulative	memory utilization	-0.81	[-0.96, -0.66]	1	Logs
➖	quality_gate_metrics_logs	memory utilization	-1.06	[-1.30, -0.82]	1	Logs bounds checks dashboard
➖	quality_gate_logs	% cpu utilization	-2.23	[-3.84, -0.62]	1	Logs bounds checks dashboard
➖	docker_containers_cpu	% cpu utilization	-3.51	[-6.48, -0.53]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	docker_containers_cpu	simple_check_run	10/10	715 ≥ 26
✅	docker_containers_memory	memory_usage	10/10	275.12MiB ≤ 370MiB
✅	docker_containers_memory	simple_check_run	10/10	676 ≥ 26
✅	file_to_blackhole_0ms_latency	memory_usage	10/10	0.19GiB ≤ 1.20GiB
✅	file_to_blackhole_0ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10	0.24GiB ≤ 1.20GiB
✅	file_to_blackhole_1000ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_100ms_latency	memory_usage	10/10	0.20GiB ≤ 1.20GiB
✅	file_to_blackhole_100ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_500ms_latency	memory_usage	10/10	0.22GiB ≤ 1.20GiB
✅	file_to_blackhole_500ms_latency	missed_bytes	10/10	0B = 0B
✅	quality_gate_idle	intake_connections	10/10	4 = 4	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	175.65MiB ≤ 181MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	4 = 4	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	503.46MiB ≤ 550MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	4 ≤ 6	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	208.07MiB ≤ 220MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	344.47 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	4 ≤ 6	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	406.10MiB ≤ 475MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.

dd-gplassard

Nice, thanks ! 👍

Emit an `action.run` span in WorkflowRunner.RunTask on service `private-action-runner`, with the action FQN as the resource name and the task id as a tag. Rshell's runCommand swaps the context service to `rshell` before invoking the interpreter so rshell's child spans inherit the rshell service. Bootstrap a fleet installer lightweight tracer in the PAR component lifecycle using api_key + configured site so spans reach the instrumentation-telemetry-intake. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

### What does this PR do? Improve #49526 by getting the trace / span IDs from the backend This way all spans are linked in the UI of Datadog! Also requires backend PR DataDog/dd-source#412290 ### Motivation ### Describe how you validated your changes ### Additional Notes --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

dd-octo-sts Bot added internal Identify a non-fork PR team/action-platform labels Apr 17, 2026

github-actions Bot added the long review PR is complex, plan time to review it label Apr 17, 2026

dd-octo-sts Bot added team/agent-health team/windows-products labels Apr 17, 2026

julesmcrt mentioned this pull request Apr 17, 2026

feat(telemetry): integrate Datadog Installer lightweight tracer DataDog/rshell#186

Merged

3 tasks

julesmcrt marked this pull request as ready for review April 17, 2026 14:07

julesmcrt requested review from a team as code owners April 17, 2026 14:07

julesmcrt requested a review from ross-m-young April 17, 2026 14:07

chatgpt-codex-connector Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread comp/privateactionrunner/telemetry/impl/telemetry.go Outdated

julesmcrt changed the title ~~feat(par): emit rshell telemetry via agenttelemetry + installer tracer~~ feat(par): emit rshell telemetry via a dedicated tracer component Apr 17, 2026

dd-gplassard approved these changes Apr 17, 2026

View reviewed changes

julesmcrt requested review from a team as code owners April 17, 2026 15:33

julesmcrt requested a review from Enzu83 April 17, 2026 15:33

dd-octo-sts Bot added team/agent-build team/container-platform The Container Platform Team team/agent-runtimes labels Apr 17, 2026

levan-m approved these changes Apr 17, 2026

View reviewed changes

dd-gplassard approved these changes Apr 17, 2026

View reviewed changes

jeremy-hanna approved these changes Apr 17, 2026

View reviewed changes

julesmcrt force-pushed the jules.macret/Q/rshell/telemetry branch from 18ca58b to e514c88 Compare April 18, 2026 01:37

julesmcrt changed the base branch from main to jules.macret/Q/lightweight-tracer-improvements April 18, 2026 01:37

julesmcrt changed the title ~~feat(par): emit rshell telemetry via a dedicated tracer component~~ [PAR] trace private action executions Apr 18, 2026

julesmcrt requested a review from dd-gplassard April 18, 2026 01:40

aiuto approved these changes Apr 18, 2026

View reviewed changes

dd-gplassard reviewed Apr 20, 2026

View reviewed changes

julesmcrt mentioned this pull request Apr 20, 2026

[ACTP] retrieve tracing informations from backend #49598

Merged

julesmcrt force-pushed the jules.macret/Q/rshell/telemetry branch from e514c88 to 914e439 Compare April 20, 2026 13:51

julesmcrt added backport/7.79.x Automatically create a backport PR to the 7.79.x branch once the PR is merged changelog/no-changelog No changelog entry needed labels Apr 20, 2026

julesmcrt merged commit ecf0698 into jules.macret/Q/lightweight-tracer-improvements Apr 20, 2026
238 checks passed

julesmcrt deleted the jules.macret/Q/rshell/telemetry branch April 20, 2026 17:12

Conversation

julesmcrt commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Span layout

Changes

Motivation

Depends on

Describe how you validated your changes

Additional notes

Uh oh!

dd-octo-sts Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Go Package Import Differences

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

dd-gplassard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

julesmcrt commented Apr 17, 2026

Uh oh!

chatgpt-codex-connector Bot commented Apr 17, 2026

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy-hanna left a comment

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files inventory check summary

Results for datadog-agent_7.80.0~devel.git.58.914e439.pipeline.108551498-1_amd64.deb:

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

dd-gplassard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

julesmcrt commented Apr 17, 2026 •

edited

Loading

dd-octo-sts Bot commented Apr 17, 2026 •

edited

Loading

dd-octo-sts Bot commented Apr 18, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Apr 18, 2026 •

edited

Loading