Skip to content

Refactor experiment signals#2944

Open
khewonc wants to merge 4 commits intomainfrom
khewonc/refactor-experiment-signals
Open

Refactor experiment signals#2944
khewonc wants to merge 4 commits intomainfrom
khewonc/refactor-experiment-signals

Conversation

@khewonc
Copy link
Copy Markdown
Collaborator

@khewonc khewonc commented Apr 27, 2026

What does this PR do?

Refactor experiment signals for more robustness. Instead of having both daemon and controller write to DDA status, have only controller write to DDA status and daemon modify DDA annotations

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

  • Agent: vX.Y.Z
  • Cluster Agent: vX.Y.Z

Describe your test plan

Write there any instructions and details you may have to test your PR.

Checklist

  • PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
  • PR has a milestone or the qa/skip-qa label
  • All commits are signed (see: signing commits)

@khewonc khewonc added this to the v1.27.0 milestone Apr 27, 2026
@khewonc khewonc marked this pull request as ready for review April 27, 2026 20:50
@khewonc khewonc requested a review from a team April 27, 2026 20:50
@khewonc khewonc requested review from a team as code owners April 27, 2026 20:50
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56971366d8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread internal/controller/datadogagent/experiment.go Outdated
Comment thread pkg/fleet/daemon.go
Comment thread pkg/fleet/experiment.go
@khewonc khewonc force-pushed the khewonc/refactor-experiment-signals branch from 5697136 to 4dac8a7 Compare April 27, 2026 21:00
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 27, 2026

Codecov Report

❌ Patch coverage is 74.01961% with 106 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.73%. Comparing base (78490c3) to head (656fd85).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
internal/controller/datadogagent/experiment.go 72.18% 30 Missing and 12 partials ⚠️
pkg/fleet/daemon.go 73.22% 25 Missing and 9 partials ⚠️
pkg/fleet/experiment.go 75.43% 25 Missing and 3 partials ⚠️
cmd/main.go 0.00% 1 Missing ⚠️
pkg/remoteconfig/updater.go 92.30% 1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (74.01%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2944      +/-   ##
==========================================
+ Coverage   40.91%   41.73%   +0.82%     
==========================================
  Files         324      327       +3     
  Lines       28743    29951    +1208     
==========================================
+ Hits        11760    12500     +740     
- Misses      16129    16570     +441     
- Partials      854      881      +27     
Flag Coverage Δ
unittests 41.73% <74.01%> (+0.82%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
internal/controller/datadogagent/revision.go 78.12% <100.00%> (ø)
pkg/fleet/remote_config.go 100.00% <ø> (ø)
cmd/main.go 6.64% <0.00%> (ø)
pkg/remoteconfig/updater.go 4.02% <92.30%> (+4.02%) ⬆️
pkg/fleet/experiment.go 77.77% <75.43%> (-5.90%) ⬇️
pkg/fleet/daemon.go 73.75% <73.22%> (+8.29%) ⬆️
internal/controller/datadogagent/experiment.go 77.60% <72.18%> (-7.56%) ⬇️

... and 12 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78490c3...656fd85. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@datadog-official
Copy link
Copy Markdown

datadog-official Bot commented Apr 27, 2026

Code Coverage

Fix all issues with BitsAI

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

Patch coverage: 73.60% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
Patch Coverage: 73.60%
Overall Coverage: 41.83% (+0.80%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 656fd85 | Docs | Datadog PR Page | Give us feedback!

@khewonc
Copy link
Copy Markdown
Collaborator Author

khewonc commented Apr 28, 2026

@codex review

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f0e5eb7b83

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread api/datadoghq/v2alpha1/datadogagent_types.go
Comment thread pkg/fleet/experiment.go

// Process annotation-based signals first — they take priority over
// automatic timeout since they represent explicit human/RC intent.
pendingClearID, err := r.processExperimentSignal(ctx, instance, newStatus, revList)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Clear no-op experiment signals before returning

manageExperiment now processes annotation signals first, but when status.experiment is nil the function returns immediately and never reaches the annotation-clearing block. That leaves no-op signals (for example rollback/promote when no experiment is active) stuck on the resource, causing repeated reprocessing and making daemon phase waits fail to converge. Ensure processed signals are cleared even when there is no current experiment status.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants