Final pass at checks loading by masci · Pull Request #10 · DataDog/datadog-agent

masci · 2016-07-06T12:53:38Z

This PR finalises the check loading cycle: now also checks written in Go are dynamically loaded and configured starting from a configuration file.

The strategy adopted to map check names like memory to a Go struct is the same adopted by the img package in the Go standard lib.

While at the job: better naming, better package organization, removed dead code

olivielpeau · 2016-07-07T10:25:26Z

👍

Load macros from configuration file

Create functionality for Hashicorp Vault

Key optimizations: - Fix #1: Pass Arc<HashSet> directly to avoid cloning container sets in predicates - Fix #4: Extract mtime once per file before sorting (avoids O(n log n) syscalls) - Fix #9: Add refresh staleness check to skip redundant file discovery - Fix #10: Wrap stats cache in Arc to avoid deep clones on cache hit - Fix #11: Use sort_unstable for timeseries points (faster, no stability needed) - Fix #13: Single division instead of two in rate calculation - Pre-allocate RawContainerData vectors with estimated capacity - Remove metric_name from projection (only needed for predicate, not output) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

# This is the 1st commit message: Keep our own copy of cacert.pem - Replace omnibus fetch from upstream with that static copy. - Include text in the BUILD file about how we check for new upstream versions. - Add explanation of why we have this. https://datadoghq.atlassian.net/browse/ABLD-169 # This is the commit message #2: just use copy for windows # This is the commit message #3: qmarks # This is the commit message #4: omnibus is to blame # This is the commit message #5: maybe # This is the commit message #6: add back in default version # This is the commit message #7: drop livestream on debug # This is the commit message #8: You're kidding, :live_stream? # This is the commit message #9: srsly # This is the commit message #10: just copy on windows # This is the commit message #11: cwd with copy probably does not work # This is the commit message #12: just give up on pkg_install for certs # This is the commit message #13: drop unneded pkg_install targets # This is the commit message #14: - use cwd to make it a little cleaner - update cert to 2025-09-09 # This is the commit message #15: comma # This is the commit message #16: Revert use of cwd on copy. It doesn't matter if it is ugly or not. We are going to delete it this quarter anyway.

### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: theo.putegnat <theo.putegnat@datadoghq.com>

Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: theo.putegnat <theo.putegnat@datadoghq.com> (cherry picked from commit 40d1f09) ___ Co-authored-by: Théo Putegnat <theo.putegnat@datadoghq.com>

Backport 40d1f09 from #45437. ___ ### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: axel.vonengel <axel.vonengel@datadoghq.com>

Backport 40d1f09 from #45437. ___ ### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: YoannGh <yoann.ghigoff@datadoghq.com> Co-authored-by: florent.clarret <florent.clarret@datadoghq.com>

### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: theo.putegnat <theo.putegnat@datadoghq.com>

Summary of Changes HIGH Priority Issues Fixed: #1: Write lock held across network I/O (impl/delegatedauth.go:270) - Refactored refreshAndGetAPIKey to release the lock before making network calls (authenticate) - The lock is now only held briefly to check/update state, not during network I/O #2: Context not propagated to signer.SignHTTP (aws.go:195) - Updated generateAwsAuthData to accept a context parameter - Changed signer.SignHTTP(context.Background(), ...) to signer.SignHTTP(ctx, ...) #3: Context not propagated to getCredentials IMDS call (aws.go:119) - Updated getCredentials to accept a context parameter - Removed ctx := context.Background() and now uses the passed context for IMDS calls MEDIUM Priority Issues Fixed: #4: No response body size limit (api/delegated_auth.go:97) - Added maxResponseBodySize = 1 * 1024 * 1024 constant (1 MB) - Wrapped response body with io.LimitReader to prevent memory exhaustion #5: No overall HTTP client timeout (api/delegated_auth.go:82) - Added httpClientTimeout = 30 * time.Second constant - Added Timeout: httpClientTimeout to the HTTP client #6: config.Set called while holding write lock (impl/delegatedauth.go:341) - Moved updateConfigWithAPIKey call outside the lock in startBackgroundRefresh - Captured the API key while holding the lock, then released it before calling config.Set #7: Blocking IMDS calls while holding write lock (impl/delegatedauth.go:127) - Refactored initializeIfNeeded to perform cloud detection without holding locks - IMDS calls now happen outside any lock, then state is updated with a brief write lock #8: Regex fails silently for non-standard formats (api/delegated_auth.go:36) - Added debug log when endpoint doesn't match known Datadog domain pattern - Updated function documentation to clarify behavior #9: Uncached IMDS credential fetch (aws.go:104) - Added documentation explaining the trade-off (refresh interval is typically 60 minutes, so caching is not critical) #10: Auth proof format undocumented (aws.go:98) - Added detailed comment documenting the auth proof format: <base64-body>|<base64-headers>|<method>|<base64-url> LOW Priority Issues Fixed: #11: Unnecessarily exported types (aws.go) - Changed SigningData to signingData (unexported) - Changed AWSAuth.AwsRegion to AWSAuth.region (unexported) - Updated all references in aws.go and aws_test.go #12: Tests exercise copy of goroutine (impl/delegatedauth_test.go:19) - Added documentation explaining why tests use a simplified goroutine pattern - Clarified that integration tests cover the actual startBackgroundRefresh function #13: Subsequent Config param silently ignored (def/delegatedauth.go:24) - Updated documentation to clearly state that only the first Config is used - Added warning log when a different Config is passed on subsequent calls

### What does this PR do? Removes several dead or redundant exclusion rules from `.golangci.yml`: **Duplicate rule (cosmetic):** - Removed a duplicate `unnecessary conversion` exclusion that appeared twice. **Dead `unused` suppressions (task #10):** - `` `eventContext` is unused `` — symbol no longer exists in the codebase. - `` `(*DatadogLogger).changeLogLevel` is unused `` — `changeLogLevel` moved to `*loggerPointer` during a refactor; the original type no longer has this method. - `` `defaultRetryDuration` is unused `` — symbol no longer exists. - `` `defaultRetries` is unused `` — symbol is used; the linter wouldn't fire on it regardless. **Dead errcheck exclusion:** - `pem.Encode` — the file that motivated this (`test/integration/utils/certificates.go`) was deleted in #39772. All remaining live `pem.Encode` call sites already check or explicitly ignore the error. **Dead revive exclusion:** - `exported const APIName` — the constant no longer exists in `pkg/serverless/trace/inferredspan/`; the file is also already globally excluded via the `paths` list. **Stale comment fix:** - Updated the comment on the `io.WriteString` errcheck exclusion — its originating file (`tools/dep_tree_resolver/go_deps.go`) was deleted in #22067. The rule itself is kept as it still covers unchecked writes to `http.ResponseWriter`. ### Motivation Dead exclusions accumulate noise in the config and make it harder to reason about which rules are actively suppressing real violations. ### Describe how you validated your changes Ran `dda inv linter.go` — 0 issues. ### Additional Notes Part of a broader linter configuration cleanup pass. Co-authored-by: pierre.gimalac <pierre.gimalac@datadoghq.com>

Massimiliano Pippi added 4 commits July 6, 2016 14:46

draft

eb976cf

load go checks

10cf35d

moar tests

9ae3ee1

removed unused field

a8b10c3

masci added the component/collector label Jul 6, 2016

olivielpeau self-assigned this Jul 6, 2016

masci merged commit cdbb21c into master Jul 7, 2016

masci deleted the massi/checks_loading_final branch July 7, 2016 11:58

safchain added a commit to safchain/datadog-agent that referenced this pull request May 20, 2020

Merge pull request DataDog#10 from lebauce/load-macros

8f0e902

Load macros from configuration file

karlhenselin mentioned this pull request Feb 7, 2025

[BUG] Deadlock / BLOCKED threads starting tomcat with dd-agent #33821

Closed

s-alad pushed a commit that referenced this pull request Nov 21, 2025

Merge pull request #10 from rapdev-io/feature/hashicorp-vault

2ca5165

Create functionality for Hashicorp Vault

pgimalac mentioned this pull request Apr 27, 2026

linter: remove dead exclusion rules from .golangci.yml #49907

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Final pass at checks loading#10

Final pass at checks loading#10
masci merged 4 commits intomasterfrom
massi/checks_loading_final

masci commented Jul 6, 2016

Uh oh!

olivielpeau commented Jul 7, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

masci commented Jul 6, 2016

Uh oh!

olivielpeau commented Jul 7, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants