Provide Omnibus installer for the project by masci · Pull Request #22 · DataDog/datadog-agent

masci · 2016-09-14T14:30:03Z

The PR implements an Omnibus installer for the project.

For the time being, to allow developers working on the new agent to have a v.5 installed on their dev boxes, I've changed the install and configuration path.

olivielpeau

Left a few comments.

The main thing would be to replace datadog-agent/dd-agent with datadog-agent6 (which would allow us to have agent 5 and 6 installed simultaneously). At some point we'll want to change this back to datadog-agent

olivielpeau · 2016-09-27T15:08:24Z

+name 'datadog-agent'
+
+source path: '..'
+


let's define here all the dependencies that datadog-agent requires (python for instance, can't think of any other one though)

olivielpeau · 2016-09-27T15:13:03Z

@@ -0,0 +1,11 @@
+name 'datadog-agent'
+
+source path: '..'


I think the path needs to be an absolute path, are you sure this works? (see http://www.rubydoc.info/github/opscode/omnibus/Omnibus/Software#source-instance_method)

olivielpeau · 2016-09-27T15:16:00Z

+# Clean up what we installed
+rm -f /etc/init.d/datadog-agent
+rm -rf /etc/datadog/agent
+rm -rf /opt/$PROJECT_NAME/*


this should be /opt/datadog/agent right?

(or rather /opt/datadog-agent6)

olivielpeau · 2016-09-27T15:33:45Z

+#
+require "./lib/ostools.rb"
+
+name 'datadog-agent'


let's temporarily name the pkg differently (so that we can have both agent 5 and 6 installed on the same machine), datadog-agent6 would make sense to me

olivielpeau · 2016-09-27T15:34:06Z

+name 'datadog-agent'
+maintainer 'Datadog Packages <package@datadoghq.com>'
+homepage 'http://www.datadoghq.com'
+install_dir '/opt/datadog/agent'


-> /opt/datadog-agent6 ?

olivielpeau · 2016-09-27T15:34:25Z

+if linux?
+  # Debian
+  if debian?
+    extra_package_file '/lib/systemd/system/datadog-agent.service'


-> datadog-agent6.service?

(etc)

olivielpeau · 2016-09-27T15:35:16Z

+  end
+
+  # Example configuration files for the agent and the checks
+  extra_package_file '/etc/datadog/agent/datadog.conf.example'


-> /etc/datadog-agent6/datadog.conf.example?

(etc)

olivielpeau

Added another bunch of comments, some of them important, others not so much, let me know what you think :)

olivielpeau · 2016-09-28T16:40:50Z

+
+# .deb specific flags
+package :deb do
+  vendor 'Datadog <info@datadoghq.com>'


unrelated to your changes, but the email should be package@datadoghq.com, could you fix it now? 🙇

(I doubt we'll ever think about fixing it otherwise)

olivielpeau · 2016-09-29T12:50:53Z

@@ -0,0 +1,17 @@
+#!/bin/sh


These package scripts need to live in the folder named after the project (see https://github.com/DataDog/omnibus-ruby/blob/datadog-5.0.0/lib/omnibus/project.rb#L571), so these scripts would need to be moved to datadog-agent6 too

olivielpeau · 2016-09-29T13:00:43Z

+  # Just a dummy file that needs to be in the RPM package list if we don't want it to be removed
+  # during RPM upgrades. (the old files from the RPM file listthat are not in the new RPM file
+  # list will get removed, that's why we need this one here)
+  extra_package_file '/usr/bin/datadog-agent6'


now that I think about it maybe we should just remove all these extra_package_files since we're not actually packaging them (at least for now), but it's just a suggestion, that's up to you really

let's remove them, I prefer to start with the bare minimum to avoid carrying on dead or unused code

olivielpeau · 2016-09-29T13:03:20Z

+# software definitions, but you can clone/fork Chef's to get you started.
+gem 'omnibus-software', git: 'git://github.com/datadog/omnibus-software.git', branch: ENV['OMNIBUS_SOFTWARE_BRANCH']
+
+gem 'ohai', '~> 8.10.0'


you can remove this dep, I added it a while ago to pin ohai but it's not needed anymore

olivielpeau

👏

* Provide Omnibus installer for the project (#22) + gitlab CI * use gitlab cache system instead of putting stuff on /cache

[cpu] Add count for number of logical processors

### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: theo.putegnat <theo.putegnat@datadoghq.com>

Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: theo.putegnat <theo.putegnat@datadoghq.com> (cherry picked from commit 40d1f09) ___ Co-authored-by: Théo Putegnat <theo.putegnat@datadoghq.com>

Backport 40d1f09 from #45437. ___ ### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: axel.vonengel <axel.vonengel@datadoghq.com>

Backport 40d1f09 from #45437. ___ ### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: YoannGh <yoann.ghigoff@datadoghq.com> Co-authored-by: florent.clarret <florent.clarret@datadoghq.com>

### What does this PR do? Skip the SSH session patcher and add a test to illustrate the current issue. In addition, adds the possibility to check specific fields in the json returned for ssh_session events. ### Motivation The retry mechanism could cause the agent to send no more than one event per minute if an SSH session was not properly resolved. Previously, the event was not sent and the agent would wait one minute before sending it with the `unknown` type. However, this `authtype` would never be resolved because the session was initialized before the agent started processing events. As a result, every subsequent SSH event would wait one minute for nothing, causing a significant delay in agent events, potentially blocking all the other events. ### Describe how you validated your changes Added a test that illustrate the issue : `TestSSHUserSessionBlocking` With this change, the ssh_session event is now sent with `authtype` set to `unknown` and directly sent. Error without commenting the patcher : ``` Error: Received unexpected error: All attempts fail: #1: not found #2: not found #3: not found #4: not found #5: not found #6: not found #7: not found #8: not found #9: not found #10: not found #11: not found #12: not found #13: not found #14: not found #15: not found #16: not found #17: not found #18: not found #19: not found #20: not found #21: not found #22: not found #23: not found #24: not found #25: not found #26: not found #27: not found #28: not found #29: not found #30: not found Test: TestSSHUserSessionBlocking/second_ssh_no_auth ``` Co-authored-by: theo.putegnat <theo.putegnat@datadoghq.com>

…ame detection Switch from quick-xml's serde deserialization to xml-rs's iterative streaming parser to address security audit findings: - #21: Stack overflow via deeply nested XML (quick-xml recurses per nesting level) - #22: CPU exhaustion via many attributes (quick-xml serde is O(n²) per attribute) - #13: XXE defense-in-depth (xml-rs has no entity expansion support) The new xml_parser module provides shared helpers with MAX_DEPTH=256 and MAX_ATTRS=256 limits, plus security regression tests. All five XML-parsing files (tomcat, jboss, weblogic, websphere, jee) are converted from serde structs to streaming event loops. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remi Hakim and others added 9 commits September 12, 2016 12:52

WIP

f360fed

WIP

d0edc09

Update omnibus_build.sh

a9fde15

Update omnibus_build.sh

eaddc65

Update omnibus_build.sh

2232570

Update omnibus_build.sh

6247649

Update omnibus_build.sh

3a188b9

sparse fixes to complete a build

8815fa6

changed install path to /opt/datadog/agent to allow v5 coexistence

7d08022

masci force-pushed the massi/build branch from d6d6b88 to 2683169 Compare September 14, 2016 15:59

minor adjustments

74c2547

masci force-pushed the massi/build branch from 2683169 to 74c2547 Compare September 15, 2016 09:26

masci changed the title ~~[WIP] Provide Omnibus installer for the project~~ Provide Omnibus installer for the project Sep 15, 2016

masci added the dev/tooling label Sep 15, 2016

olivielpeau requested changes Sep 27, 2016

View reviewed changes

Massimiliano Pippi added 2 commits September 27, 2016 19:11

use datadog-agent6 as the project name

a8d00dd

s/datadog-agent/datadog-agent6

4ed203a

olivielpeau requested changes Sep 29, 2016

View reviewed changes

Massimiliano Pippi added 2 commits September 29, 2016 15:32

fixed according to PR comments

3e459f9

use https instead unsecure git://

1b33160

olivielpeau approved these changes Sep 30, 2016

View reviewed changes

execute bundle install in the entrypoint

382f8fa

masci merged commit 4240d00 into master Sep 30, 2016

masci deleted the massi/build branch September 30, 2016 14:21

masci added a commit that referenced this pull request Oct 6, 2016

Provide Omnibus installer for the project (#22) + gitlab CI

8600540

masci added a commit that referenced this pull request Oct 6, 2016

[tooling] Gitlab integration (#26)

db98f2b

* Provide Omnibus installer for the project (#22) + gitlab CI * use gitlab cache system instead of putting stuff on /cache

pgimalac pushed a commit that referenced this pull request Apr 5, 2023

Merge pull request #22 from DataDog/zeller/windows-core-count

e296cdd

[cpu] Add count for number of logical processors

pgimalac pushed a commit that referenced this pull request Apr 24, 2023

Merge pull request #22 from DataDog/zeller/windows-core-count

1b8cd44

[cpu] Add count for number of logical processors

karlhenselin mentioned this pull request Feb 7, 2025

[BUG] Deadlock / BLOCKED threads starting tomcat with dd-agent #33821

Closed

vitkyrka mentioned this pull request Apr 13, 2026

fix(discovery): replace quick-xml serde with xml-rs streaming parser for JEE service name detection #49292

Merged

Conversation

masci commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

olivielpeau left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivielpeau Sep 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivielpeau Sep 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivielpeau left a comment

Choose a reason for hiding this comment

Uh oh!

olivielpeau Sep 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivielpeau Sep 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivielpeau left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

masci commented Sep 14, 2016 •

edited

Loading

olivielpeau Sep 27, 2016 •

edited

Loading

olivielpeau Sep 27, 2016 •

edited

Loading

olivielpeau Sep 28, 2016 •

edited

Loading

olivielpeau Sep 29, 2016 •

edited

Loading