add send_host_metadata option to disable host metadata collection

[xvello]

What does this PR do?

Allow to disable host metadata collection with the send_host_metadata option. This is needed if dsd6 is running alongside agent5, and might be needed for agent6 in some edge cases (one agent/pod in openshift, dedicated agent for event collection...), although these cases might be tackled differently.

Please discuss whether the common option for agent is legit or whether that should be a dsd-only option.

Docker image datadog/dogstatsd:beta is built with this branch to enable for beta testing.

[masci]

Looks good to me, left a couple of nits otherwise 👍 .
Can we add a log INFO or maybe WARNING when the metadata collection is turned off for the agent? For dsd is less important but we can do the same.

[masci]

nit: can we call this metaScheduler or something? var collector scheduler is confusing...

[masci]

Instead of removing this, can we say something like "the host metadata collector cannot be configured and is enabled by default unless all the metadata collection is turned off by setting send_host_metadata to false". (Just an example, feel free to rephrase).

[xvello]

Good point. One thing I'm not clear about is whether this option should disable the whole metaScheduler or only the 'host' collector.

In dsd it disables the whole thing as 'host' is the only enabled collector, but what about other collectors for the agent? Should they be forcefully disabled, even if listed in metadata_collectors?

[masci]

Good question. With the old agent there was no way to distinguish metadata collection (the "host" metadata actually contained a lot of stuff), so the option completely turned off the feature. If we want to do the same, the scheduler shouldn't start. At the moment I don't see any issue with this, since we don't have additional collectors yet, and we might reconsider in the (near) future.

[olivielpeau]

Just a quick comment on the feature itself: we need to make it extra-clear that turning host metadata collection off should not be done unless there is another agent running with host metadata collection on, on the same host.

Turning host metadata collection off on a "regular" installation can lead to a lot of trouble:

• duplicate hosts on DD
• double-billing (hosts are counted twice and billed twice)

We've had cases (in the past, on agent5) of users turning it off and having very bad surprises afterwards, so let's be careful :)

So, more specifically here, I think we should log a WARNING when it's turned off on the agent, and say everywhere that turning it off on an agent can lead to double-billing.

[olivielpeau]

could you make it clear here that host metadata is enabled by default, and that disabling it is needed when running alongside an existing datadog-agent? (currently it's not as clear IMO, but let me know what you think :) )

[xvello]

Added the warning and updated the example conf with the consequences that might have.
For consistency, the option disables the whole metadataScheduler for the agent too.
Does that look good?

[olivielpeau]

a small comment on the description in the example config file, but other than that LGTM! 👍

[olivielpeau]

the last sentence sounds a bit weird IMO (In that case, only one agent should then)

[masci]

I think enable_metadata_collection is the right name provided that the option prevents the scheduler to start altogether.

[masci]

Should be Datadog.BindEnv("enable_metadata_collection") I think

[masci]

do we need the extra \n?