Dual shipping v2 + Refactor and optimize HTTP compression

[gh123man]

What does this PR do?

• Allow any number of reliable additional endpoints (previously we only supported 2)
• Http gzip compression happens once for all destinations (big performance improvement when using multiple endpoints)

Measured improvements:

• ~10% maximum throughput (bytes/sec) improvement for HTTP endpoints (regardless of configuration)
• Any additional HTTP endpoints have very little CPU overhead since compression is now done once for all destinations

Summary of changes

To make review easier since this is a large PR, here are the highlights/major areas to look at:

Overview

The pipeline has rearranged:

OLD

Collector -> Decoder -> Processor -> Sender -> Auditor

NEW

Collector -> Decoder -> Processor -> Strategy -> Sender -> Destination -> Auditor 

Here is an informal diagram of the data flow before and after moving things around:

The sender used to contain the strategy and the destinations. It managed the actual process of encoding and sending logs which was mixed between those 3 pieces. Now they are broken apart with the following responsibilities:

• Strategy (batch/stream) deals with converting Messages into Payloads as well as encoding the payload (gzip for HTTP)
• Sender deals only with taking payloads from the strategy and sending them to one or more destinations. This is where dual shipping and additional endpoints are managed.
• Destinations now have their own read loop and write to the auditor input channel directly (instead of being brokered through the sender).

Notable changes

• The concept of Main vs additional endpoints is largely removed. There are now only Endpoints that are either is_reliable or not. Reliable endpoints are all treated the same and can block the pipeline.
• gzip encoding has moved from the HTTP destination into the batch strategy. This is an optimization so that we only have to zip payloads once when we have multiple endpoints or are using concurrency.
• Concurrent sending of HTTP payloads has been isolated into the HTTP destination. It previously was duplicated in batch_strategy and the http destination to handle async vs non-async sending. (this was required first in order to move gzip compression)
• Retry logic for both tcp and HTTP have now been isolated in the destination implementations (previously was handled in the sender via errors from the destinations)
• SendAsync is removed from the destination as now destinations can have retrying toggled. Removes a lot of duplicate code and now all endpoints follow the same codepath.

Config changes

• Additional endpoints will now use the same compression settings as the main endpoint. This could be a breaking change for users who relied on custom compression settings (though unlikely).
• Additional endpoints will now use the same retry/backoff settings as the main endpoint. Retries were not previously supported so this is not breaking.

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test your changes

Non Agent-core teams

• please do basic sanity tests to ensure events sent through the pipeline are working as expected

Agent core QA

Local QA

1. start 2 mock intakes:

docker run --rm -it --name mockhttp1 -p 9998:9998 --rm datadog/agent-dev:bench-logs-intake-dev -b :9998 -d
docker run --rm -it --name mockhttp2 -p 9999:9999 --rm datadog/agent-dev:bench-logs-intake-dev -b :9999 -d

the -d flag puts them in debug mode so they will stream logs to stdout

2. start an agent:

docker run -d --name dd-agent \
    --net=host \
    -e DD_API_KEY="<API_KEY>"\
    -e DD_LOGS_ENABLED=true \
    -e DD_LOGS_CONFIG_LOGS_NO_SSL=true \
    -e DD_LOGS_CONFIG_USE_HTTP=true \
    -e DD_LOGS_CONFIG_LOGS_DD_URL="localhost:9998" \
    -e DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"<API_KEY>\", \"Host\": \"localhost\", \"Port\": 9999, \"is_reliable\": true}]" \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
   <RC_IMAGE>

3. get some logs flowing through the agent. Warning if you use DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL you will create a feedback loop from the mock intake containers. I'd recommend spawning a dedicated logging container.

4. Watch the logs flow in both intakes.

5. Kill one of the intakes

• make sure logs are still flowing to the other one

6. Kill the second intake (both are now dead)

• check agent status to see that # of bytes read is not increasing (pipeline should be blocked)

7. Restart one of the intakes - logs should start flowing
8. restart the other intake - logs should start flowing.

App QA

Now instead of using mock intakes - use two real Datadog intakes. You will need 2 API keys each from different orgs.

docker run -d --name dd-agent \
    --net=host \
    -e DD_API_KEY="<API_KEY>"\
    -e DD_LOGS_ENABLED=true \
    -e DD_LOGS_CONFIG_USE_HTTP=true \
    -e DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"<ANOTHER_API_KEY>\", \"Host\": \"agent-intake.logs.datadoghq.com\", \"Port\": 10516, \"is_reliable\": true}]" \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
   <RC_IMAGE>

Stream some logs and watch the livetail in both orgs for the logs.

Further testing

For extra validation you can:

• run more than 2 destinations
• mix reliable and unreliable destinations
• make a destination that produces random failures

Reviewer's Checklist

• If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
• The appropriate team/.. label has been applied, if known.
• A release note has been added or the changelog/no-changelog label has been applied.
• Changed code has automated tests for its functionality.
• Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
• If applicable, docs team has been notified or an issue has been opened on the documentation repo.
• If applicable, the need-change/operator and need-change/helm labels have been applied.
• If applicable, the config template has been updated.

[ruthnaebeck]

Docs review

[djmitche]

Thanks for the great PR description and diagrams -- they really helped me to understand this from a high level. Let's make sure that information is captured in the codebase itself, for everyone who reads this code after it's landed!

I have a bunch of notes below, but I think the big "request changes" bit is the races in the sender.

[sgnn7]

@gh123man Great improvements all around! Left a number of comments but they're either nits or questions in general.

[ruthnaebeck]

Docs review

[djmitche]

Nice fixes to the sender!

[ruthnaebeck]

👍 for docs

[sgnn7]

@gh123man Just a few nits but LGTM!