pkg/trace/{api,agent,pb}: introduce new client payload formats: TracerPayload & TraceChunk

[Barbayar]

What does this PR do?

1. Updates the trace payload from agent to backend

Before:

message TracePayload {
        string hostName = 1;
        string env = 2;
        repeated APITrace traces = 3;
        repeated Span transactions = 4;
}

After:

message TracePayload {
        string hostName = 1;
        string env = 2;
        repeated Span transactions = 4;
        repeated TracerPayload tracerPayloads = 5;
        map<string, string> tags = 6;
        string agentVersion = 7;
}

On backend, we would like to receive the payloads from the tracer as they are.
We also updated the payload from tracer to agent.

Before:

[][]*Span

After:

message TraceChunk {
	repeated Span spans = 1;
	map<string, string> tags = 2;
	int32 priority = 3;
	string origin = 4;
}

message TracerPayload {
	repeated TraceChunk chunks = 1;
	map<string, string> tags = 2;
	string service = 3;
	int64 traceCount = 4;
	string containerID = 5;
	string lang = 6;
	string langVersion = 7;
	string langInterpreter = 8;
	string langInterpreterVendor = 9;
	string tracerVersion = 10;
	bool computedTopLevel = 11;
	bool computedStats = 12;
	int32 sequence = 13;
	string runtimeID = 14;
}

2. Adds a new endpoint to receive the payload in new format from tracers

• /v0.6/traces

3. [serverless] Sets global tags at agent payload level instead of each span

• we can propagate the global tags on backend
• it helps the customers to reduce their traffic to us

4. [container] Sets container tags at tracer payload level instead of each top level span

• we can propagate container tags on backend
• it helps the customers to reduce their traffic to us

5. Removes redundant data

• Removes _dd.origin tag from each spans as it's set at trace chunk level
• Removes _sampling_priority_v1 tag from each spans as it's set at trace chunk level

6. Removed Transactions from the payload from agent to backend

Spans in Transactions are a subset of spans in TracerPayloads. Instead of duplicating them in Transactions, we decided to keep them in TracerPayload, and just mark them with _dd.analyzed.

Motivation

We would like receive the following information from the tracer to enrich our features.

• agent tags
• tracer tags
• trace tags

To do that, we need to receive two level of structured data from tracer instead of [][]*Span.

Additional Notes

This PR doesn't contain any logical change. But, there will be 2 more PRs after this one gets merged.

• A new endpoint in trace-agent that receives structured payload from traces
• A new trace payload format we are going to use in future intake

Describe how to test your changes

As this PR doesn't introduce a new feature,

• making sure it didn't break anything
• making sure the performance wasn't degraded

in the usual environment should enough for the test.

RFC

Checklist

• A release note has been added or the changelog/no-changelog label has been applied.
• The need-change/operator and need-change/helm labels has been applied if applicable.
• The appropriate team/.. label has been applied, if known.
• If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
• The config template has been updated if applicable.

Note: Adding GitHub labels is only possible for contributors with write access.

[bits-bot]

All committers have signed the CLA.

[gbbr]

This is ok by me. Just a small curiosity around the conversion code.

[gbbr]

I've skimmed through the PR and left some comments. I'm not done, I'm yet to review the writer and some other bits, but I'm submitting them so you can get a head start.

[j-rufino]

Hi,

As discussed offline, this PR changes are coupled to (and blocked by) related changes

[gbbr]

I think we should call this AgentPayload, like in the RFC. WDYT?