Support OTEL data semantics

[pavolloffay]

OpenTelemetry defines different data semantics than Hypertrace. For instance OTEL uses db.system={redis,cassandra} to define database type and HT uses tags starting with database name redis. to define the database type. There are probably more inconsistencies than just databases.

https://github.com/open-telemetry/opentelemetry-specification/blob/master/semantic_conventions/trace/database.yaml#L8
https://github.com/hypertrace/span-normalizer/blob/main/raw-span-constants/src/main/proto/org/hypertrace/core/span/constants/v1/span_attribute.proto#L125

Here is an example of redis database span from OTEL agent and legacy java agent:

OTEL agent:

legacy agent:

This resulted in backends not being recognized by the platform

OTEL agent:

legacy agent:

There are probably more inconsistencies that the backends/databases.

[pavolloffay]

cc) @buchi-busireddy @kotharironak @davexroth @mohit-a21

[jcchavezs]

So to be clear, is your suggestion that we translate the tags from e.g. redis.command + redis.args into db.system + db.statement while I think this makes sense to some degree I also believe it will be an endless task as different libraries can choose different semantics. In the other hand, if we decide to do such a thing (which I am not 100% sure we do already or not) I would say maybe it is time to think of how we differentiate from actual tags and synthetic tags (like those we create) because I would have to present a user a set of tags with repeated values but different semantics. Ideally we should keep data immutable and anything we do additionally to be explicit that it is additionally added.

[pavolloffay]

So to be clear, is your suggestion that we translate the tags from e.g. redis.command + redis.args into db.system + db.statement

My suggestion is that the platform should understand the OpenTelemetry tags to properly derive entities such as backends/databases. Not only entities, but there might also be more inconsistencies. How it is implemented I will leave it to the maintainers.

[kotharironak]

I can see three possibilities here,

1. transform new backend constants db.system + db.statement -> existing constants (span_attribute.proto) we have, in this example redis.command, redis.connection, etc. This is our existing approach and will need the support of adding new constants per different backend supported.

2. adds new constants for db.system + db.statement, and converts existing constants to db.system + db.statetement, add supports for deriving entities from newly added constants. (suggested by @jcchavezs). This will fix the required number of constants for backend derivatives.

3. simply add new constants of db.system + db.statement in addition to our existing backend system constants (span_attribute.proto), and add entity resolver for these newly added constants.

@jcchavezs @pavolloffay @buchi-busireddy @rish691 Could you vote on the approach to take?

[pavolloffay]

To be clear I have used the database/backend as an example here. The processing pipeline should understand all OpenTelemetry semantic attributes from https://github.com/open-telemetry/opentelemetry-specification/tree/master/semantic_conventions/trace.

[davexroth]

Yes, this is something we've talked about in the past, and following the otel semantic conventions is something we should absolutely do.

We've already gone down this path a little with rpc.system and enduser.id attributes, and we should continue to make these the standard attributes on the platform.

We'll obviously need to keep supporting the older attributes we were using previously for backwards compatibility.

[mohit-a21]

Since we plan to depend on named fields in the data model for processing in downstream services, we should start adding named field for the backends too. We have a SQL specific field defined here: Sql.avdl
We added support for rpc.* tags by adding rpc.avdl and then translating grpc specific tags to Grpc.avdl.
We should be able to do the same for backends/databases where we define a Db.avdl which has generic DB related info(system, statement & connection_string) and then DB specific event fields(Sql, Mongo etc).

This way we would not spill these tags to downstream services.