feat(opentelemetry sink): introduce otlp encoder

[pront]

Summary

The main purpose of this feature is to simplify OTEL sink configuration from:

otel_sink:
  inputs:
    - otel.logs
  type: opentelemetry
  protocol:
    type: http
    uri: http://localhost:5318/v1/logs
    method: post
  encoding:
     codec: protobuf
     protobuf:
       desc_file: path/to/opentelemetry-proto.desc
       message_type: opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest
    framing:
      method: "bytes"
    request:
      headers:
      content-type: "application/x-protobuf"

to

otel_sink:
  inputs:
    - source0.logs
  type: opentelemetry
  protocol:
    type: http
    uri: http://otel-collector-sink:5318/v1/logs
    method: post
  encoding:
     codec: otlp

I had to implement default for EncodingConfigWithFraming otherwise users would have to set a codec that would always be ignored. Hence the numerous CUE changes.

Vector configuration

How did you test this PR?

E2E tests were adapted.

I will also follow up in another PR to improve the E2E so we can test:

• metrics and traces by using https://github.com/open-telemetry/opentelemetry-collector-contrib/pkgs/container/opentelemetry-collector-contrib%2Ftelemetrygen
• both HTTP and GRPC

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Related:

• feat(opentelemetry source): support OTLP logs, metrics & traces #23524
• Support sending OpenTelemetry metrics #17310

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[pront]

/ci-run-e2e-opentelemetry-logs

[ozanichkovsky]

@pront thank you for doing all this work related to improving open telemetry source/sink in Vector.

I tried out use_otlp_encoding from version 0.50.0 of vector and for some reason it didn't work until I recompiled it with disabling JSON based protobuf decoding. I suppose the reason could be related to .desc file I created by using protoc command manually. Do I create it wrongly?

I also compiled this version and now it just works.

Any suggestions on why I can't achieve direct output from source to sink in 0.50.0?

[thomasqueirozb]

e2e test and docs look really nice now!

[pront]

@pront thank you for doing all this work related to improving open telemetry source/sink in Vector.

I tried out use_otlp_encoding from version 0.50.0 of vector and for some reason it didn't work until I recompiled it with disabling JSON based protobuf decoding. I suppose the reason could be related to .desc file I created by using protoc command manually. Do I create it wrongly?

Hello, @ozanichkovsky, can you create a discussion and provide your config, sample input data, and what changes you made before re-compiling?

I also compiled this version and now it just works.

Good to hear, you can use the nightly version after this PR is merged.

[pront]

/ci-run-component-features

[pront]

e2e test and docs look really nice now!

I want to follow up and add metrics and traces to the E2E. Should be a matter of using an enhanced generator, maybe telemetrygen.

I will take another look at this PR, try locally to also push through metrics and traces and then I will enqueue. If you have more comments by then, just leave them as review comments.

[pront]

/ci-run-component-features