ARROW-15061: [C++] Add logging for kernel functions and exec plan nodes

[mbrobbel]

Adds spans and events for exec plan and exec nodes.

I use the following setup to debug traces:

docker-compose.yml:

version: "2"
services:

  jaeger-all-in-one:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "14268"
      - "14250"

  otel-collector:
    image: otel/opentelemetry-collector-contrib-dev:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./output:/var/output
    ports:
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "4317"        # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
    depends_on:      
      - jaeger-all-in-one

  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: "0.0.0.0:4318"
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  parquet:
    path: /var/output/log.parquet

service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - logging
        - jaeger
        - parquet
    metrics:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - logging
        - prometheus
        - parquet

prometheus.yaml:

scrape_configs:
  - job_name: "otel-collector"
    scrape_interval: 10s
    static_configs:
      - targets: ["otel-collector:8889"]
      - targets: ["otel-collector:8888"]

Start the services and then use the instructions from #11906.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-15061

[lidavidm]

Cool! The macro definitions look fairly useful.

[lidavidm]

If you have a screenshot or any quick example of the output to share here it would also be useful, I think.

[mbrobbel]

Some example output in Jaeger:
TPC-H query 1:

TPC-H query 2:

[lidavidm]

Thanks, this looks good overall. I left some small comments.

[lidavidm]

Thanks for doing this. Once we get this in, I'm pretty excited to see what sorts of analysis/visualization we can do.

[lidavidm]

This macro really makes me think we should consider just adding Span as a part of Future (and see how it impacts performance). That can be done later, though, I think we can get this in first and continue refining how we use OpenTelemetry.

[westonpace]

@lidavidm @mbrobbel Is there any reason we can't merge this? If you want I can take a look through tomorrow and merge. Otherwise I took a quick scan and I think this is helpful information to get us started with instrumentation.

[mbrobbel]

@lidavidm @mbrobbel Is there any reason we can't merge this? If you want I can take a look through tomorrow and merge. Otherwise I took a quick scan and I think this is helpful information to get us started with instrumentation.

There are still some comments from @lidavidm that I need to address, but I can do that tomorrow.

[lidavidm]

No further comments from me - let's merge once OrderBySinkNode is fully instrumented.

[lidavidm]

I want to base #11964 on top of this since the helpers here will simplify it.

[mbrobbel]

@westonpace @lidavidm this is now ready for a final review.

[lidavidm]

Thanks for working through this!

[lidavidm]

(For the record, edited description to remove username pings; those get put into the commit message and then generate a lot of notification spam.)

[ursabot]

Benchmark runs are scheduled for baseline = 74deb45 and contender = 9b53235. 9b53235 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.36%] ursa-i9-9960x
[Finished ⬇️0.69% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java