Auto-instrumentation honouring the `OTEL_PYTHON_X_PROVIDER` env-vars

[TBBle]

Is your feature request related to a problem?

I am auto-instrumenting a process in a container in kubernetes, and have this in my Dockerfile:

ENTRYPOINT [ "poetry", "run", "opentelemetry-instrument", "uvicorn", "platform_service.app:create_app", "--factory" ]

Sometimes I want to disable the instrumentation, and rather than override the ENTRYPOINT or CMD of the container from the Deployment (which requires copying the other parameters out into the kubernetes resource without opentelemetry-instrument) or adding additional complexity into the Dockerfile, I'd like an env-var to disable the auto-instrumentation.

Describe the solution you'd like

As it happens, there are appropriate env-vars: OTEL_PYTHON_TRACER_PROVIDER, OTEL_PYTHON_METER_PROVIDER, and OTEL_PYTHON_LOG_EMITTER_PROVIDER, which I would happily set to default, to use the NoOpXProvider entry-points from opentelemetry-api, rather than the default behaviour (equivalent to sdk from the opentelemetry-sdk entrypoints.

However, there's two issues there:

• There's no NoOpLogEmitterProvider or even an ABC for LogEmitterProvider. This would need to be added to the opentelemetry-sdk package, since it's not part of the OpenTelemetry API.
• _OTelSDKConfigurator._configure calls _initialize_components, and the functions the latter calls do not honour the *_PROVIDER env-vars as they call set_X_provider directly. The env-vars are processed by get_X_provider if set_X_provider has not already been called.

The latter point is interesting as, because of the way opentelemetry-instrument works, it offers those env-vars as command-line options. (Well, not the logger, right now, as the variable starts with _, as did the metrics env-var before 1.12).

Describe alternatives you've considered

• Copying the command-line to run without OpenTelemetry into the kubernetes Deployment.
• Generating a separate container image with stripped command-line, so that only the default ProxyXProviders are loaded. (I can then use the env-vars to specify default as described above.
• Adding my own BaseDistro implementation that implements support for OTEL_PYTHON_X_PROVIDER, and using that instead.
  • Technically, what I would be adding is an _OTelSDKConfigurator subclass, as that's the thing that triggers _initialize_components. There's no actual tie between a distro entry point and a configurator entry point, so DefaultDistro would be fine as-is.
  • If I was doing that, I'd probably ignore OTEL_PYTHON_X_PROVIDER and just provide a big "Disable everything" switch instead which skips all setup.
• Implementing the documented none values for the OTEL_X_EXPORTER env-var and using those to disable export instead.
  • Edit: This is implemented, but not as entry-points. It's handled directly in the exporter auto-setup code that processes those env-vars. This is working for me locally with v1.11.1.
• Adding a command-line and env-var to opentelemetry-instrument which disables auto-instrumentation completely, somewhat along the lines of OTEL_PYTHON_DISABLED_INSTRUMENTATIONS but earlying-out of initialize in sitecustomize.py.
  • For logging, I see that OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED has just been added by Support logs SDK auto instrumentation enable/disable with env #2728 (and which defaults to false, and isn't visible in opentelemetry-instrumentation help), which has a similar effect, but inside _initialize_components. So maybe OTEL_PYTHON_AUTO_INSTRUMENTATION_ENABLED is the generalised version of that.
• OpenTelemetryDistro should not set OTEL_TRACES_EXPORTER if it is not already set, but leave that to the user, as it does for logs and metrics, although with stabilisation of metrics in v1.12, I guess that'll change to enable OTEL_METRICS_EXPORTER instead.
  • I ended up doing this, but in-passing for creating a project-local OTel Distribution library as a temporary fix for other things.

Additional context

I also noticed an odd issue that the get_X_provider code is set up to use default_X if the environment variable is not set, but the callers all ensure the environment variable is set before calling it. Perhaps the default_X entrypoints should point at the ProxyXProvider (except for logging, where that doesn't make sense, and since the code lives in the SDK, it can declare its own provider as default), and a noop_X entry-point could be used for the no-op providers explicitly, then we could always call opentelemetry.util._providers._load_provider and let its env-var handling do the heavy lifting.

It also might make sense to make _load_provider take a default provider to use if the env-var is not set, rather than falling back to default_X if the env-var is not set. Or just remove the "env-var not set" support from _load_provider.

[TBBle]

So I'm not sure why, but it looks like if I don't provide the OTEL_X_EXPORTER env-vars to the project, I don't get any errors, so I assume it's used the (in-effect) none option, even those grpc option is documented as the default.

It's doubly-weird because I have opentelemetry-distro installed, and its OpenTelemetryDistro should be being chosen and setting OTEL_TRACES_EXPORTER if not already set.

It's possible that the exporters are just silent if there's nothing listening for them, I guess. I'm not sure if that's a problem or not, but it means this issue isn't a problem for me, and we can consider that one of the alternatives.

Edit: Gah. Turns out that #2594 was hiding the errors from opentelemetry.exporter.otlp.proto.grpc.exporter in this case. So this isn't a solution.

I did try removing opentelemetry-distro since my understanding of the auto-instrumentation code was that if I did that, it'd fall back to DefaultDistro and hence honour the OTEL_X_EXPORTER env-vars I am passing to the project, but it didn't work: without opentelemetry-distro's opentelemetry_configurator entry point, nothing causes _OTelSDKConfigurator to be instantiated, hence _initialize_components is never called.

[TBBle]

Looking at the code while I was implementing a custom distro package, I've just noticed that "none" option for the OTEL_X_EXPORTER env-vars is supported, it's just handled directly in the auto-instrumentation code, rather than being an entrypoint. I'd only looked in setup.cfg for it, which is an oversight on my part.