k8s container attributes not included when exporting to Stackdriver

[james-tisato-kortical]

Describe your environment.

• Python 3.6.7, opencensus 0.5.0, opencensus-ext-stackdriver 0.3.0
• Instrumenting code running in a GKE Kubernetes cluster

Steps to reproduce.
Execute any tracing or stats collection code that exports to Stackdriver - the k8s metadata attributes will be missing from the results.

What is the expected behavior?
Results exported to Stackdriver should have associated attribute labels for project id, cluster name, pod name etc.

What is the actual behavior?
All k8s attributes are missing from the results I see in Stackdriver.

Additional context.
I think there is a bug in contrib/opencensus-ext-stackdriver/opencensus/ext/stackdriver/trace_exporter/__init__.py within set_monitored_resource_attributes:

def set_monitored_resource_attributes(span):
    """Set labels to span that can be used for tracing.
    :param span: Span object
    """
    resource = monitored_resource.get_instance()
    if resource is not None:
        resource_type = resource.get_type()
        resource_labels = resource.get_labels()

        if resource_type == 'gke_container':
            resource_type = 'k8s_container'
            ...

This code is still checking for resources of type gke_container, whereas opencensus/common/monitored_resource/monitored_resource.py now uses k8s_container when populating the resources to be automatically added to the Stackdriver results. Therefore, the above if block is entirely skipped and no k8s attributes are included.

The same bug appears to be present in the equivalent file for Stackdriver stats exporting (gke_container is also present there).

[james-tisato-kortical]

I monkey-patched in a fix locally and found a follow-on problem.

The code within set_monitored_resource_attributes still uses the old attribute key names when calling set_attribute_label - e.g. set_attribute_label(span, resource_type, resource_labels, 'cluster_name') doesn't work because the correct cluster name attribute key is now k8s.io/cluster/name (not cluster_name) after this change: 1d8bf1c

I think set_monitored_resource_attributes needs to be updated to use CLUSTER_NAME_KEY, POD_NAME_KEY etc. from k8s_utils.py to ensure consistency.

def set_monitored_resource_attributes(span):
    """Set labels to span that can be used for tracing.
    :param span: Span object
    """
    resource = monitored_resource.get_instance()
    if resource is not None:
        resource_type = resource.get_type()
        resource_labels = resource.get_labels()

        if resource_type == 'gke_container':
            resource_type = 'k8s_container'
            set_attribute_label(span, resource_type, resource_labels,
                                'project_id')
            set_attribute_label(span, resource_type, resource_labels,
                                'cluster_name')
            set_attribute_label(span, resource_type, resource_labels,
                                'container_name')
            set_attribute_label(span, resource_type, resource_labels,
                                'namespace_id', 'namespace_name')
            set_attribute_label(span, resource_type, resource_labels,
                                'pod_id', 'pod_name')
            set_attribute_label(span, resource_type, resource_labels,
                                'zone', 'location')
            ...

[c24t]

Thanks for raising the issue @james-tisato-kortical, let me know if #656 fixes it for you.

[seawolf42]

What release did this fix go out in? I am seeing what appears to be the same problem:

Error handling metric export
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (container_name namespace_name).: timeSeries[0-2]"
	debug_error_string = "{"created":"@1569538715.171019416","description":"Error received from peer ipv4:108.177.112.95:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"One or more TimeSeries could not be written: The set of resource labels is incomplete. Missing labels: (container_name namespace_name).: timeSeries[0-2]","grpc_status":3}"

Is this the same issue?

[lzchen]

@seawolf42
The PR was put into the 0.6.0 release.
cc @c24t

[seawolf42]

In that case, is what I'm seeing a different issue or a regression?