Make KUBECONFIG a template variable that is interpolated in templates (config map, cli, env)

[razvan]

Problem

Prometheus requires the kube configuration file to be present for service discovery with Kubernetes. The only process who has access to this file on the nodes is the agent. It cannot be set by the operators.

Proposed solution

Make KUBECONFIG a variable known by the agent similarly to {{packageroot}} or {{configroot}}. Allow operators to use this variable in templates and replace it with the appropriate path.

Example

The monitoring operator creates a configmap with the fils prometheus.yaml. Therein the k8s service discovery is configured like this snipped shows:

    kubernetes_sd_configs:
      - role: pod
        kubeconfig_file: {{KUBECONFIG}} 
        namespaces:
          names:
            - {}
        selectors:
          - role: pod

[razvan]

After more discussions, we agree that this is not the best way to proceed for multiple reasons:

• it makes the agent diverge even more from the k8s kublet behaviour.
• the kube config file used by the agent should not be shared with operators for security reasons.