Istio sidecar injection needs to be opt out rather than opt in

[mdemirhan]

Expected Behavior

Copying/pasting relevant threads from Slack:

Matt Moore: Can't we opt containers into sidecar injection instead of out of injection?
removing the namespace annotation makes the activator stop getting sidecard. I don't understand the logic of only having an opt-out per Pod
I think it would be good if we could get sidecar injection without requiring everything in the namespace to have it or opt out as well

[tcnghia]

Per https://istio.io/docs/setup/kubernetes/sidecar-injection/#policy

policy=disabled --> The sidecar injector will not inject the sidecar into pods by default. Add the sidecar.istio.io/inject annotation with value true to the pod template spec to enable injection.

policy=enabled --> The sidecar injector will inject the sidecar into pods by default. Add the sidecar.istio.io/inject annotation with value false to the pod template spec to disable injection

@mattmoor

[tcnghia]

So the annotation is to use for negating the default policy.

[mattmoor]

I just deleted the annotation from config/100-namespace.yaml and brought our stuff up. The Activator was 1/1, so no sidecar. Would be good to get independent confirmation.

[tcnghia]

I tried with this sample.yaml

apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio-injection: enabled
  name: policy-enabled
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio-injection: disabled
  name: policy-disabled
---
apiVersion: v1
kind: Namespace
metadata:
  name: policy-default
---
apiVersion: v1
kind: Pod
metadata:
  name: nolabel
  namespace: policy-enabled
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: nolabel
  namespace: policy-disabled
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: nolabel
  namespace: policy-default
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: labeled-true
  namespace: policy-enabled
  annotations:
    "sidecar.istio.io/inject": "true"
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: labeled-true
  namespace: policy-disabled
  annotations:
    "sidecar.istio.io/inject": "true"
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: labeled-true
  namespace: policy-default
  annotations:
    "sidecar.istio.io/inject": "true"
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: labeled-false
  namespace: policy-enabled
  annotations:
    "sidecar.istio.io/inject": "off"
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: labeled-false
  namespace: policy-disabled
  annotations:
    "sidecar.istio.io/inject": "off"
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin
---
apiVersion: v1
kind: Pod
metadata:
  name: labeled-false
  namespace: policy-default
  annotations:
    "sidecar.istio.io/inject": "off"
spec:
  containers:
  - image: docker.io/citizenstig/httpbin
    imagePullPolicy: IfNotPresent
    name: httpbin

It looks like the annotation is only effective when policy=enabled and annotation sidecar.istio.io/inject=off

$ k get pods --all-namespaces=true | grep "label\|NAME"
NAMESPACE                NAME                                                READY     STATUS      RESTARTS   AGE
policy-default           labeled-false                                       1/1       Running     0          37s
policy-default           labeled-true                                        1/1       Running     0          38s
policy-default           nolabel                                             1/1       Running     0          38s
policy-disabled          labeled-false                                       1/1       Running     0          37s
policy-disabled          labeled-true                                        1/1       Running     0          38s
policy-disabled          nolabel                                             1/1       Running     0          38s
policy-enabled           labeled-false                                       1/1       Running     0          38s
policy-enabled           labeled-true                                        2/2       Running     0          6s
policy-enabled           nolabel                                             2/2       Running     0          6s

[tcnghia]

I contacted Istio team and they think this is a bug, I'll file one.

[tcnghia]

Filed istio/istio#6476

[tcnghia]

Summary of the current behavior

Namespace label istio-injection	ConfigMap key policy	Pod annotation sidecar.istio.io/inject	Injection
enabled	enabled	true	yes
enabled	enabled	false	no
enabled	enabled		yes
enabled	disabled	true	yes
enabled	disabled	false	no
enabled	disabled		no
disabled	enabled	true	no
disabled	enabled	false	no
disabled	enabled		no
disabled	disabled	true	no
disabled	disabled	false	no
disabled	disabled		no

So ConfigMap chooses whether we are opt-in, or opt-out (by using the Pod annotation). The namespace label needs to be set to enabled for all namespaces that we want injection to happen.

Our ConfigMap is set to policy=enabled: which means that:

• in namespaces that we disable injection: no injection will happen, ever.
• in namespaces that we enable injection: injection will happen unless we opt-out at Pod level.

If we set ConfigMap to policy=disabled, then

• in namespaces that we disable injection: no injection will happen, ever.
• in namespaces that we enable injection: injection will not happen unless we opt-in at Pod level.

Currently we can't have both of these policy, since the setting is applied globally in the injector ConfigMap.

[tcnghia]

This is being changed in istio/istio#6439, probably will release in 1.0

[jchesterpivotal]

Stumbled across this while working on knative/build#216.

Installing the helm stable/concourse chart is unsuccessful until I target a namespace without istio sidecar injection. More annoyingly, it is not immediately obvious that this was the cause. I expect other helm charts will misbehave similarly.