Istio Installation without sidecar injection failing (Istio version 1.8)

[dtsioumas]

What version of Knative?

0.19.0

Expected Behavior

Installing successfully Istio without sidecar injection.

Actual Behavior

Installation procedure failing for Istio without sidecar injection.

Steps to Reproduce the Problem

• Install the knative CRDs:

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.19.0/serving-crds.yaml

• Install the core components of Serving:

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.19.0/serving-core.yaml

• Install Istio without a sidecar injection:

istio-minimal-operator.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
      # The third-party-jwt is not enabled on all k8s.
      # See: https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens
      jwtPolicy: first-party-jwt

  addonComponents:
    pilot:
      enabled: true
    prometheus:
      enabled: false

  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              targetPort: 8080
              name: http2
            - port: 443
              targetPort: 8443
              name: https

[user@xxx]$ istioctl install -f istio-minimal-operator.yaml
This will install the Istio profile into the cluster. Proceed? (y/N) y
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/v1.8/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
Error: failed to install manifests: errors occurred during operation: failed to create Istio control plane with spec: 
hub:"docker.io/istio" tag:1.8.0 mesh_config:<key:"defaultConfig" value:map[proxyMetadata:map[]] > mesh_config:<key:"enablePrometheusMerge" value:true > components:<base:<enabled:<BoolValue:<value:true > > > pilot:<enabled:<BoolValue:<value:true > > tag:<nil> spec:<nil> k8s:<env:<name:"POD_NAME" valueFrom:<fieldRef:<apiVersion:"v1" fieldPath:"metadata.name" > > > env:<name:"POD_NAMESPACE" valueFrom:<fieldRef:<apiVersion:"v1" fieldPath:"metadata.namespace" > > > readiness_probe:<httpGet:<path:"/ready" port:<IntOrString:<type:0 intVal:8080 strVal:"" > > > initialDelaySeconds:1 timeoutSeconds:5 periodSeconds:3 > strategy:<rollingUpdate:<maxUnavailable:<IntOrString:<type:1 intVal:0 strVal:"25%" > > maxSurge:<IntOrString:<type:1 intVal:0 strVal:"100%" > > > > > > cni:<enabled:<BoolValue:<> > tag:<nil> spec:<nil> > istiod_remote:<enabled:<BoolValue:<> > tag:<nil> spec:<nil> > ingress_gateways:<enabled:<BoolValue:<value:true > > name:"istio-ingressgateway" tag:<nil> k8s:<env:<name:"ISTIO_META_ROUTER_MODE" value:"standard" > hpa_spec:<scaleTargetRef:<kind:"Deployment" name:"istio-ingressgateway" apiVersion:"apps/v1" > minReplicas:1 maxReplicas:5 metrics:<type:"Resource" resource:<name:"cpu" targetAverageUtilization:80 > > > resources:<limits:<key:"cpu" value:"2000m" > limits:<key:"memory" value:"1024Mi" > requests:<key:"cpu" value:"100m" > requests:<key:"memory" value:"128Mi" > > service:<ports:<name:"status-port" protocol:"TCP" port:15021 targetPort:<IntOrString:<type:0 intVal:15021 strVal:"" > > > ports:<name:"http2" protocol:"TCP" port:80 targetPort:<IntOrString:<type:0 intVal:8080 strVal:"" > > > ports:<name:"https" protocol:"TCP" port:443 targetPort:<IntOrString:<type:0 intVal:8443 strVal:"" > > > ports:<name:"tcp-istiod" protocol:"TCP" port:15012 targetPort:<IntOrString:<type:0 intVal:15012 strVal:"" > > > ports:<name:"tls" protocol:"TCP" port:15443 targetPort:<IntOrString:<type:0 intVal:15443 strVal:"" > > > > strategy:<rollingUpdate:<maxUnavailable:<IntOrString:<type:1 intVal:0 strVal:"25%" > > maxSurge:<IntOrString:<type:1 intVal:0 strVal:"100%" > > > > > > ingress_gateways:<enabled:<BoolValue:<value:true > > name:"cluster-local-gateway" label:<key:"app" value:"cluster-local-gateway" > label:<key:"istio" value:"cluster-local-gateway" > tag:<nil> k8s:<service:<ports:<name:"status-port" port:15020 > ports:<name:"http2" port:80 targetPort:<IntOrString:<type:0 intVal:8080 strVal:"" > > > ports:<name:"https" port:443 targetPort:<IntOrString:<type:0 intVal:8443 strVal:"" > > > type:"ClusterIP" > > > egress_gateways:<enabled:<BoolValue:<> > name:"istio-egressgateway" tag:<nil> k8s:<env:<name:"ISTIO_META_ROUTER_MODE" value:"standard" > hpa_spec:<scaleTargetRef:<kind:"Deployment" name:"istio-egressgateway" apiVersion:"apps/v1" > minReplicas:1 maxReplicas:5 metrics:<type:"Resource" resource:<name:"cpu" targetAverageUtilization:80 > > > resources:<limits:<key:"cpu" value:"2000m" > limits:<key:"memory" value:"1024Mi" > requests:<key:"cpu" value:"100m" > requests:<key:"memory" value:"128Mi" > > service:<ports:<name:"http2" protocol:"TCP" port:80 targetPort:<IntOrString:<type:0 intVal:8080 strVal:"" > > > ports:<name:"https" protocol:"TCP" port:443 targetPort:<IntOrString:<type:0 intVal:8443 strVal:"" > > > ports:<name:"tls" protocol:"TCP" port:15443 targetPort:<IntOrString:<type:0 intVal:15443 strVal:"" > > > > strategy:<rollingUpdate:<maxUnavailable:<IntOrString:<type:1 intVal:0 strVal:"25%" > > maxSurge:<IntOrString:<type:1 intVal:0 strVal:"100%" > > > > > > > addon_components:<key:"istiocoredns" value:<enabled:<BoolValue:<> > spec:<nil> > > addon_components:<key:"pilot" value:<enabled:<BoolValue:<value:true > > spec:<nil> > > addon_components:<key:"prometheus" value:<enabled:<BoolValue:<> > spec:<nil> > > values:<key:"base" value:map[enableCRDTemplates:false validationURL:] > values:<key:"clusterResources" value:true > values:<key:"gateways" value:map[istio-egressgateway:map[autoscaleEnabled:true env:map[] name:istio-egressgateway secretVolumes:[map[mountPath:/etc/istio/egressgateway-certs name:egressgateway-certs secretName:istio-egressgateway-certs] map[mountPath:/etc/istio/egressgateway-ca-certs name:egressgateway-ca-certs secretName:istio-egressgateway-ca-certs]] type:ClusterIP zvpn:map[]] istio-ingressgateway:map[autoscaleEnabled:true env:map[] name:istio-ingressgateway secretVolumes:[map[mountPath:/etc/istio/ingressgateway-certs name:ingressgateway-certs secretName:istio-ingressgateway-certs] map[mountPath:/etc/istio/ingressgateway-ca-certs name:ingressgateway-ca-certs secretName:istio-ingressgateway-ca-certs]] type:LoadBalancer zvpn:map[]]] > values:<key:"global" value:map[arch:map[amd64:2 ppc64le:2 s390x:2] configValidation:true defaultNodeSelector:map[] defaultPodDisruptionBudget:map[enabled:true] defaultResources:map[requests:map[cpu:10m]] imagePullPolicy: imagePullSecrets:[] istioNamespace:istio-system istiod:map[enableAnalysis:false] jwtPolicy:first-party-jwt logAsJson:false logging:map[level:default:info] meshExpansion:map[enabled:false useILB:false] meshNetworks:map[] mountMtlsCerts:false multiCluster:map[clusterName: enabled:false] network: omitSidecarInjectorConfigMap:false oneNamespace:false operatorManageWebhooks:false pilotCertProvider:istiod priorityClassName: proxy:map[autoInject:disabled clusterDomain:cluster.local componentLogLevel:misc:error enableCoreDump:false excludeIPRanges: excludeInboundPorts: excludeOutboundPorts: image:proxyv2 includeIPRanges:* logLevel:warning privileged:false readinessFailureThreshold:30 readinessInitialDelaySeconds:1 readinessPeriodSeconds:2 resources:map[limits:map[cpu:2000m memory:1024Mi] requests:map[cpu:100m memory:128Mi]] statusPort:15020 tracer:zipkin] proxy_init:map[image:proxyv2 resources:map[limits:map[cpu:2000m memory:1024Mi] requests:map[cpu:10m memory:10Mi]]] sds:map[token:map[aud:istio-ca]] sts:map[servicePort:0] tracer:map[datadog:map[] lightstep:map[] stackdriver:map[] zipkin:map[]] useMCP:false] > values:<key:"istiocoredns" value:map[coreDNSImage:coredns/coredns coreDNSPluginImage:istio/coredns-plugin:0.2-istio-1.1 coreDNSTag:1.6.2] > values:<key:"istiodRemote" value:map[injectionURL:] > values:<key:"pilot" value:map[autoscaleEnabled:true autoscaleMax:5 autoscaleMin:1 configMap:true cpu:map[targetAverageUtilization:80] deploymentLabels:<nil> enableProtocolSniffingForInbound:true enableProtocolSniffingForOutbound:true env:map[ENABLE_LEGACY_FSGROUP_INJECTION:false] image:pilot keepaliveMaxServerConnectionAge:30m nodeSelector:map[] replicaCount:1 traceSampling:1] > values:<key:"sidecarInjectorWebhook" value:map[enableNamespacesByDefault:false objectSelector:map[autoInject:true enabled:false] rewriteAppHTTPProbe:true] > values:<key:"telemetry" value:map[enabled:true v2:map[enabled:true metadataExchange:map[wasmEnabled:false] prometheus:map[enabled:true wasmEnabled:false] stackdriver:map[configOverride:map[] enabled:false logging:false monitoring:false topology:false]]] > 
error: component "prometheus" is not longer supported. Please remove it from the addonComponent configuration. See https://istio.io/latest/blog/2020/addon-rework/ for more info

Environment info

[user@xxx]$ istioctl version
client version: 1.8.0
control plane version: 1.8.0
data plane version: 1.8.0 (2 proxies)

[user@xxx]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"archive", BuildDate:"2020-10-16T20:58:56Z", GoVersion:"go1.15.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

[user@xxx]$ kubectl get nodes
NAME                                          STATUS   ROLES    AGE    VERSION
knative-cluster-master-0   Ready    master   2d9h   v1.19.3
knative-cluster-node-0     Ready    <none>   2d9h   v1.19.3
knative-cluster-node-1     Ready    <none>   2d9h   v1.19.3

Possible Solution

According to official documentation Istio version 1.8 the Installation of addons by istioctl is removed.

Removing prometheus value from addonComponents it seems solving the problem:

istio-minimal-operator.yaml

[user@xxx]$ cat istio-minimal-operator.yaml 
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
      # The third-party-jwt is not enabled on all k8s.
      # See: https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens
      jwtPolicy: first-party-jwt

  addonComponents:
    pilot:
      enabled: true

  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - port: 15020
              name: status-port
            - port: 80
              targetPort: 8080
              name: http2
            - port: 443
              targetPort: 8443
              name: https

[user@xxx]$ istioctl install -f istio-minimal-operator.yaml
This will install the Istio profile into the cluster. Proceed? (y/N) y
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/v1.8/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
✔ Istio core installed                                                                                                                                               
✔ Istiod installed                                                                                                                                                   
✔ Addons installed                                                                                                                                                   
✔ Ingress gateways installed                                                                                                                                         
✔ Installation complete       

[nak3]

Thank you for the report. knative-extensions/net-istio#426 starts testing the Istio 1.8 with the updated manifest. Please give us some time.
/cc @arturenault

[arturenault]

Yup, I noticed this during installation. Can update the docs accordingly.
/assign

[arturenault]

/close

fixed by the above pr. not sure why it didn't close this

[knative-prow-robot]

@arturenault: Closing this issue.

Details

In response to this:

/close

fixed by the above pr. not sure why it didn't close this

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.