storage-version-migration job never completes on istio enabled cluster

[AceHack]

What version of Knative?

0.15.1

Expected Behavior

storage-version-migration job to finish on istio enabled cluster

Actual Behavior

storage-version-migration job never completes on istio enabled cluster

Steps to Reproduce the Problem

Install 0.15.1 on istio enabled cluster with auto-injection enabled

Related

This is the same issue from here
knative/eventing-operator#194

Also, it's related to:
istio/istio#6324

Both
https://github.com/knative/serving/blob/release-0.15/config/post-install/storage-version-migration.yaml
and
https://github.com/knative/serving/blob/release-0.15/config/post-install/default-domain.yaml

Need

    metadata:
      annotations:
        sidecar.istio.io/inject: "false"

I'm happy to make the PR to fix this

[laballab]

I'm having this issue as well 😕

[ssandanshi]

I would appreciate it if this was resolved as well.

[dprotaso]

Posted in slack but worth mentioning here:

I’m confused - why do our controllers work but not these jobs. For example the storage migration job just hits the K8s API directly.

serving/config/core/deployments/controller.yaml

Lines 27 to 32 in b221eab

	metadata:
	annotations:
	cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
	labels:
	app: controller
	serving.knative.dev/release: devel

[tcnghia]

@dprotaso Controllers allow Pod restarts, giving time for the sidecar to start up. I think batch jobs are different.

[AceHack]

@dprotaso it's related to this istio issue where istio does not work correctly with k8s jobs
istio/istio#6324

[dprotaso]

that work around mentioned in the issue isn't great :(

[dprotaso]

thanks for the context