mutation failed: cannot decode incoming new object: json: unknown field "subresource"

[howardjohn]

/area networking

What version of Knative?

0.23

Expected Behavior

Hello world works

Actual Behavior

Kservice stuck, logs are showing a bunch of errors like:

{"severity":"INFO","timestamp":"2021-06-01T18:52:01.271342634Z","logger":"webhook","caller":"webhook/admission.go:131","message":"remote admission controller audit annotations=map[string]string(nil)","commit":"35efb31","knative.dev/pod":"webhook-6f4d97c469-dvfg7","knative.dev/kind":"serving.knative.dev/v1, Kind=Route","knative.dev/namespace":"default","knative.dev/name":"hello","knative.dev/operation":"UPDATE","knative.dev/resource":"serving.knative.dev/v1, Resource=routes","knative.dev/subresource":"status","knative.dev/userinfo":"{system:serviceaccount:knative-serving:controller 515c0bce-588c-44d2-b9ee-cdade6b60f1d [system:serviceaccounts system:serviceaccounts:knative-serving system:authenticated] map[authentication.kubernetes.io/pod-name:[controller-7c697c6857-7r5zj] authentication.kubernetes.io/pod-uid:[aa02fc23-43e6-4519-9e28-002016b408e7]]}","admissionreview/uid":"daf45280-2ffd-441a-8bcd-2fe6ee9047cd","admissionreview/allowed":false,"admissionreview/result":"&Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:Failure,Message:mutation failed: cannot decode incoming new object: json: unknown field \"subresource\",Reason:BadRequest,Details:nil,Code:400,}"}

Steps to Reproduce the Problem

kind create cluster
istioctl install -y
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.23.0/serving-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.23.0/serving-core.yaml
kubectl wait pod --timeout=-1s --for=condition=Ready -l '!job-name' -n knative-serving

kubectl apply --filename https://github.com/knative/net-istio/releases/download/v0.23.0/net-istio.yaml
kubectl replace --filename https://github.com/knative/serving/releases/download/v0.23.0/serving-default-domain.yaml


cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go
          ports:
            - containerPort: 8080
          env:
            - name: TARGET
              value: "Knative"
EOF

After manually deleting all the mutating and validating webhooks seems thing to start working

[nak3]

/area API

I tried serving v0.23 but it worked fine on my end.
json: unknown field \"subresource\" seems to be API issue 🤔 Forwarding to API team.

[julz]

@howardjohn are you able to consistently reproduce this on a clean kind cluster? If so could you let us know version of kind you're using, OS, any other info that may be helpful

/needs-user-input

[julz]

/triage needs-user-input

[daisy-ycguo]

I met with the same problem on Kubernetes 1.22. It used to work fine on Kubernetes 1.21.
@julz，could you check your Kubernetes version ?

[vdemeester]

This is also something happening on tektoncd/pipeline with 1.22 (OCP 4.9 nightly and even k8s 1.22 it seems)

[julz]

Created #11805 to get 1.22 in to CI and see if we can reproduce there.