Vanishing revisions

[duglin]

In what area(s)?

/area API
/area autoscale

What version of Knative?

0.6.0
Running on IKS with istio-lean, not full istio.

The issue

After I let a service rest and scale down, some revisions vanish and the service isn't accessible.

To reproduce, use this script:

$ cat bug
#!/bin/bash

set -x

kubectl delete ksvc/echo > /dev/null 2>&1 && sleep 3

cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: echo
spec:
  template:
    metadata:
      name: echo-v1
    spec:
      containers:
        - image: duglin/echo
EOF

URL=$(kubectl get ksvc/echo -ojsonpath="{.status.url}")

sleep 120

cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: echo
spec:
  template:
    metadata:
      name: echo-v2
    spec:
      containers:
        - image: duglin/echo
  traffic:
  - tag: old
    revisionName: echo-v1
    percent: 50
  - tag: new
    revisionName: echo-v2
    percent: 50
EOF

sleep 120
cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: echo
spec:
  template:
    spec:
      containers:
        - image: duglin/echo
  traffic:
  - tag: old
    revisionName: echo-v1
    percent: 50
  - tag: new
    revisionName: echo-v2
    percent: 50
  - tag: latest
    latestRevision: true
    percent: 0
EOF

while kubectl get pods | grep echo ; do
  sleep 10
done

kubectl get ksvc
kubectl get revisions

curl -v $URL

kubectl get ksvc
kubectl get revisions

What you should see, near the end, after all of the pods vanish is output like this:

+ sleep 10
+ kubectl get pods
+ grep echo
No resources found.
+ kubectl get ksvc
NAME   URL                                                             LATESTCREATED   LATESTREADY   READY   REASON
echo   http://echo-default.knv06.us-south.containers.appdomain.cloud   echo-fjlln      echo-fjlln    False   RevisionMissing
+ kubectl get revisions
NAME         SERVICE NAME   GENERATION   READY   REASON
echo-fjlln   echo-fjlln     3            True    
+ curl -v http://echo-default.knv06.us-south.containers.appdomain.cloud
* Rebuilt URL to: http://echo-default.knv06.us-south.containers.appdomain.cloud/
*   Trying 169.62.159.45...
* Connected to echo-default.knv06.us-south.containers.appdomain.cloud (169.62.159.45) port 80 (#0)
> GET / HTTP/1.1
> Host: echo-default.knv06.us-south.containers.appdomain.cloud
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< Date: Fri, 31 May 2019 17:31:02 GMT
< Content-Length: 0
< Connection: keep-alive
< 
* Connection #0 to host echo-default.knv06.us-south.containers.appdomain.cloud left intact
+ kubectl get ksvc
NAME   URL                                                             LATESTCREATED   LATESTREADY   READY   REASON
echo   http://echo-default.knv06.us-south.containers.appdomain.cloud   echo-fjlln      echo-fjlln    False   RevisionMissing
+ kubectl get revisions
NAME         SERVICE NAME   GENERATION   READY   REASON
echo-fjlln   echo-fjlln     3            True    

The key things to notice is that:

• the service says "RevisionMissing"
• the # of revisions is now just one instead of 3, despite all 3 being reference by the service
• the curl to the service fails, which isn't unexpected given the other errors

[mattmoor]

What GC settings do you have?

[duglin]

just the defaults.

However, in trying to track down things I restarted my kn-serving controller and now I can't reproduce it using the above script because none of the pods (for any revision) scale down to zero. And, to make it even weirder, when I rename my service ttt instead of echo, not only do things scale down properly but the original bug is gone - the revisions do not vanish. I verified that everything named "echo" is deleted between tests too - go figure.

[duglin]

Just to add more info, in case something jumps out at you:

• doing kn service create echo --image duglin/echo works - meaning the KnService is created, and initial pod is created, and then the pod is scaled down to zero after about 100 seconds
• however, creating it via:

cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: echo
spec:
  template:
    metadata:
      name: echo-v1
    spec:
      containers:
        - image: duglin/echo
EOF

fails - meaning, the pod never scales downs.

• but removing the metadata and name fields does result it in working/scaling down.

So perhaps the BYO logic is influencing things? But it's weird that it only does it for "echo" and not "ttt" :-)

[mattmoor]

With the linked PR merged, and the disappearing repo, is any work left for this?

[duglin]

since I can't reproduce this reliably I'm gonna close this and will reopen if I see it again.