Adding new Istio routerule causes small window of 503s

[loewenstein]

Steps:

1. bazel run sample/helloworld:everything.create
2. while true; do curl -is --header 'Host:demo.myhost.net' -w "%{http_code}\n" -o /dev/null http://${SERVICE_IP}; done | grep 503
3. bazel run sample/helloworld:updated_everything.apply

Expected:
App stays available, i.e. command from 3. stays quiet

Actual:
App is partially unavailable.

We should add a readiness check to the configuration.

[loewenstein]

I improved the monitoring of status changes with

#!/usr/bin/env bash

LAST_STATUS="Unknown"
CURRENT_STATUS="Unknown"

while true
do
  CURRENT_STATUS=$(curl -is --header 'Host:demo.myhost.net' -w "%{http_code}\n" -o /dev/null  http://${SERVICE_IP})
  if [ $CURRENT_STATUS != $LAST_STATUS ]; then
    echo "$(date): ${LAST_STATUS} -> ${CURRENT_STATUS}"
  fi
  LAST_STATUS=$CURRENT_STATUS
done

I added

readynessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 3

to both configuration.yaml and updated_configuration.yaml.

However, I continue to see transitions to HTTP 503 when I switch back and forth between bazel run sample/helloworld:everything.apply and bazel run sample/helloworld:updated_everything.apply.

$ sample/helloworld/status.sh
Fri Mar 16 11:25:47 CET 2018: Unknown -> 404
Fri Mar 16 11:27:56 CET 2018: 404 -> 503
Fri Mar 16 11:28:19 CET 2018: 503 -> 200
Fri Mar 16 11:31:25 CET 2018: 200 -> 503
Fri Mar 16 11:31:25 CET 2018: 503 -> 200
Fri Mar 16 11:32:55 CET 2018: 200 -> 503
Fri Mar 16 11:32:56 CET 2018: 503 -> 200
Fri Mar 16 11:33:18 CET 2018: 200 -> 503
Fri Mar 16 11:34:55 CET 2018: 503 -> 200
Fri Mar 16 12:29:26 CET 2018: 200 -> 503
Fri Mar 16 12:29:26 CET 2018: 503 -> 200
Fri Mar 16 12:29:48 CET 2018: 200 -> 503
Fri Mar 16 12:29:50 CET 2018: 503 -> 200
Fri Mar 16 12:30:42 CET 2018: 200 -> 503
Fri Mar 16 12:30:43 CET 2018: 503 -> 200

@mattmoor Is there anything I miss? Otherwise I suppose there is more than a missing readiness probe at work. I'll dig a bit in the code if I can figure out what might be going on.

[mattmoor]

That's be great.

Adding @vaikas-google @ian-mi in case they have any ideas what might be causing this.

[loewenstein]

Could this simple be an Istio bug, i.e. https://docs.google.com/document/d/1UXJIqfUGkZlw1aZYSkQDYkq1TnYbOaEkohtoTS2mz1g/edit#heading=h.x9snb54sjlu9

In any case I will submit a PR adding readiness probes.

[mattmoor]

I think that's most likely

[mattmoor]

Looks like readYnessProbe was a typo (should be: readinessProbe). I started to look into stricter JSON validation yesterday, but Go finally got around to adding an option to disallow unknown fields to encoding/json in 1.10, which K8s isn't yet on (there are issues).

[mattmoor]

/assign @tcnghia