Mysterious Warn logs in dev loop when container is terminated and watching for changes

[tejal29]

I am testing a scenario where users have a deployed container fails before status check phase is complete.

As expected Skaffold

1. reports the container is terminated with exit code.
2. Tries fetching logs which arent there.
3. quits the status check loop and watches for changes

Unexpected:

1. I see the following warn messages in the watching phase WARN[0046] exit status 1
2. WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed is incorrect because the deploy already happened and we don't clean up.

Waiting for deployments to stabilize...
 - deployment/leeroy-web is ready. [1/2 deployment(s) still pending]
WARN[0045] exit status 1                                
 - deployment/leeroy-app: container leeroy-app terminated with exit code 2
    - pod/leeroy-app-794c7c586d-2qrh2: container leeroy-app terminated with exit code 2
      > Error retrieving logs for pod leeroy-app-794c7c586d-2qrh2. Try `kubectl logs leeroy-app-794c7c586d-2qrh2 -n default -c leeroy-app`
 - deployment/leeroy-app failed. Error: container leeroy-app terminated with exit code 2.
WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed 
Watching for changes...
WARN[0046] exit status 1                                
WARN[0060] exit status 1                                
WARN[0090] exit status 1  

How to reproduce

1. Add a lag in status check so the container has time to crash before status check ends.

+++ b/pkg/skaffold/deploy/status/status_check.go
@@ -97,6 +97,7 @@ func NewStatusChecker(cfg Config, labeller *label.DefaultLabeller) Checker {
 // Run runs the status checks on deployments and pods deployed in current skaffold dev iteration.
 func (s statusChecker) Check(ctx context.Context, out io.Writer) error {
        event.StatusCheckEventStarted()
+       time.Sleep(1*time.Second)
        errCode, err := s.statusCheck(ctx, out)
        event.StatusCheckEventEnded(errCode, err)
        return err

3. LOCAL=true make

4. run skaffold dev

5. After the first deploy, add a NPE to mimic container crash

--- a/examples/microservices/leeroy-app/app.go
+++ b/examples/microservices/leeroy-app/app.go
@@ -12,6 +12,7 @@ func handler(w http.ResponseWriter, r *http.Request) {
 
 func main() {
        log.Print("leeroy app server ready")
+       fmt.Fprintf(nil, "leeroooooy app!!\n")
        http.HandleFunc("/", handler)
        http.ListenAndServe(":50051", nil)
 }