Add logs for container in bad status

[yanweiguo]

In what area(s)?

/area monitoring

Describe the feature

Add logs to indicate that user-container steps into/recovers from bad status.

If a request results in container crash and restart, the request returns 502 but there's no logs/events/ways to know what had happened.

For example, follow the steps below:

1. Deploy autoscale-go sample app with 256Mi as memory resources limits.
2. Run watch kubectl get pods to keep pods status refreshed.
3. Send requests with ?bloat=1000 multiple times:
   curl -H "Host: autoscale-go.default.example.com" http://<Istio-gateway IP>?bloat=500
4. You will get response with 502 and see the pod change status into OOMKilled for a few seconds and the user-container get restarted.

There is Last State in the output of kubectl get pod -oyaml result:

Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Tue, 25 Jun 2019 17:51:04 -0700
      Finished:     Tue, 25 Jun 2019 18:08:50 -0700

whose started time is the first OOMKilled event happened and finished time is the last OOMKilled event finished(during that period the container is ready for most of the time).

There are no other logs from K8S or Knative components indicating OOMKilled happened. So unless the pod is kept alive and the operator check the correct pod, no ways to know what resulted in the 502 response.

[yanweiguo]

There's also a similar feature request in k8s: kubernetes/kubernetes#69676

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

[knative-housekeeping-robot]

Stale issues rot after 30 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.
Rotten issues close after an additional 30 days of inactivity.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle rotten

[knative-housekeeping-robot]

Rotten issues close after 30 days of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/close

[knative-prow-robot]

@knative-housekeeping-robot: Closing this issue.

Details

In response to this:

Rotten issues close after 30 days of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh by adding the comment /remove-lifecycle rotten.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.