Update reconciler to update observed generation when ready status is False#4594
Conversation
knative-prow-robot
left a comment
There was a problem hiding this comment.
@taragu: 0 warnings.
Details
In response to this:
Fixes #4561
Proposed Changes
- Update reconciler to update observed generation when ready status is False
Release Note
NONE/cc @dgerd
/cc @mattmoor
/cc @vagababov
/cc @markusthoemmes
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
|
Hi @taragu. Thanks for your PR. I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
I'd expect some unit tests to change as well... /ok-to-test |
| service.Status.MarkConfigurationNotReconciled() | ||
| } else { | ||
| } | ||
| if config.Generation == config.Status.ObservedGeneration || (rc != nil && rc.Status == corev1.ConditionFalse) { |
There was a problem hiding this comment.
I suspect the bug here is that Configuration / Route aren't updating the ObservedGeneration for which they are failing.
There was a problem hiding this comment.
+1
This condition will be more clear if we instead make sure that we bump "ObservedGeneration" in the Route and Configuration reconcilers when "Ready" is updated to "False".
There was a problem hiding this comment.
+1
For this case, "ObservedGeneration" in the configuration reconciler is already updated.
https://github.com/knative/serving/blob/master/pkg/reconciler/configuration/configuration.go#L167
In router reconciler, "ObeservedGeneration" will not be updated until route is successfully synced.
https://github.com/knative/serving/blob/master/pkg/reconciler/route/route.go#L242
Implementation in Route and Configuration reconcilers is a little different.
5cf40fc to
af65283
Compare
| rc := route.Status.GetCondition(apis.ConditionReady) | ||
| if rc != nil && rc.Status == corev1.ConditionFalse { | ||
| logger.Infof("Updating ObservedGeneration") | ||
| existing.Status.ObservedGeneration++ |
There was a problem hiding this comment.
| existing.Status.ObservedGeneration++ | |
| existing.Status.ObservedGeneration = existing.Generation |
We might miss spec updates and catch up to multiple changes.
There was a problem hiding this comment.
I don't think the change you have here is right. I think this should be taken care of by line 188
There was a problem hiding this comment.
@mattmoor I think line 188 is updating the ObservedGeneration for service.Status, versus here we are updating the ObservedGeneration for route.Status
There was a problem hiding this comment.
Oh, I misread.
Status fields for resources should only be modified by the reconciler for that resource. So this is likely the change we want, but we want it in the Route reconciler, not the Service reconciler.
|
|
||
| rc := route.Status.GetCondition(apis.ConditionReady) | ||
| if rc != nil && rc.Status == corev1.ConditionFalse { | ||
| logger.Infof("Updating ObservedGeneration") |
There was a problem hiding this comment.
I'd remove this, it's going to be super noisy and doesn't include any state.
| rc := route.Status.GetCondition(apis.ConditionReady) | ||
| if rc != nil && rc.Status == corev1.ConditionFalse { | ||
| logger.Infof("Updating ObservedGeneration") | ||
| existing.Status.ObservedGeneration++ |
There was a problem hiding this comment.
I don't think the change you have here is right. I think this should be taken care of by line 188
af65283 to
23e58a8
Compare
| } | ||
|
|
||
| // MarkRouteReadyFalse marks the service `RouteReady` condition to the `False` state. | ||
| func (ss *ServiceStatus) MarkRouteReadyFalse(reason, messageFormat string) { |
There was a problem hiding this comment.
I think a better name would be MarkRouteFailed.
| } | ||
| } | ||
|
|
||
| // WithRouteObservedGeneration sets the .Status.ObservedGeneration of the Route. |
There was a problem hiding this comment.
| // WithRouteObservedGeneration sets the .Status.ObservedGeneration of the Route. | |
| // WithRouteObservedGeneration propagates `.Metadata.Generation` to `.Status.ObservedGeneration`. |
23e58a8 to
6c9b19e
Compare
|
The following is the coverage report on pkg/.
|
|
|
||
| rc := route.Status.GetCondition(apis.ConditionReady) | ||
| if rc != nil && rc.Status == corev1.ConditionFalse { | ||
| existing.Status.ObservedGeneration = existing.Generation |
There was a problem hiding this comment.
Changes to anything in Route's Status should only occur in pkg/reconciler/route/route.go
6c9b19e to
e465bb4
Compare
e465bb4 to
e954c2d
Compare
d628ef6 to
3257723
Compare
|
/test pull-knative-serving-build-tests |
| // assumptions about defaulting. | ||
| r.SetDefaults(v1beta1.WithUpgradeViaDefaulting(ctx)) | ||
| r.Status.InitializeConditions() | ||
| r.Status.ObservedGeneration = r.Generation |
There was a problem hiding this comment.
Okay so I spent some more time looking at this. As not all returned errors from this method result in Conditions being updated to "False" which was my original assumption. We likely have some other bugs to fix in this reconciler because of this, but to prevent scope creep we should move it back to where you originally had it.
So this should be on:
- Line 229 after
configureTrafficfails (where you originally had it) - Line 287 after we reconcile successfully
Sorry for the back and forth here.
| } | ||
|
|
||
| t.Logf("Creating a new Service %s.", names.Service) | ||
| svc, err := CreateLatestService(t, clients, *names, fopt...) |
There was a problem hiding this comment.
You should just use this directly in your test. I don't think the rest of this function is really necessary to accomplish what you want.
| // track how long it took for name to get into the state checked by inState. | ||
| func WaitForRouteState(client *test.ServingBetaClients, name string, inState func(r *v1beta1.Route) (bool, error), desc string) error { | ||
| // ignoreNotFound indicates that we will keep polling even if the route is not found. | ||
| func WaitForRouteState(client *test.ServingBetaClients, name string, inState func(r *v1beta1.Route) (bool, error), desc string, ignoreNotFound bool) error { |
There was a problem hiding this comment.
This just doesn't make sense to me. If we have already waited for the Service to be Ready=False there is no way that the Route doesn't exist yet unless something else is going wrong.
There was a problem hiding this comment.
@dgerd I tried removing this change after I updated the test to use an invalid RevisionName in TrafficTarget instead of an invalid image name, and I hit the error routes.serving.knative.dev "route-not-ready-oddmxzeh" not found again https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/4594/pull-knative-serving-integration-tests/1151260492208017413
1674f30 to
8a1b852
Compare
|
Looks like the test failures aren't related to changes in this PR. @dgerd I'm not able to workaround the Another test |
| } | ||
|
|
||
| t.Logf("Waiting for Service %q to transition to Ready == false.", names.Service) | ||
| if err := v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceNotReady, "ServiceIsNotReady"); err != nil { |
There was a problem hiding this comment.
I think this line is the problem.
v1a1test.IsServiceNotReady is not checking for Service's Ready Condition to move from Unknown -> False, but is just checking that the Service is not happy. This is going to return immediately every time and not wait.
Since this check is not waiting for the Service reconciler to run it is very likely that the Route has yet to be created which is going to cause the next wait to fail. This is why adding in the 'ignoreNotFound' is working for you.
There was a problem hiding this comment.
Now this waits for [observed]generation to match and Ready to be false, which I believe is the desired end-state.
| }, | ||
| }) | ||
|
|
||
| t.Log("Creating a new Service with RouteReady == Unknown and route Ready == False") |
There was a problem hiding this comment.
When the Service is created all conditions are set to "Unknown" this is the default state. At no point in this test will RoutesReady be Unknown and Ready be False. The transition will go:
- Service.Ready = Unknown, Service.RoutesReady = Unknown
- Route gets Created
- Route.Ready = False as Revision is missing
- Service.RoutesReady = False as Route.Ready = False.
- At the same time as 4 Service.Ready = False as one of it's sub-conditions (RoutesReady) is False
|
|
||
| names := test.ResourceNames{ | ||
| Service: test.ObjectNameForTest(t), | ||
| Image: "helloworld", |
There was a problem hiding this comment.
Use test constants. Also we are looking to remove helloworld. See #4690
| Image: "helloworld", | |
| Image: test.PizzaPlanet1, |
| // track how long it took for name to get into the state checked by inState. | ||
| func WaitForRouteState(client *test.ServingAlphaClients, name string, inState func(r *v1alpha1.Route) (bool, error), desc string) error { | ||
| // ignoreNotFound indicates that we will keep polling even if the route is not found. | ||
| func WaitForRouteState(client *test.ServingAlphaClients, name string, inState func(r *v1alpha1.Route) (bool, error), desc string, ignoreNotFound bool) error { |
There was a problem hiding this comment.
Lets avoid updating this signature. See comment above for what I think is going on here.
390dac6 to
b77ad3c
Compare
| // there's no ready Revision, therefore we could receive either err message from Configuration or | ||
| // from Route. The error propagated from Route is not as informative to users comparing to the | ||
| // error propagated from Configuration, so we would like to improve the error coming from Route. | ||
| cond := r.Status.GetCondition(v1alpha1.ConfigurationConditionReady) |
There was a problem hiding this comment.
This condition is looking at "ConfigurationsReady" which should never receive Ready information from the Route. It should only contain the propagated up information from the Configuration
There was a problem hiding this comment.
b77ad3c to
3ae54d9
Compare
dgerd
left a comment
There was a problem hiding this comment.
I think we are getting close. A lot of minor suggestions on the tests. Please make the changes in your own branch rather than taking the Github suggestion directly. It will make merging and fixing references easier.
| }, | ||
| }) | ||
|
|
||
| t.Log("Creating a new Service with RouteReady == Unknown and route Ready == False") |
There was a problem hiding this comment.
| t.Log("Creating a new Service with RouteReady == Unknown and route Ready == False") | |
| t.Log("Creating a new Service with an invalid traffic target.") |
| t.Fatalf("Failed to create initial Service %q: %#v", names.Service, err) | ||
| } | ||
|
|
||
| t.Logf("Waiting for Service %q to transition to Ready == false.", names.Service) |
There was a problem hiding this comment.
| t.Logf("Waiting for Service %q to transition to Ready == false.", names.Service) | |
| t.Logf("Waiting for Service %q to transition to Ready == False.", names.Service) |
|
|
||
| t.Logf("Waiting for Service %q to transition to Ready == false.", names.Service) | ||
| if err := v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceNotReady, "ServiceIsNotReady"); err != nil { | ||
| t.Fatalf("Failed waiting for Service %q to transition to Ready false: %#v", names.Service, err) |
There was a problem hiding this comment.
| t.Fatalf("Failed waiting for Service %q to transition to Ready false: %#v", names.Service, err) | |
| t.Fatalf("Failed waiting for Service %q to transition to Ready == False: %#v", names.Service, err) |
| t.Fatalf("Failed waiting for Service %q to transition to Ready false: %#v", names.Service, err) | ||
| } | ||
|
|
||
| t.Logf("Waiting for Route %q to transition to Ready == false.", serviceresourcenames.Route(svc)) |
There was a problem hiding this comment.
Nit: You don't need the wait here or below as you are not waiting for this change to happen, but validating that when Service is reporting False that your sub-conditions are also false. I am fine keeping it as is because I understand the helpers are nice here, but we should update the text on the log.
| t.Logf("Waiting for Route %q to transition to Ready == false.", serviceresourcenames.Route(svc)) | |
| t.Logf("Validating Route %q has reconciled to Ready == False.", serviceresourcenames.Route(svc)) |
| } | ||
|
|
||
| // Wait for RouteReady to become false | ||
| t.Log("Wait for the service RouteReady to be false") |
There was a problem hiding this comment.
| t.Log("Wait for the service RouteReady to be false") | |
| t.Log("Validating Service %q has reconciled to RoutesReady == False.", names.Service) |
| t.Logf("Waiting for Route %q to transition to Ready == false.", serviceresourcenames.Route(svc)) | ||
| // Check Route is not ready | ||
| if err = v1a1test.WaitForRouteState(clients.ServingAlphaClient, serviceresourcenames.Route(svc), v1a1test.IsRouteNotReady, "RouteIsNotReady"); err != nil { | ||
| t.Fatalf("The Route %s was marked as Ready to serve traffic but it should not be: %#v", names.Route, err) |
There was a problem hiding this comment.
nit: pick either %q or %s and use consistently. Also use serviceresourcenames.Route(svc) as it is what is actually being used in the Wait function.
| t.Fatalf("The Route %s was marked as Ready to serve traffic but it should not be: %#v", names.Route, err) | |
| t.Fatalf("The Route %q was marked as Ready to serve traffic but it should not be: %#v", serviceresourcenames.Route(svc), err) |
| // Wait for RouteReady to become false | ||
| t.Log("Wait for the service RouteReady to be false") | ||
| if err = v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceNotRouteReady, "ServiceRouteReadyFalse"); err != nil { | ||
| t.Fatalf("Service RouteReady is not set to false") |
There was a problem hiding this comment.
Add service name and error. Use consistent styling.
| t.Fatalf("Service RouteReady is not set to false") | |
| t.Fatalf("Service %q was not marked RouteReady == False: %#v, names.Service, err) |
|
|
||
| // IsServiceNotReady will check the status conditions of the service and return true if the service is | ||
| // not ready. | ||
| // IsServiceNotReady will check the status conditions of the service and return false if the service is |
There was a problem hiding this comment.
Don't take take the suggestion straight from github as the line bleeds over, but use consistent style with below.
| // IsServiceNotReady will check the status conditions of the service and return false if the service is | |
| // IsServiceNotReady checks the Ready status condition of the service and returns true only if Ready is set to False. |
| return result != nil && result.Status == corev1.ConditionFalse, nil | ||
| } | ||
|
|
||
| // IsServiceNotRouteReady checks the RouteReady status of the service and return false if the service's |
There was a problem hiding this comment.
| // IsServiceNotRouteReady checks the RouteReady status of the service and return false if the service's | |
| // IsServiceNotRouteReady checks the RouteReady status of the service and returns true only if RoutesReady is set to False. |
|
|
||
| // IsServiceNotRouteReady checks the RouteReady status of the service and return false if the service's | ||
| // RouteReady is true. | ||
| func IsServiceNotRouteReady(s *v1alpha1.Service) (bool, error) { |
There was a problem hiding this comment.
nit: The condition is actually "RoutesReady" it makes more sense grammatically if the "not' is moved.
| func IsServiceNotRouteReady(s *v1alpha1.Service) (bool, error) { | |
| func IsServiceRoutesNotReady(s *v1alpha1.Service) (bool, error) { |
| func IsServiceNotReady(s *v1alpha1.Service) (bool, error) { | ||
| return s.Generation == s.Status.ObservedGeneration && !s.Status.IsReady(), nil | ||
| result := s.Status.GetCondition(v1alpha1.ServiceConditionReady) | ||
| return result != nil && result.Status == corev1.ConditionFalse, nil |
There was a problem hiding this comment.
Can we keep the observed generation check?
| // IsServiceRoutesNotReady checks the RoutesReady status of the service and returns true only if RoutesReady is set to False. | ||
| func IsServiceRoutesNotReady(s *v1alpha1.Service) (bool, error) { | ||
| result := s.Status.GetCondition(v1alpha1.ServiceConditionRoutesReady) | ||
| return result != nil && result.Status == corev1.ConditionFalse, nil |
There was a problem hiding this comment.
Add an observed generation check?
b0e84ec to
304a583
Compare
| } | ||
|
|
||
| t.Logf("Waiting for Service %q to transition to Ready == false.", names.Service) | ||
| if err := v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceNotReady, "ServiceIsNotReady"); err != nil { |
There was a problem hiding this comment.
Now this waits for [observed]generation to match and Ready to be false, which I believe is the desired end-state.
|
|
||
| t.Logf("Validating Route %q has reconciled to Ready == False.", serviceresourcenames.Route(svc)) | ||
| // Check Route is not ready | ||
| if err = v1a1test.WaitForRouteState(clients.ServingAlphaClient, serviceresourcenames.Route(svc), v1a1test.IsRouteNotReady, "RouteIsNotReady"); err != nil { |
There was a problem hiding this comment.
At this point we should be able to replace WaitFor here with Check since the Service has already observed the bad state, so we should get that same bad state.
|
|
||
| // Wait for RoutesReady to become False | ||
| t.Logf("Validating Service %q has reconciled to RoutesReady == False.", names.Service) | ||
| if err = v1a1test.WaitForServiceState(clients.ServingAlphaClient, names.Service, v1a1test.IsServiceRoutesNotReady, "ServiceRouteReadyFalse"); err != nil { |
There was a problem hiding this comment.
Same, use Check instead of WaitFor.
304a583 to
a22ba07
Compare
a22ba07 to
979204d
Compare
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgerd, taragu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Fixes #4561
Proposed Changes
Release Note
/cc @dgerd
/cc @mattmoor
/cc @vagababov
/cc @markusthoemmes