Bug 1881147: OpenStack: Don't failover api vip if loadbalanced endpoint is responding by mandre · Pull Request #2110 · openshift/machine-config-operator

mandre · 2020-09-21T15:54:30Z

Manual cherry-pick of #2077 to release-4.5.

When using CNV or other operators that modify how the node is connected to the network, we may end up in the case where the configured VRRP interface no longer has an address in the network that it is configured to hold virtual IPs in. This patch takes a page from what we do for HAProxy and adds a monitor side car container that checks keepalived and reloads it when necessary. This ports openshift#1124 to OpenStack platform, alongside with fixes from openshift#1508 and openshift#1604.

openshift-ci-robot · 2020-09-21T15:54:35Z

@mandre: This pull request references Bugzilla bug 1881147, which is invalid:

expected dependent Bugzilla bug 1875005 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1881147: OpenStack: Don't failover api vip if loadbalanced endpoint is responding

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mandre · 2020-09-21T15:56:15Z

/test e2e-openstack

mandre · 2020-09-23T09:05:45Z

/test e2e-openstack

We're having some issues in ci with api flakiness that appear to be related to VIP failover. In order to reduce the need to do failovers during normal circumstances, we want to leave the VIP alone as long as the loadbalanced api endpoint on the node is responding. Currently if the local api service on a node stops responding it will trigger a failover. This is unnecessary since in most cases haproxy will continue to distribute the traffic while the local api restarts. In order to get this behavior, the chk_ocp vrrp_script is split in two. The reason for this is that we still want to handle the case where all haproxy instances in the cluster go down, but at least one api service is still functional. One check looks at the haproxy endpoint only. This one has a higher weight since it's the preferred situation. The other check looks for either the haproxy endpoint or the local endpoint. This means that if haproxy is up then both checks will succeed and the node will have maximum priority regardless of the state of its local api. However, if haproxy goes down but the local api is still working then at least the priority will be higher than the minimum because the local api is responding. It was also necessary to add a check that the haproxy firewall rule is in place. It does us no good to have the loadbalancer working if traffic isn't being routed to it. This ports openshift#1893 to OpenStack platform.

Partial port of openshift#1768 to OpenStack platform

Since we moved to keepalived healthchecking against haproxy, we want haproxy to handle most failures so the VIP doesn't have to move. However, previously the time it took for haproxy to recognize an outage on a node was longer than it was for keepalived, which resulted in the VIP moving before haproxy removed the failing backend. This change makes the haproxy interval 1 second, so it should notice outages in 2 seconds or less (because it has a fall value of 2). The keepalived interval is changed to 2, which means it will detect failures in 2 to 4 seconds (also a fall value of 2). This means haproxy should deal with api outages before keepalived does and allow the VIP to stay on the same node. This ports openshift#2075 to OpenStack platform.

mandre · 2020-09-23T10:36:21Z

/test e2e-openstack

mandre · 2020-09-23T11:26:02Z

/bugzilla refresh

openshift-ci-robot · 2020-09-23T11:26:11Z

@mandre: This pull request references Bugzilla bug 1881147, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.z) matches configured target release for branch (4.5.z)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1875005 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
dependent Bugzilla bug 1875005 targets the "4.6.0" release, which is one of the valid target releases: 4.6.0, 4.6.z
bug has dependents

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mandre · 2020-09-23T16:19:04Z

/test e2e-openstack

Fedosin · 2020-09-24T07:20:56Z

/lgtm

openshift-ci-robot · 2020-09-24T07:21:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fedosin, mandre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~templates/common/openstack/OWNERS~~ [Fedosin,mandre]
~~templates/master/00-master/openstack/OWNERS~~ [Fedosin,mandre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-09-24T07:30:46Z

/retest