Bug 1875005: OpenStack: Don't failover api vip if loadbalanced endpoint is responding by mandre · Pull Request #2077 · openshift/machine-config-operator

mandre · 2020-09-11T08:45:42Z

This ports #1893 to OpenStack platform, together with the dependent #1124 PR, and fixes from #1508 and #1604, and a sprinkle of #1768 on top.

When using CNV or other operators that modify how the node is connected to the network, we may end up in the case where the configured VRRP interface no longer has an address in the network that it is configured to hold virtual IPs in. This patch takes a page from what we do for HAProxy and adds a monitor side car container that checks keepalived and reloads it when necessary. This ports openshift#1124 to OpenStack platform, alongside with fixes from openshift#1508 and openshift#1604.

mandre · 2020-09-11T08:45:57Z

/test e2e-openstack

We're having some issues in ci with api flakiness that appear to be related to VIP failover. In order to reduce the need to do failovers during normal circumstances, we want to leave the VIP alone as long as the loadbalanced api endpoint on the node is responding. Currently if the local api service on a node stops responding it will trigger a failover. This is unnecessary since in most cases haproxy will continue to distribute the traffic while the local api restarts. In order to get this behavior, the chk_ocp vrrp_script is split in two. The reason for this is that we still want to handle the case where all haproxy instances in the cluster go down, but at least one api service is still functional. One check looks at the haproxy endpoint only. This one has a higher weight since it's the preferred situation. The other check looks for either the haproxy endpoint or the local endpoint. This means that if haproxy is up then both checks will succeed and the node will have maximum priority regardless of the state of its local api. However, if haproxy goes down but the local api is still working then at least the priority will be higher than the minimum because the local api is responding. It was also necessary to add a check that the haproxy firewall rule is in place. It does us no good to have the loadbalancer working if traffic isn't being routed to it. This ports openshift#1893 to OpenStack platform.

mandre · 2020-09-11T12:22:54Z

/retest

mandre · 2020-09-11T12:53:06Z

/test e2e-openstack

Partial port of openshift#1768 to OpenStack platform

mandre · 2020-09-11T15:22:36Z

/test e2e-openstack

iamemilio · 2020-09-11T16:58:57Z

/retest OpenStack job timed out

openshift-ci-robot · 2020-09-11T17:42:17Z

@mandre: This pull request references Bugzilla bug 1875005, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.6.0) matches configured target release for branch (4.6.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1875005: OpenStack: Don't failover api vip if loadbalanced endpoint is responding

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Fedosin

/lgtm
/hold

openshift-ci-robot · 2020-09-11T18:17:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fedosin, mandre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~templates/common/openstack/OWNERS~~ [Fedosin,mandre]
~~templates/master/00-master/openstack/OWNERS~~ [Fedosin,mandre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mandre · 2020-09-11T19:07:44Z

/test e2e-openstack

mandre · 2020-09-12T13:15:03Z

/test e2e-openstack

mandre · 2020-09-12T16:46:08Z

/hold cancel

openshift-bot · 2020-09-12T16:57:24Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T17:10:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T18:28:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T18:41:50Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T18:54:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T19:33:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T19:59:40Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T20:38:36Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T20:52:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T21:30:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T21:56:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T22:09:28Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T22:40:41Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T22:58:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-09-12T23:01:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-09-12T23:13:51Z

@mandre: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/okd-e2e-aws	`da7a95d`	link	`/test okd-e2e-aws`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2020-09-12T23:15:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-09-13T00:44:46Z

@mandre: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#2077

Bugzilla bug 1875005 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1875005: OpenStack: Don't failover api vip if loadbalanced endpoint is responding

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mandre · 2020-09-16T07:22:04Z

Sigh... we were missing #2075.

mandre · 2020-09-21T12:44:37Z

/cherry-pick release-4.5

openshift-cherrypick-robot · 2020-09-21T12:44:43Z

@mandre: #2077 failed to apply on top of branch "release-4.5":

Applying: OpenStack: Fix keepalived dysfunction on vrrp iface change
Using index info to reconstruct a base tree...
M	templates/common/openstack/files/openstack-keepalived.yaml
Falling back to patching base and 3-way merge...
Auto-merging templates/common/openstack/files/openstack-keepalived.yaml
CONFLICT (content): Merge conflict in templates/common/openstack/files/openstack-keepalived.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 OpenStack: Fix keepalived dysfunction on vrrp iface change
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from adduarte and iamemilio September 11, 2020 08:45

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 11, 2020

mandre force-pushed the openstack-keepalived-check branch from c55bfc9 to 63a3fe5 Compare September 11, 2020 10:05

OpenStack: Add security context for ipfilters to run

da7a95d

Partial port of openshift#1768 to OpenStack platform

mandre force-pushed the openstack-keepalived-check branch from cb2b91b to da7a95d Compare September 11, 2020 15:21

mandre changed the title ~~OpenStack: Don't failover api vip if loadbalanced endpoint is responding~~ Bug 1875005: OpenStack: Don't failover api vip if loadbalanced endpoint is responding Sep 11, 2020

openshift-ci-robot added the bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. label Sep 11, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Sep 11, 2020

Fedosin approved these changes Sep 11, 2020

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 11, 2020

openshift-ci-robot assigned Fedosin Sep 11, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 11, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 12, 2020

openshift-merge-robot merged commit 7eba765 into openshift:master Sep 13, 2020

openshift-ci-robot mentioned this pull request Sep 18, 2020

Bug 1875005: OpenStack: Reverse haproxy and keepalived check timings #2091

Merged

mandre mentioned this pull request Sep 21, 2020

Bug 1881147: OpenStack: Don't failover api vip if loadbalanced endpoint is responding #2110

Merged

mandre mentioned this pull request Oct 5, 2020

[wip] vsphere - excessive vip migration #2138

Closed

Conversation

mandre commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mandre commented Sep 11, 2020

Uh oh!

mandre commented Sep 11, 2020

Uh oh!

mandre commented Sep 11, 2020

Uh oh!

mandre commented Sep 11, 2020

Uh oh!

iamemilio commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 11, 2020

Uh oh!

Fedosin left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Sep 11, 2020

Uh oh!

mandre commented Sep 11, 2020

Uh oh!

mandre commented Sep 12, 2020

Uh oh!

mandre commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-ci-robot commented Sep 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Sep 12, 2020

Uh oh!

openshift-ci-robot commented Sep 13, 2020

Uh oh!

mandre commented Sep 16, 2020

Uh oh!

mandre commented Sep 21, 2020

Uh oh!

openshift-cherrypick-robot commented Sep 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mandre commented Sep 11, 2020 •

edited

Loading

iamemilio commented Sep 11, 2020 •

edited

Loading

openshift-ci-robot commented Sep 12, 2020 •

edited

Loading