Skip to content

Bug 1875005: OpenStack: Reverse haproxy and keepalived check timings#2091

Merged
openshift-merge-robot merged 1 commit intoopenshift:masterfrom
mandre:openstack-keepalived-check
Sep 18, 2020
Merged

Bug 1875005: OpenStack: Reverse haproxy and keepalived check timings#2091
openshift-merge-robot merged 1 commit intoopenshift:masterfrom
mandre:openstack-keepalived-check

Conversation

@mandre
Copy link
Copy Markdown
Member

@mandre mandre commented Sep 16, 2020

Since we moved to keepalived healthchecking against haproxy, we want
haproxy to handle most failures so the VIP doesn't have to move.
However, previously the time it took for haproxy to recognize an
outage on a node was longer than it was for keepalived, which resulted
in the VIP moving before haproxy removed the failing backend.

This change makes the haproxy interval 1 second, so it should notice
outages in 2 seconds or less (because it has a fall value of 2).
The keepalived interval is changed to 2, which means it will detect
failures in 2 to 4 seconds (also a fall value of 2). This means
haproxy should deal with api outages before keepalived does and
allow the VIP to stay on the same node.

This ports
#2075 to
OpenStack platform.

@mandre
OpenStack: Reverse haproxy and keepalived check timings
4b9265a 
Since we moved to keepalived healthchecking against haproxy, we want
haproxy to handle most failures so the VIP doesn't have to move.
However, previously the time it took for haproxy to recognize an
outage on a node was longer than it was for keepalived, which resulted
in the VIP moving before haproxy removed the failing backend.

This change makes the haproxy interval 1 second, so it should notice
outages in 2 seconds or less (because it has a fall value of 2).
The keepalived interval is changed to 2, which means it will detect
failures in 2 to 4 seconds (also a fall value of 2). This means
haproxy should deal with api outages before keepalived does and
allow the VIP to stay on the same node.

This ports
openshift#2075 to
OpenStack platform.
@openshift-ci-robot openshift-ci-robot added bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Sep 16, 2020
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Sep 16, 2020

@mandre: This pull request references Bugzilla bug 1875005, which is invalid:

  • expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1875005: OpenStack: Reverse haproxy and keepalived check timings

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 16, 2020
@mandre
Copy link
Copy Markdown
Member Author

mandre commented Sep 16, 2020

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Sep 16, 2020
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Sep 16, 2020

@mandre: This pull request references Bugzilla bug 1875005, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mandre
Copy link
Copy Markdown
Member Author

mandre commented Sep 16, 2020

/test e2e-openstack

1 similar comment
@mandre
Copy link
Copy Markdown
Member Author

mandre commented Sep 16, 2020

/test e2e-openstack

@iamemilio
Copy link
Copy Markdown

iamemilio commented Sep 16, 2020

/lgtm
/hold until passing e2e-openstack

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 16, 2020
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 16, 2020
@pierreprinetti
Copy link
Copy Markdown
Contributor

pierreprinetti commented Sep 16, 2020

/retest

Fedosin
Fedosin approved these changes Sep 16, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@Fedosin Fedosin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Sep 16, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fedosin, iamemilio, mandre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mandre
Copy link
Copy Markdown
Member Author

mandre commented Sep 17, 2020

/test e2e-openstack

@mandre
Copy link
Copy Markdown
Member Author

mandre commented Sep 17, 2020

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 17, 2020
@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 17, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

5 similar comments
@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 17, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 17, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 17, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 17, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 17, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 18, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Copy Markdown
Contributor

openshift-bot commented Sep 18, 2020

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Sep 18, 2020

@mandre: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/okd-e2e-aws 4b9265a link /test okd-e2e-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 8879b5e into openshift:master Sep 18, 2020
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Sep 18, 2020

@mandre: All pull requests linked via external trackers have merged:

Bugzilla bug 1875005 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1875005: OpenStack: Reverse haproxy and keepalived check timings

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mandre mandre mentioned this pull request Oct 5, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iamemilio iamemilio Awaiting requested review from iamemilio

1 more reviewer

@Fedosin Fedosin Fedosin approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@iamemilio iamemilio

@Fedosin Fedosin

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-urgent Referenced Bugzilla bug's severity is urgent for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@mandre @openshift-ci-robot @iamemilio @pierreprinetti @openshift-bot @Fedosin @openshift-merge-robot