OCPBUGS-24012: Add templates for gc_thresh sysctls by jmencak · Pull Request #4048 · openshift/machine-config-operator

jmencak · 2023-11-29T07:43:01Z

- What I did

The net.ipv[46].neigh.default.gc_thresh[1-3] have traditionally been set by the Node Tuning Operator. With OpenShift 4.13, NTO is now an optional operator. These sysctls tune kernel's ARP cache. For large scale clusters, the default kernel parameters are too low and result in issues such as rhbz#1384746. In the absence of NTO and customer clusters getting larger, these sysctls need to be set elsewhere.

Setting these values early on cluster boot (prior to container startups) will also prevent issues such as OCPBUGS-24012.

Fixes: OCPBUGS-24012

- How to verify it

Check the file /etc/sysctl.d/gc-thresh.conf exists in a new cluster with:

net.ipv4.neigh.default.gc_thresh1=8192
net.ipv4.neigh.default.gc_thresh2=32768
net.ipv4.neigh.default.gc_thresh3=65536
net.ipv6.neigh.default.gc_thresh1=8192
net.ipv6.neigh.default.gc_thresh2=32768
net.ipv6.neigh.default.gc_thresh3=65536

- Description for the changelog
Add /etc/sysctl.d/gc-thresh.conf file to bump ARP cache limits.

The net.ipv[46].neigh.default.gc_thresh[1-3] have traditionally been set by the Node Tuning Operator. With OpenShift 4.13, NTO is now an optional operator. These sysctls tune kernel's ARP cache. For large scale clusters, the default kernel parameters are too low and result in issues such as rhbz#1384746. In the absence of NTO and customer clusters getting larger, these sysctls need to be set elsewhere. Setting these values early on cluster boot (prior to container startups) will also prevent issues such as OCPBUGS-24012. Fixes: OCPBUGS-24012

openshift-ci-robot · 2023-11-29T07:43:07Z

@jmencak: This pull request references Jira Issue OCPBUGS-24012, which is invalid:

expected the bug to target the "4.15.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did

The net.ipv[46].neigh.default.gc_thresh[1-3] have traditionally been set by the Node Tuning Operator. With OpenShift 4.13, NTO is now an optional operator. These sysctls tune kernel's ARP cache. For large scale clusters, the default kernel parameters are too low and result in issues such as rhbz#1384746. In the absence of NTO and customer clusters getting larger, these sysctls need to be set elsewhere.

Setting these values early on cluster boot (prior to container startups) will also prevent issues such as OCPBUGS-24012.

Fixes: OCPBUGS-24012

- How to verify it

Check the file /etc/sysctl.d/gc-thresh.conf exists in a new cluster with:
net.ipv4.neigh.default.gc_thresh1=8192
net.ipv4.neigh.default.gc_thresh2=32768
net.ipv4.neigh.default.gc_thresh3=65536
net.ipv6.neigh.default.gc_thresh1=8192
net.ipv6.neigh.default.gc_thresh2=32768
net.ipv6.neigh.default.gc_thresh3=65536
- Description for the changelog
Add /etc/sysctl.d/gc-thresh.conf file to bump ARP cache limits.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jmencak · 2023-11-29T07:44:27Z

/jira refresh

openshift-ci-robot · 2023-11-29T07:44:33Z

@jmencak: This pull request references Jira Issue OCPBUGS-24012, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.15.0) matches configured target version for branch (4.15.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (liqcui@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jmencak · 2023-11-29T11:06:46Z

/retest

yuqi-zhang

Am I correct in understanding that for most clusters, we don't hit these thresholds, so it's mostly a safe change?

Can approve from the MCO code standpoint but it would be best if someone could test whether this is working properly

jmencak · 2023-11-30T19:20:54Z

Am I correct in understanding that for most clusters, we don't hit these thresholds, so it's mostly a safe change?

That's correct. These sysctls are not needed on regular-sized clusters. The issues arise with 100s of nodes. With OpenShiftSDN, I also remember large number of routes were also causing the same issues, but I cannot reproduce the "large route number" issue with OVNKubernetes, so it seems only large number of nodes issue.

jmencak · 2023-12-04T08:25:26Z

@dagrayvid , you're most familiar with this given the work on optional NTO and #3440

Can I have a review of this, please? Thank you.

dagrayvid · 2023-12-04T14:20:11Z

It looks good to me.

I wonder if we want to standardize on a file name scheme for the sysctl .conf files in this directory? I see one that is named <name>-conf.yaml, two that are named <name>.yaml, and three that are named <name>.conf.yaml.

It's not urgent to fix this in this PR.

jmencak · 2023-12-04T15:05:29Z

It looks good to me.

I wonder if we want to standardize on a file name scheme for the sysctl .conf files in this directory? I see one that is named <name>-conf.yaml, two that are named <name>.yaml, and three that are named <name>.conf.yaml.

It's not urgent to fix this in this PR.

Good point, @dagrayvid. Not sure on what would the MCO team want to standardize on. At this point, I'm aiming to get the fix in for 4.15.0 and we're running out of time. The cleanup should be a part of another PR in my view.

dagrayvid · 2023-12-04T16:58:24Z

/lgtm

openshift-ci · 2023-12-04T17:02:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dagrayvid, jmencak, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-12-04T19:09:55Z

/retest-required

Remaining retests: 0 against base HEAD cc572a0 and 2 for PR HEAD 5588797 in total

yuqi-zhang · 2023-12-05T03:22:33Z

The naming schema fortunately doesn't affect actual processing at all, so that's a minor point we can clean up in a later PR.

There's a known hypershift failure, once that's fixed this should be able to merge.

openshift-ci-robot · 2023-12-05T03:53:19Z

/retest-required

Remaining retests: 0 against base HEAD f691dcd and 1 for PR HEAD 5588797 in total

jmencak · 2023-12-05T07:05:26Z

/retest

jmencak · 2023-12-05T10:19:41Z

/test e2e-hypershift

jmencak · 2023-12-06T08:02:44Z

/retest

openshift-ci-robot · 2023-12-06T23:36:49Z

/retest-required

Remaining retests: 0 against base HEAD 57c1b4e and 0 for PR HEAD 5588797 in total

openshift-ci-robot · 2023-12-07T04:56:15Z

/hold

Revision 5588797 was retested 3 times: holding

jmencak · 2023-12-07T07:16:03Z

/hold cancel
/retest

openshift-ci · 2023-12-07T09:15:13Z

@jmencak: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2023-12-07T09:19:09Z

@jmencak: Jira Issue OCPBUGS-24012: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#4048

Jira Issue OCPBUGS-24012 has been moved to the MODIFIED state.

Details

In response to this:

- What I did

The net.ipv[46].neigh.default.gc_thresh[1-3] have traditionally been set by the Node Tuning Operator. With OpenShift 4.13, NTO is now an optional operator. These sysctls tune kernel's ARP cache. For large scale clusters, the default kernel parameters are too low and result in issues such as rhbz#1384746. In the absence of NTO and customer clusters getting larger, these sysctls need to be set elsewhere.

Setting these values early on cluster boot (prior to container startups) will also prevent issues such as OCPBUGS-24012.

Fixes: OCPBUGS-24012

- How to verify it

Check the file /etc/sysctl.d/gc-thresh.conf exists in a new cluster with:
net.ipv4.neigh.default.gc_thresh1=8192
net.ipv4.neigh.default.gc_thresh2=32768
net.ipv4.neigh.default.gc_thresh3=65536
net.ipv6.neigh.default.gc_thresh1=8192
net.ipv6.neigh.default.gc_thresh2=32768
net.ipv6.neigh.default.gc_thresh3=65536
- Description for the changelog
Add /etc/sysctl.d/gc-thresh.conf file to bump ARP cache limits.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2023-12-07T12:16:54Z

[ART PR BUILD NOTIFIER]

This PR has been included in build openshift-proxy-pull-test-container-v4.16.0-202312071150.p0.gdb97b28.assembly.stream for distgit openshift-proxy-pull-test.
All builds following this will include this PR.

openshift-merge-robot · 2023-12-08T05:15:10Z

Fix included in accepted release 4.15.0-0.nightly-2023-12-07-225558

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Nov 29, 2023

openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Nov 29, 2023

openshift-ci Bot requested review from cgwalters and yuqi-zhang November 29, 2023 07:45

yuqi-zhang approved these changes Nov 30, 2023

View reviewed changes

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 30, 2023

openshift-ci Bot assigned dagrayvid Dec 4, 2023

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Dec 4, 2023

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 7, 2023

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 7, 2023

openshift-merge-bot Bot merged commit db97b28 into openshift:master Dec 7, 2023

jmencak deleted the 4.15-gc-thresh branch December 7, 2023 09:36

Conversation

jmencak commented Nov 29, 2023

Uh oh!

openshift-ci-robot commented Nov 29, 2023

Uh oh!

jmencak commented Nov 29, 2023

Uh oh!

openshift-ci-robot commented Nov 29, 2023

Uh oh!

jmencak commented Nov 29, 2023

Uh oh!

yuqi-zhang left a comment

Choose a reason for hiding this comment

Uh oh!

jmencak commented Nov 30, 2023

Uh oh!

jmencak commented Dec 4, 2023

Uh oh!

dagrayvid commented Dec 4, 2023

Uh oh!

jmencak commented Dec 4, 2023

Uh oh!

dagrayvid commented Dec 4, 2023

Uh oh!

openshift-ci Bot commented Dec 4, 2023

Uh oh!

openshift-ci-robot commented Dec 4, 2023

Uh oh!

yuqi-zhang commented Dec 5, 2023

Uh oh!

openshift-ci-robot commented Dec 5, 2023

Uh oh!

jmencak commented Dec 5, 2023

Uh oh!

jmencak commented Dec 5, 2023

Uh oh!

jmencak commented Dec 6, 2023

Uh oh!

openshift-ci-robot commented Dec 6, 2023

Uh oh!

openshift-ci-robot commented Dec 7, 2023

Uh oh!

jmencak commented Dec 7, 2023

Uh oh!

openshift-ci Bot commented Dec 7, 2023

Uh oh!

openshift-ci-robot commented Dec 7, 2023

Uh oh!

openshift-bot commented Dec 7, 2023

Uh oh!

openshift-merge-robot commented Dec 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants