Remove instanceType option `m4` from AWS IPI default deployment by mtulio · Pull Request #5162 · openshift/installer

mtulio · 2021-08-20T13:49:07Z

The deployments in AWS region us-east-1 (biggest region) is being impacted with a cluster more expensive and without the benefits of Nitro Instances (faster CPU, network, EBS, etc). The evaluation process on that Region choose the oldest instanceType when the evaluation process of preferred zone - it expects that the instance is offered in all zones. The region us-east-1 has 6 zones, and one us-east-1e is not offering Nitro Instances. Looking the zones across all regions[1], this is the unique exception.

Initially I've studied two other options, there is:
A) Create a restricted list of zones for each region
B) Choose the preferred type from the Types when a type has available in >= 3 zones (N*Masters)

But when looking the availability of m5 across all zones[1], remove the m4 from the list seems to be a better option.

Further we could improve the capacity election when m5 instance is not available, but I guess it will be a more complex implementation, and this current change could be bring to stakeholders and short term benefit.

Please see the details of study in SPLAT-254

Stakeholders impacted:

all users running default AWS IPI

[1] Filters by region of describe_instance_type_offerings used by installer to choose the preferred instance type.

                count(m5.large)  count(m4.large)  count(m5VSm4)
region                                                         
af-south-1                    3                0              3
ap-east-1                     3                0              3
ap-northeast-1                3                3              0
ap-northeast-2                4                2              2
ap-south-1                    3                2              1
ap-southeast-1                3                3              0
ap-southeast-2                3                3              0
ca-central-1                  3                2              1
eu-central-1                  3                3              0
eu-north-1                    3                0              3
eu-south-1                    3                0              3
eu-west-1                     3                3              0
eu-west-2                     3                3              0
eu-west-3                     3                0              3
me-south-1                    3                0              3
sa-east-1                     3                2              1
us-east-1                     5                6             -1  <-- here
us-east-2                     3                3              0
us-west-1                     2                2              0
us-west-2                     4                3              1

The deployments in AWS region us-east-1 (biggest region) is being impacted with a cluster more expensive and without the benefits of Nitro Instances (faster CPU, network, EBS, etc). The evaluation process on that Region choose the oldest instanceType when the evaluation process of preferred zone - it expects that the instance is offered in all zones. The region us-east-1 has 6 zones, and one us-east-1e is not offering Nitro Instances. Looking the zones across all regions[1], this is the unique exception. Regarding the performance and costs, the nitro is a better choice and it should not impact the HA on that specific zone that will use 5, instead of 6 zones.

jstuever · 2021-08-24T07:33:57Z

There may be an issue here if m5 does not exist in all six of us-east-1 zones. The PreferredInstanceType function ensures that the selected type exists in all selected zones. For the control plane machines, the selected zones defaults to all zones in the region. Therefore, with the most basic install-config.yaml, I expect this change is always breaking for us-east-1 region. The current e2e-aws test succeeded in us-west-1, which is not the targeted region.

jstuever · 2021-08-24T07:34:03Z

/hold
Need to test for this manually.

jstuever · 2021-08-24T07:34:17Z

/test e2e-aws

jstuever · 2021-08-24T07:42:56Z

Also keeping in mind our CI sets the default to m5 and selects only the zones where this type exists. As these values are patched into the install-config.yaml, we will probably not see the issue I mentioned above in CI.

AWS recently launched the instance m6i, new generation of Nitro instance for general purpose (m5). ATM the instance is available in a few regions, but the fallback will work well for m5 that was tested on PR openshift#5162

jstuever · 2021-08-26T22:37:25Z

/hold cancel
I appear to have been wrong. I tested this manually using 6 masters. It picked up the m5.xlarge type without issue and simply ignored the problematic zone reusing the first one a second time.

mtulio · 2021-09-03T13:56:23Z

I tested this manually using 6 masters. It picked up the m5.xlarge type without issue and simply ignored the problematic zone reusing the first one a second time.

@jstuever yes, the getInstanceTypeZoneInfo() will not return the "missing AZ" if the query set (types) has only m5.

rvanderp3 · 2021-09-30T13:52:48Z

/retest

jstuever

/lgtm
/approve

openshift-ci · 2021-10-06T16:27:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jstuever

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/types/aws/OWNERS~~ [jstuever]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-10-06T17:09:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T17:45:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T18:21:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T18:33:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T19:21:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T19:45:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T19:57:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T20:09:08Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T20:45:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-10-06T21:17:39Z

@mtulio: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-workers-rhel7	`d767935`	link	false	`/test e2e-aws-workers-rhel7`
ci/prow/e2e-aws-single-node	`d767935`	link	false	`/test e2e-aws-single-node`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-10-06T21:21:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T21:45:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T22:45:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T23:09:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T23:21:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-06T23:45:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-07T00:57:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-07T01:21:08Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-10-07T01:45:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci Bot requested review from jstuever and patrickdillon August 20, 2021 13:50

mtulio changed the title ~~refact: remove instance option m4 from IPI~~ refact: remove instance option m4 from AWS IPI default deployment Aug 20, 2021

mtulio changed the title ~~refact: remove instance option m4 from AWS IPI default deployment~~ remove instance option m4 from AWS IPI default deployment Aug 20, 2021

mtulio changed the title ~~remove instance option m4 from AWS IPI default deployment~~ Remove instanceType option m4 from AWS IPI default deployment Aug 20, 2021

mtulio mentioned this pull request Aug 20, 2021

[installer][aws] adjusting description of new default instance type for control planes openshift/openshift-docs#35681

Closed

mtulio force-pushed the SPLAT-254e3 branch from 9c80579 to d767935 Compare August 20, 2021 17:49

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 24, 2021

mtulio mentioned this pull request Aug 24, 2021

aws-type: changing preferred instance type to m6i, instead of m5/m4 mtulio/installer#3

Closed

3 tasks

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 26, 2021

jstuever approved these changes Oct 6, 2021

View reviewed changes

openshift-ci Bot assigned jstuever Oct 6, 2021

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 6, 2021

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 6, 2021

openshift-merge-robot merged commit dd37fc0 into openshift:master Oct 7, 2021

mtulio deleted the SPLAT-254e3 branch October 7, 2021 17:47

Conversation

mtulio commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jstuever commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jstuever commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jstuever commented Aug 24, 2021

Uh oh!

jstuever commented Aug 24, 2021

Uh oh!

jstuever commented Aug 26, 2021

Uh oh!

mtulio commented Sep 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rvanderp3 commented Sep 30, 2021

Uh oh!

jstuever left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-ci Bot commented Oct 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 6, 2021

Uh oh!

openshift-bot commented Oct 7, 2021

Uh oh!

openshift-bot commented Oct 7, 2021

Uh oh!

openshift-bot commented Oct 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mtulio commented Aug 20, 2021 •

edited

Loading

jstuever commented Aug 24, 2021 •

edited

Loading

jstuever commented Aug 24, 2021 •

edited

Loading

mtulio commented Sep 3, 2021 •

edited

Loading

openshift-ci Bot commented Oct 6, 2021 •

edited

Loading