Skip to content

ci/installer-5327 : aws-ec2 - change instance to m6i #23031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mtulio wants to merge 3 commits into from
Closed

ci/installer-5327 : aws-ec2 - change instance to m6i #23031

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0941789
ci/installer-5327 : aws-ec2 - change instance to m6i
mtulio Oct 26, 2021
bb6ba29
ci/installer-5327 : aws-ec2 - supporting preferred instance as the in…
mtulio Oct 28, 2021
8bd275b
ci/installer-5327 : fix SIZE_VARIANT to be compliant with ref
mtulio Dec 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 100 additions & 27 deletions ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,58 +5,133 @@ set -o errexit
set -o pipefail

# TODO: move to image
curl -L https://github.com/mikefarah/yq/releases/download/3.3.0/yq_linux_amd64 -o /tmp/yq && chmod +x /tmp/yq
curl -sL https://github.com/mikefarah/yq/releases/download/3.3.0/yq_linux_amd64 -o /tmp/yq && chmod +x /tmp/yq

export AWS_SHARED_CREDENTIALS_FILE="${CLUSTER_PROFILE_DIR}/.awscred"

declare -g MASTER_FAMILY
declare -g MASTER_TYPE

CONFIG="${SHARED_DIR}/install-config.yaml"
REGION="${LEASED_RESOURCE}"

expiration_date=$(date -d '8 hours' --iso=minutes --utc)
existing_zones_setting=$(/tmp/yq r "${CONFIG}" 'controlPlane.platform.aws.zones')

function join_by { local IFS="$1"; shift; echo "$*"; }

REGION="${LEASED_RESOURCE}"
# Size definition (ipi-conf-aws-ref.yaml): ref.env["SIZE_VARIANT"].documentation
function get_master_size {
master_size=xlarge
if [[ "${SIZE_VARIANT:-}" == "xlarge" ]]; then
master_size=8xlarge
elif [[ "${SIZE_VARIANT:-}" == "large" ]]; then
master_size=4xlarge
elif [[ "${SIZE_VARIANT:-}" == "compact" ]]; then
master_size=2xlarge
fi
echo ${master_size}
}

function set_MASTER_TYPE {
MASTER_TYPE="${MASTER_FAMILY}.$(get_master_size)"
}

function choose_preferred_type {
local type_preferred
local type_backup
local zones_cnt_pref
local zones_cnt_backup

type_preferred="${1}.$(get_master_size)"
type_backup="${2}.$(get_master_size)"

if [[ ${existing_zones_setting} == "" ]]; then
zones_cnt_pref=$(aws ec2 describe-instance-type-offerings \
--region "${REGION}" \
--location-type availability-zone \
--filters Name=instance-type,Values=\"${type_preferred}\" \
--query 'InstanceTypeOfferings[].Location' \
--output json | jq -r '. | length')
zones_cnt_backup=$(aws ec2 describe-instance-type-offerings \
--region "${REGION}" \
--location-type availability-zone \
--filters Name=instance-type,Values=\"${type_backup}\" \
--query 'InstanceTypeOfferings[].Location' \
--output json | jq -r '. | length')
else
zones_expr=$(echo "${existing_zones_setting}" |jq -r '. | join("|")')
zones_cnt_pref=$(aws ec2 describe-instance-type-offerings \
--region "${REGION}" \
--location-type availability-zone \
--filters Name=instance-type,Values=\"${type_preferred}\" \
--query 'InstanceTypeOfferings[].Location' \
--output json \
| jq -r ".[] |select(.? | match(\"${zones_expr}\"))" \
| wc -l)
zones_cnt_backup=$(aws ec2 describe-instance-type-offerings \
--region "${REGION}" \
--location-type availability-zone \
--filters Name=instance-type,Values=\"${type_backup}\" \
--query 'InstanceTypeOfferings[].Location' \
--output json \
| jq -r ".[] |select(.? | match(\"${zones_expr}\"))" \
| wc -l)
fi

if [[ ${zones_cnt_pref} -ge ${zones_cnt_backup} ]]; then
Copy link
Copy Markdown
Contributor

@staebler staebler Nov 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In practice this probably ends up the same, but I would have expected that the preferred family be used as long as there was at least one zone (or perhaps 2 zones) offering the preferred family.

Copy link
Copy Markdown
Contributor Author

@mtulio mtulio Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iiuc, I am not sure if it will ends up the same, it will help to select the correct instance on the region which does not support the preferred one (see the original PR for the table of availability):

output of $ cat install-config.yaml |yq -r '. | (.platform.aws, .controlPlane.platform.aws)',

for LEASED_RESOURCE=sa-east-1:

{
  "region": "sa-east-1",
  "userTags": {
    "expirationDate": "2021-12-01T22:34+00:00"
  }
}
{
  "zones": [],
  "type": "m5.2xlarge"
}

for LEASED_RESOURCE=us-east-1:

{
  "region": "us-east-1",
  "userTags": {
    "expirationDate": "2021-12-01T22:35+00:00"
  }
}
{
  "zones": [],
  "type": "m6i.2xlarge"
}

Copy link
Copy Markdown
Contributor Author

@mtulio mtulio Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but I would have expected that the preferred family be used as long as there was at least one zone (or perhaps 2 zones)

if it's not a problem, we could change to something like:

if [[ ${zones_cnt_pref} -ge 1 ]]; then

wdyt?

Copy link
Copy Markdown
Contributor Author

@mtulio mtulio Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating the latest availability:

                count(m5.xlarge)  count(m6i.xlarge)  diff(m5_m6i)
region                                                           
af-south-1                     3                  0             3
ap-east-1                      3                  0             3
ap-northeast-1                 3                  2             1
ap-northeast-2                 4                  3             1
ap-northeast-3                 3                  0             3
ap-south-1                     3                  3             0
ap-southeast-1                 3                  3             0
ap-southeast-2                 3                  2             1
ca-central-1                   3                  0             3
eu-central-1                   3                  3             0
eu-north-1                     3                  0             3
eu-south-1                     3                  0             3
eu-west-1                      3                  3             0
eu-west-2                      3                  0             3
eu-west-3                      3                  3             0
me-south-1                     3                  0             3
sa-east-1                      3                  2             1
us-east-1                      5                  5             0
us-east-2                      3                  3             0
us-west-1                      2                  2             0
us-west-2                      6                  4             2

Copy link
Copy Markdown
Contributor

@staebler staebler Mar 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you switch this logic to check if there are enough zones for the preferred or backup type, then make sure that the zones with availability matches exactly when the zones are explicitly specified in the install config.

MASTER_FAMILY="${1}"
else
MASTER_FAMILY="${2}"
fi
}

# BootstrapInstanceType gets its value from pkg/types/aws/defaults/platform.go
architecture="amd64"
arch_instance_type=m5

# choose the preferred instance type depending of the availability on the
# region. If the availability is equal, the preferred will be chosen, otherwise
# the backup will be.
# See more in PreferredInstanceType() pkg/asset/machines/aws/instance_types.go
instance_preferred=m6i
instance_backup=m5
arch_instance_type=${instance_preferred}

if [[ "${CLUSTER_TYPE}" == "aws-arm64" ]]; then
architecture="arm64"
arch_instance_type=m6g
MASTER_FAMILY="${arch_instance_type}"
else
choose_preferred_type "${instance_preferred}" "${instance_backup}"
arch_instance_type=${MASTER_FAMILY}
fi

set_MASTER_TYPE
BOOTSTRAP_NODE_TYPE=${arch_instance_type}.large

workers=3
if [[ "${SIZE_VARIANT}" == "compact" ]]; then
if [[ "${SIZE_VARIANT:-}" == "compact" ]]; then
workers=0
fi
master_type=null
if [[ "${SIZE_VARIANT}" == "xlarge" ]]; then
master_type=${arch_instance_type}.8xlarge
elif [[ "${SIZE_VARIANT}" == "large" ]]; then
master_type=${arch_instance_type}.4xlarge
elif [[ "${SIZE_VARIANT}" == "compact" ]]; then
master_type=${arch_instance_type}.2xlarge
fi

# Generate working availability zones from the region
mapfile -t AVAILABILITY_ZONES < <(aws --region "${REGION}" ec2 describe-availability-zones | jq -r '.AvailabilityZones[] | select(.State == "available") | .ZoneName' | sort -u)
# Generate availability zones with OpenShift Installer required instance types

if [[ "${COMPUTE_NODE_TYPE}" == "${BOOTSTRAP_NODE_TYPE}" && "${COMPUTE_NODE_TYPE}" == "${master_type}" ]]; then ## all regions are the same
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${master_type}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 1 ' | awk '{print $2}')
elif [[ "${master_type}" == null && "${COMPUTE_NODE_TYPE}" == null ]]; then ## two null regions
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${master_type}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 1 ' | awk '{print $2}')
elif [[ "${master_type}" == null || "${COMPUTE_NODE_TYPE}" == null ]]; then ## one null region
if [[ "${BOOTSTRAP_NODE_TYPE}" == "${COMPUTE_NODE_TYPE}" || "${BOOTSTRAP_NODE_TYPE}" == "${master_type}" || "${master_type}" == "${COMPUTE_NODE_TYPE}" ]]; then ## "one null region and duplicates"
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${master_type}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 1 ' | awk '{print $2}')
if [[ "${COMPUTE_NODE_TYPE}" == "${BOOTSTRAP_NODE_TYPE}" && "${COMPUTE_NODE_TYPE}" == "${MASTER_TYPE}" ]]; then ## all regions are the same
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${MASTER_TYPE}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 1 ' | awk '{print $2}')
elif [[ "${MASTER_TYPE}" == null && "${COMPUTE_NODE_TYPE}" == null ]]; then ## two null regions
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${MASTER_TYPE}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 1 ' | awk '{print $2}')
elif [[ "${MASTER_TYPE}" == null || "${COMPUTE_NODE_TYPE}" == null ]]; then ## one null region
if [[ "${BOOTSTRAP_NODE_TYPE}" == "${COMPUTE_NODE_TYPE}" || "${BOOTSTRAP_NODE_TYPE}" == "${MASTER_TYPE}" || "${MASTER_TYPE}" == "${COMPUTE_NODE_TYPE}" ]]; then ## "one null region and duplicates"
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${MASTER_TYPE}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 1 ' | awk '{print $2}')
else ## "one null region and no duplicates"
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${master_type}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 2 ' | awk '{print $2}')
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${MASTER_TYPE}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 2 ' | awk '{print $2}')
fi
elif [[ "${BOOTSTRAP_NODE_TYPE}" == "${COMPUTE_NODE_TYPE}" || "${BOOTSTRAP_NODE_TYPE}" == "${master_type}" || "${master_type}" == "${COMPUTE_NODE_TYPE}" ]]; then ## duplicates regions with no null region
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${master_type}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 2 ' | awk '{print $2}')
elif [[ "${BOOTSTRAP_NODE_TYPE}" != "${COMPUTE_NODE_TYPE}" && "${COMPUTE_NODE_TYPE}" != "${master_type}" ]]; then # three different regions
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${master_type}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 3 ' | awk '{print $2}')
elif [[ "${BOOTSTRAP_NODE_TYPE}" == "${COMPUTE_NODE_TYPE}" || "${BOOTSTRAP_NODE_TYPE}" == "${MASTER_TYPE}" || "${MASTER_TYPE}" == "${COMPUTE_NODE_TYPE}" ]]; then ## duplicates regions with no null region
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${MASTER_TYPE}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 2 ' | awk '{print $2}')
elif [[ "${BOOTSTRAP_NODE_TYPE}" != "${COMPUTE_NODE_TYPE}" && "${COMPUTE_NODE_TYPE}" != "${MASTER_TYPE}" ]]; then # three different regions
mapfile -t INSTANCE_ZONES < <(aws --region "${REGION}" ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values="${BOOTSTRAP_NODE_TYPE}","${MASTER_TYPE}","${COMPUTE_NODE_TYPE}" | jq -r '.InstanceTypeOfferings[].Location' | sort | uniq -c | grep ' 3 ' | awk '{print $2}')
fi
# Generate availability zones based on these 2 criterias
mapfile -t ZONES < <(echo "${AVAILABILITY_ZONES[@]}" "${INSTANCE_ZONES[@]}" | sed 's/ /\n/g' | sort -R | uniq -d)
Expand All @@ -65,8 +140,6 @@ MAX_ZONES_COUNT="${#ZONES[@]}"
# Save max zones count information to ${SHARED_DIR} for use in other scenarios
echo "${MAX_ZONES_COUNT}" >> "${SHARED_DIR}/maxzonescount"

existing_zones_setting=$(/tmp/yq r "${CONFIG}" 'controlPlane.platform.aws.zones')

if [[ ${existing_zones_setting} == "" ]]; then
ZONES_COUNT=${ZONES_COUNT:-2}
ZONES=("${ZONES[@]:0:${ZONES_COUNT}}")
Expand Down Expand Up @@ -101,7 +174,7 @@ controlPlane:
name: master
platform:
aws:
type: ${master_type}
type: ${MASTER_TYPE}
compute:
- architecture: ${architecture}
name: worker
Expand Down