Skip to content

USHIFT-1363: microshift multi-image test job updates #40720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 2 commits into from
Jul 4, 2023
Merged

USHIFT-1363: microshift multi-image test job updates #40720

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8ae1606
increase vm timeouts from 5 to 8 minutes
dhellmann Jun 30, 2023
e3d6888
USHIFT-1363: start using in-repo harness for running test scenarios
dhellmann Jul 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions ...try/openshift/microshift/e2e/metal-tests/openshift-microshift-e2e-metal-tests-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,21 @@ EOF
# If more tests are to be run in parallel the code should go in here #
######################################################################

# Copy the scenario settings from the remote host back here so the
# test runner has access to them.
scp -r "${INSTANCE_PREFIX}:/home/${HOST_USER}/microshift/_output/test-images/scenario-info" "${ARTIFACT_DIR}"

# Run the scenario tests, if the phase script exists
# (we can clean this up after the main PR lands)
cd /microshift/test || true
if [ -f ./bin/ci_phase_test.sh ]; then
./bin/ci_phase_test.sh
fi

# Copy the scenario settings from the remote host here *again* to
# include the log files and other outputs.
scp -r "${INSTANCE_PREFIX}:/home/${HOST_USER}/microshift/_output/test-images/scenario-info" "${ARTIFACT_DIR}"

# VM mapping
# e2e: vm0
# cncf: vm1, vm2
Expand Down
3 changes: 2 additions & 1 deletion ...egistry/openshift/microshift/infra/aws/ec2/openshift-microshift-infra-aws-ec2-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,9 +221,10 @@ Resources:
FromPort: 6443
Comment thread
dhellmann marked this conversation as resolved.
Outdated
ToPort: 6453
CidrIp: 0.0.0.0/0
# Range for all dynamic port-forwarding for scenario tests, see boot phase for use
- IpProtocol: tcp
FromPort: 7000
ToPort: 7010
ToPort: 8000
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 30000
Expand Down
28 changes: 23 additions & 5 deletions ...istry/openshift/microshift/infra/iso-boot/openshift-microshift-infra-iso-boot-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,11 @@ IP_ADDRESS="$(cat "${SHARED_DIR}"/public_address)"
HOST_USER="$(cat "${SHARED_DIR}"/ssh_user)"
INSTANCE_PREFIX="${HOST_USER}@${IP_ADDRESS}"

# May need to update openshift-microshift-infra-aws-ec2-commands.sh to
# open other ports if changing these settings.
SSH_EXTERNAL_BASE_PORT=7000
API_EXTERNAL_BASE_PORT=6000
API_EXTERNAL_BASE_PORT=7100
LB_EXTERNAL_BASE_PORT=7500

echo "Using Host $IP_ADDRESS"

Expand Down Expand Up @@ -42,7 +45,6 @@ do
set -xeuo pipefail

cd ~/microshift
./scripts/devenv-builder/manage-vm.sh config

ISO_FILE=\$(find ~/microshift/_output/image-builder -name "*.iso")
sudo bash -c " \
Expand All @@ -60,7 +62,7 @@ sudo bash -c " \
"

VM_IP=\$(./scripts/devenv-builder/manage-vm.sh ip -n ${VM_NAME})
timeout 5m bash -c "until ssh -oStrictHostKeyChecking=accept-new redhat@\${VM_IP} 'echo hello'; do sleep 5; done"
timeout 8m bash -c "until ssh -oStrictHostKeyChecking=accept-new redhat@\${VM_IP} 'echo hello'; do sleep 5; done"

cat << EOF2 > /tmp/config.yaml
apiServer:
Expand Down Expand Up @@ -93,8 +95,8 @@ set -xeuo pipefail
cd ~/microshift
VM_IP=\$(./scripts/devenv-builder/manage-vm.sh ip -n ${VM_NAME})

timeout 5m bash -c "until ssh redhat@\${VM_IP} hostname; do sleep 5; done"
timeout 5m bash -c "date; until ssh redhat@\${VM_IP} \"sudo systemctl status greenboot-healthcheck | grep 'active (exited)'\"; do sleep 5; done; date"
timeout 8m bash -c "until ssh redhat@\${VM_IP} hostname; do sleep 5; done"
timeout 8m bash -c "date; until ssh redhat@\${VM_IP} \"sudo systemctl status greenboot-healthcheck | grep 'active (exited)'\"; do sleep 5; done; date"
Copy link
Copy Markdown
Contributor

@pacevedom pacevedom Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me wonder if we should use beefier VMs (since there is enough resources) to shorten everything, and have a small "minimum requirements test" that runs fast enough to validate the 2 core, 2G are still ok.

Copy link
Copy Markdown
Contributor Author

@dhellmann dhellmann Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That could make sense, yeah. I'm not sure if creating the VMs is limited by CPU, memory, or I/O.


# Setup external access with port forwarding to allow running commands and tests from the CI container.
sudo /sbin/iptables -I FORWARD -o virbr0 -p tcp -d \${VM_IP} --dport 6443 -j ACCEPT
Expand Down Expand Up @@ -127,3 +129,19 @@ do
echo "${SSH_EXTERNAL_PORT}" > "${SHARED_DIR}/vm_ssh_port_${i}"
echo "redhat" > "${SHARED_DIR}/vm_user_${i}"
done

# Install the settings for the scenario runner. The ssh keys have
# already been copied into place in the iso-build step.
SETTINGS_FILE="${SHARED_DIR}/scenario_settings.sh"
cat <<EOF >"${SETTINGS_FILE}"
PUBLIC_IP=${IP_ADDRESS}
SSH_PUBLIC_KEY=~/.ssh/id_rsa.pub
SSH_PRIVATE_KEY=~/.ssh/id_rsa
EOF
scp "${SETTINGS_FILE}" "${INSTANCE_PREFIX}:/home/${HOST_USER}/microshift/test/"

# Run the in-repo ci phase script to create the VMs for the test scenarios.
ssh "${INSTANCE_PREFIX}" "/home/${HOST_USER}/microshift/test/bin/ci_phase_iso_boot.sh ${API_EXTERNAL_PORT} ${SSH_EXTERNAL_PORT} ${LB_EXTERNAL_BASE_PORT}"

# Copy the scenario info directory from the host to our artifact directory.
scp -r "${INSTANCE_PREFIX}:/home/${HOST_USER}/microshift/_output/test-images/scenario-info" "${ARTIFACT_DIR}"
37 changes: 22 additions & 15 deletions ...try/openshift/microshift/infra/iso-build/openshift-microshift-infra-iso-build-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ cat <<EOF > /tmp/iso.sh
#!/bin/bash
set -xeuo pipefail

if ! sudo subscription-manager status >&/dev/null; then
sudo subscription-manager register \
--org="\$(cat /tmp/subscription-manager-org)" \
--activationkey="\$(cat /tmp/subscription-manager-act-key)"
fi

chmod 0755 ~
mkdir ~/rpms
tar -xf /tmp/rpms.tar -C ~/rpms
Expand All @@ -33,28 +39,29 @@ cp /tmp/ssh-publickey ~/.ssh/id_rsa.pub
cp /tmp/ssh-privatekey ~/.ssh/id_rsa
chmod 0400 ~/.ssh/id_rsa*

sudo subscription-manager register \
--org="$(cat /var/run/rhsm/subscription-manager-org)" \
--activationkey="$(cat /var/run/rhsm/subscription-manager-act-key)"
# Set up the pull secret in the expected location
export PULL_SECRET="\${HOME}/.pull-secret.json"
cp /tmp/pull-secret "\${PULL_SECRET}"

cd ~/microshift
# Get firewalld and repos in place. Use scripts to get the right repos
# for each branch.
./scripts/devenv-builder/configure-vm.sh --no-build-deps --force-firewall /tmp/pull-secret
./scripts/image-builder/configure.sh
./scripts/image-builder/build.sh -pull_secret_file /tmp/pull-secret -microshift_rpms ~/rpms -authorized_keys_file /tmp/ssh-publickey -open_firewall_ports 6443:tcp

./test/bin/ci_phase_iso_build.sh

./scripts/image-builder/build.sh -pull_secret_file "\${PULL_SECRET}" -microshift_rpms ~/rpms -authorized_keys_file ~/.ssh/id_rsa.pub -open_firewall_ports 6443:tcp
Comment on lines +48 to +50
Copy link
Copy Markdown
Contributor

@pacevedom pacevedom Jul 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note for the future, as it is part of #40429, the rpms.tar we are copying into the metal host are the same rpms being built there. The -microshift_rpms option could be removed so that it takes the ones it just built.

Copy link
Copy Markdown
Contributor Author

@dhellmann dhellmann Jul 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can do that in the PR where you remove the build step from this pipeline?

EOF
chmod +x /tmp/iso.sh

tar czf /tmp/microshift.tgz /microshift

scp \
/rpms.tar \
/tmp/iso.sh \
"${CLUSTER_PROFILE_DIR}/pull-secret" \
${CLUSTER_PROFILE_DIR}/ssh-privatekey \
${CLUSTER_PROFILE_DIR}/ssh-publickey \
/tmp/microshift.tgz \
"${INSTANCE_PREFIX}:/tmp"
/rpms.tar \
/tmp/iso.sh \
/var/run/rhsm/subscription-manager-org \
/var/run/rhsm/subscription-manager-act-key \
"${CLUSTER_PROFILE_DIR}/pull-secret" \
"${CLUSTER_PROFILE_DIR}/ssh-privatekey" \
"${CLUSTER_PROFILE_DIR}/ssh-publickey" \
/tmp/microshift.tgz \
"${INSTANCE_PREFIX}:/tmp"

ssh "${INSTANCE_PREFIX}" "/tmp/iso.sh"