diff --git a/README.md b/README.md index ed5419ab80..70943c3bf5 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,7 @@ To build MicroShift from source and contribute to its development, refer to the - [MicroShift Host Networking Overview](./docs/network/host_networking.md) - [MicroShift Traffic Flows Overview](./docs/network/ovn_kubernetes_traffic_flows.md) - [Testing MicroShift Integration with Greenboot](./docs/greenboot_dev.md) +- [Multinode Testing Environment for MicroShift](./docs/multinode/setup.md) ## Community Community documentation sources are managed at and published on . diff --git a/docs/config/microshift-starter.ks b/docs/config/microshift-starter.ks index 51fbe6f9a5..08cb843100 100644 --- a/docs/config/microshift-starter.ks +++ b/docs/config/microshift-starter.ks @@ -5,7 +5,7 @@ text reboot # Configure network to use DHCP and activate on boot -network --bootproto=dhcp --device=link --activate --onboot=on --hostname=microshift-starter --noipv6 +network --bootproto=dhcp --device=link --activate --onboot=on # Partition disk with a 1GB boot XFS partition and a 10GB LVM volume containing system root # The remainder of the volume will be used by the CSI driver for storing data @@ -30,14 +30,17 @@ cdrom # Post install configuration %post --log=/var/log/anaconda/post-install.log --erroronfail -# Allow the default user to run sudo commands with password -echo -e 'redhat\tALL=(ALL)\tPASSWD: ALL' > /etc/sudoers.d/microshift +# Allow the default user to run sudo commands without password +echo -e 'redhat\tALL=(ALL)\tNOPASSWD: ALL' > /etc/sudoers.d/microshift # Import Red Hat public keys to allow RPM GPG check (not necessary if a system is registered) if ! subscription-manager status >& /dev/null ; then rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-* fi +# Make the KUBECONFIG from MicroShift directly available for the root user +echo -e 'export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig' >> /root/.bash_profile + # Configure systemd journal service to persist logs between boots and limit their size to 1G sudo mkdir -p /etc/systemd/journald.conf.d cat > /etc/systemd/journald.conf.d/microshift.conf < While it is possible to configure the MicroShift development environment on +> your hypervisor host, it is recommended to use a dedicated virtual machine +> to simplify the installation and configuration process. + +## Create ISO Image +Log into the `development host` and run the following commands to download +the released MicroShift RPMs to be included in the ISO image. +``` +MICROSHIFT_VERSION=4.14 +MICROSHIFT_RPMS_DIR=$(mktemp -d /tmp/microshift-rpms.XXXXXXXXXX) +OCP_REPO_NAME="rhocp-${MICROSHIFT_VERSION}-for-rhel-9-$(uname -m)-rpms" + +sudo subscription-manager repos --enable ${OCP_REPO_NAME} +sudo dnf download --downloaddir "${MICROSHIFT_RPMS_DIR}" microshift\* +chmod go+rx "${MICROSHIFT_RPMS_DIR}" +``` + +> Until the MicroShift 4.14 software is released, it is necessary to compile the +> MicroShift RPMs from the latest sources on the `development host` and copy them +> into the `${MICROSHIFT_RPMS_DIR}` directory. +> +> See the [RPM Packages](../devenv_setup.md#rpm-packages) documentation +> for more information on building MicroShift RPMs. + +Proceed by building the ISO image with your pull secret, the downloaded MicroShift +RPMS, your SSH authorized keys and the firewall configuration necessary for the +multiple nodes to access each other. +``` +cd ~/microshift/ +./scripts/image-builder/build.sh \ + -pull_secret_file ~/.pull-secret.json \ + -microshift_rpms "${MICROSHIFT_RPMS_DIR}" \ + -authorized_keys_file ~/.ssh/authorized_keys \ + -open_firewall_ports 6443:tcp,9642:tcp +rm -rf "${MICROSHIFT_RPMS_DIR}" +``` + +> The multinode configuration procedure uses SSH for copying files among the +> nodes and remotely running scripts on the primary and secondary hosts. +> Having your SSH keys authorized in the image would make the node configuration +> procedure more convenient and streamlined. + +## Install Virtual Machines +Log into the `hypervisor host` and follow the instructions described in the +[Install MicroShift for Edge](../rhel4edge_iso.md#install-microshift-for-edge) +section to create the `microshift-pri` and `microshift-sec` virtual machines +for running primary and secondary instances. + +After the virtual machines are up and running, use the `virsh` command to determine their IP addresses. +``` +sudo virsh domifaddr microshift-pri +sudo virsh domifaddr microshift-sec +``` diff --git a/docs/multinode/config_rpm.md b/docs/multinode/config_rpm.md new file mode 100644 index 0000000000..da3ff21a2c --- /dev/null +++ b/docs/multinode/config_rpm.md @@ -0,0 +1,37 @@ +# RHEL with MicroShift RPMs +Log into the `hypervisor host` and follow the instructions described in the +[Getting Started with MicroShift](../getting_started.md) document to create +and configure the `microshift-pri` and `microshift-sec` virtual machines for +running primary and secondary instances. + +> Until the MicroShift 4.14 software is released, it is necessary to compile the +> MicroShift RPMs from the latest sources on the `development host` and copy +> them to the `microshift-pri` and `microshift-sec` virtual machines. +> +> See the [RPM Packages](../devenv_setup.md#rpm-packages) documentation +> for more information on building MicroShift RPMs. +> +> Run the following command on the `microshift-pri` and `microshift-sec` +> virtual machines to upgrade the software. +> ``` +> sudo dnf localinstall -y microshift*.rpm +> ``` + +After the virtual machines are up and running, use the `virsh` command to determine their IP addresses. +``` +sudo virsh domifaddr microshift-pri +sudo virsh domifaddr microshift-sec +``` + +The multinode configuration procedure uses SSH for copying files among the +nodes and remotely running scripts on the primary and secondary hosts. +Having your SSH keys authorized in the nodes would make their configuration +procedure more convenient and streamlined. +``` +PRI_ADDR=192.168.122.118 +SEC_ADDR=192.168.122.70 + +# Use 'redhat' without quotes as a password when prompted +ssh-copy-id redhat@${PRI_ADDR} +ssh-copy-id redhat@${SEC_ADDR} +``` diff --git a/docs/multinode/setup.md b/docs/multinode/setup.md new file mode 100644 index 0000000000..dd5e95cf6c --- /dev/null +++ b/docs/multinode/setup.md @@ -0,0 +1,77 @@ +# Multinode Testing Environment for MicroShift +This document describes an opinionated, non-production setup of a two-node MicroShift +setup to facilitate running tests. + +MicroShift does **not** support a multinode mode of operation, while some of +the tests require it to run successfully. The instructions in this document are +the recommended way to configure a non-production two-node setup to be used +**only** for running tests. + +## Prerequisites +A physical `hypervisor host` with the [libvirt](https://libvirt.org/) virtualization +platform to be used for bootstrapping MicroShift primary and secondary hosts for +running tests. + +Depending on the desired MicroShift installation method, follow the instructions +to set up one of the configurations for the primary and secondary hosts: +* [RHEL with MicroShift RPMs](./config_rpm.md) +* [MicroShift on RHEL for Edge](./config_r4e.md) + +Log into the `hypervisor host` and set the environment variables that denote +primary and secondary host names and IP addresses to be used in the subsequent +commands. +``` +PRI_HOST=microshift-pri +PRI_ADDR=192.168.122.118 + +SEC_HOST=microshift-sec +SEC_ADDR=192.168.122.70 +``` + +## Configure Primary and Secondary Hosts +Run the following commands to copy the configuration script to the primary host +and run it remotely. +``` +git clone https://github.com/openshift/microshift.git ~/microshift +cd ~/microshift/ +scp -o StrictHostKeyChecking=no ./scripts/multinode/configure-pri.sh redhat@${PRI_ADDR}: +ssh redhat@${PRI_ADDR} ./configure-pri.sh "${PRI_HOST}" "${PRI_ADDR}" "${SEC_HOST}" "${SEC_ADDR}" +``` + +If the configuration script runs successfully, it prints the list of the +`kubelet` configuration files that need to be copied to the secondary host. + +Copy the `kubelet` configuration files from the primary to the secondary host. +``` +scp -3 -o StrictHostKeyChecking=no \ + redhat@${PRI_ADDR}:/home/redhat/kubelet-${SEC_HOST}.{key,crt} \ + redhat@${PRI_ADDR}:/home/redhat/kubeconfig-${PRI_HOST} \ + redhat@${SEC_ADDR}: +``` + +Run the following commands to copy the configuration script to the secondary host +and run it remotely. +``` +cd ~/microshift/ +scp -o StrictHostKeyChecking=no ./scripts/multinode/configure-sec.sh redhat@${SEC_ADDR}: +ssh redhat@${SEC_ADDR} ./configure-sec.sh "${PRI_HOST}" "${PRI_ADDR}" "${SEC_HOST}" "${SEC_ADDR}" +``` + +## Run Tests +Before running tests, make sure that the `microshift-pri` host name is resolved +and accessible from the `hypervisor host`. + +Set the `KUBECONFIG` variable using the configuration file from the primary host. +``` +export KUBECONFIG=$(mktemp /tmp/microshift-kubeconfig.XXXXXXXXXX) +scp redhat@${PRI_ADDR}:/home/redhat/kubeconfig-${PRI_HOST} ${KUBECONFIG} +``` + +Verify that the cluster has **two** nodes in the `Ready` status and wait until +all the pods are in the `Running` status. +``` +oc get nodes +watch oc get pods -A +``` + +Run your test suite of choice using the primary and secondary MicroShift instances. diff --git a/scripts/configure-multinode.sh b/scripts/configure-multinode.sh deleted file mode 100755 index b14f498476..0000000000 --- a/scripts/configure-multinode.sh +++ /dev/null @@ -1,179 +0,0 @@ -#!/bin/bash -set -e - -WORKER_NODE_HOSTNAME=${WORKER_NODE_HOSTNAME:-"worker-node"} -: "${WORKER_NODE_IP:=$(getent hosts "${WORKER_NODE_HOSTNAME}" | awk '{print $1}')}" -KUBELET_DIR=${KUBELET_DIR:-"/home/microshift/kubelet"} - -remote() { - if [ "$#" -lt 2 ]; then - echo "Usage: remote remote_host command [arg1 arg2 ...]" - return 1 - fi - - remote_host="$1" - shift - - ssh -q -t -t "microshift@${remote_host}" "$(printf "%q " "$@")" -} - -# pre-approve worker node host key -ssh-keyscan "${WORKER_NODE_HOSTNAME}" >> ~/.ssh/known_hosts - -### ON MASTER NODE - -sudo systemctl stop microshift - -# configure microshift to run in multinode mode -echo 'mtu: 1422' | sudo tee /etc/microshift/ovn.yaml -cat << EOF | sudo tee /etc/microshift/config.yaml -apiServer: - subjectAltNames: - - "10.43.0.1" -EOF - -# TODO - microshift-etcd isn't happy if microshift is also a transient service unit -# sudo systemd-run --unit=microshift-multinode \ -# --description="Multinode MicroShift" \ -# --property="WorkingDirectory=/usr/bin" \ -# --property="User=root" \ -# --property="Type=notify" \ -# --property="Delegate=yes" \ -# --property="CPUAccounting=yes" \ -# --property="BlockIOAccounting=yes" \ -# --property="MemoryAccounting=yes" \ -# --property="LimitNOFILE=1048576" \ -# --property="TimeoutStartSec=2m" \ -# /usr/bin/microshift run --multinode - -sudo cp -n /usr/lib/systemd/system/microshift.service /usr/lib/systemd/system/microshift.service.backup -cat << EOF | sudo tee /usr/lib/systemd/system/microshift.service -[Unit] -Description=MicroShift -Wants=network-online.target crio.service openvswitch.service microshift-ovs-init.service -After=network-online.target crio.service openvswitch.service microshift-ovs-init.service - -# Control shutdown order by declaring this service to start Before the kubepods.slice -# transient systemd unit; this makes system shutdown delay MicroShift shutdown until -# all the pod containers are down. This is important because some services need to talk -# to the MicroShift API during shutdown (i.e. releasing leader election locks or cleaning -# up other resources) MicroShift restart or manual stop will not stop the kubepods. -Before=kubepods.slice - -[Service] -WorkingDirectory=/usr/bin/ -ExecStart=microshift run --multinode -Restart=always -User=root -Type=notify -Delegate=yes -CPUAccounting=yes -BlockIOAccounting=yes -MemoryAccounting=yes -LimitNOFILE=1048576 -TimeoutStartSec=2m - -[Install] -WantedBy=multi-user.target -EOF - -sudo systemctl daemon-reload -sudo systemctl start microshift - -# generate serving certs for worker node's kubelet - -go install github.com/cloudflare/cfssl/cmd/...@latest -export PATH="${PATH}:${HOME}/go/bin" - -cat << EOF >csr.json -{ - "CN": "system:node:${WORKER_NODE_IP}", - "key": { - "algo": "rsa", - "size": 2048 - }, - "hosts": [ - "${WORKER_NODE_HOSTNAME}", - "${WORKER_NODE_IP}" - ], - "names": [ - { - "O": "system:nodes" - } - ] -} -EOF - -cfssl genkey csr.json | cfssljson -bare kubelet-"${WORKER_NODE_HOSTNAME}" - -cat < csr.yaml -apiVersion: certificates.k8s.io/v1 -kind: CertificateSigningRequest -metadata: - name: csr-kubelet -spec: - request: $(base64 < kubelet-"${WORKER_NODE_HOSTNAME}".csr | tr -d '\n') - signerName: kubernetes.io/kubelet-serving - usages: - - digital signature - - key encipherment - - server auth -EOF - -oc apply -f csr.yaml - -kubectl certificate approve csr-kubelet -kubectl get csr csr-kubelet -o jsonpath='{.status.certificate}' | base64 --decode > kubelet.crt - -remote "${WORKER_NODE_HOSTNAME}" mkdir -p "${KUBELET_DIR}" -scp kubelet-"${WORKER_NODE_HOSTNAME}"-key.pem microshift@"${WORKER_NODE_HOSTNAME}":"${KUBELET_DIR}"/kubelet.key -scp kubelet.crt microshift@"${WORKER_NODE_HOSTNAME}":"${KUBELET_DIR}"/kubelet.crt -sudo cp /var/lib/microshift/resources/kubeadmin/"${HOSTNAME}"/kubeconfig ~/bootstrap-kubeconfig -sudo chmod a+r ~/bootstrap-kubeconfig -scp ~/bootstrap-kubeconfig microshift@"${WORKER_NODE_HOSTNAME}":"${KUBELET_DIR}"/bootstrap-kubeconfig - -# stop firewall on master (to allow connections to port 9642 from worker-node and allow OVN to work properly) -# TODO edit firewall rules instead of stopping firewall -sudo systemctl stop firewalld; -sudo systemctl disable firewalld; - -### ON WORKER NODE - -# stop firewall on worker (to allow connections to kubelet from master) -# TODO edit firewall rules instead of stopping firewall -remote "${WORKER_NODE_HOSTNAME}" sudo systemctl stop firewalld; -remote "${WORKER_NODE_HOSTNAME}" sudo systemctl disable firewalld; - -# download kubelet -remote "${WORKER_NODE_HOSTNAME}" curl -LO https://dl.k8s.io/release/v1.26.1/bin/linux/amd64/kubelet --output-dir "${KUBELET_DIR}" -# shellcheck disable=SC2029 -echo "8b99dd73f309ca1ac4005db638e82f949ffcfb877a060089ec0e729503db8198 ${KUBELET_DIR}/kubelet" | ssh microshift@"${WORKER_NODE_HOSTNAME}" "cat > ${KUBELET_DIR}/kubelet.sha256" -remote "${WORKER_NODE_HOSTNAME}" sha256sum --check "${KUBELET_DIR}"/kubelet.sha256 -remote "${WORKER_NODE_HOSTNAME}" chmod +x "${KUBELET_DIR}"/kubelet - -# wanted at this path by OVN -remote "${WORKER_NODE_HOSTNAME}" sudo mkdir -p /var/lib/microshift/resources/kubeadmin -remote "${WORKER_NODE_HOSTNAME}" sudo ln "${KUBELET_DIR}"/bootstrap-kubeconfig /var/lib/microshift/resources/kubeadmin/kubeconfig - -# start crio & kubelet -remote "${WORKER_NODE_HOSTNAME}" sudo systemctl enable --now crio - -# disable selinux TODO remove once selinux is working properly again -remote "${WORKER_NODE_HOSTNAME}" sudo setenforce 0 - -remote "${WORKER_NODE_HOSTNAME}" sudo systemd-run --unit=kubelet --description="Kubelet" \ - --property=Environment="PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/bin" \ - "${KUBELET_DIR}"/kubelet \ - --container-runtime-endpoint=/var/run/crio/crio.sock \ - --resolv-conf=/etc/resolv.conf \ - --rotate-certificates=true \ - --kubeconfig="${KUBELET_DIR}"/kubeconfig \ - --lock-file=/var/run/lock/kubelet.lock \ - --exit-on-lock-contention \ - --fail-swap-on=false \ - --cgroup-driver=systemd \ - --tls-cert-file="${KUBELET_DIR}"/kubelet.crt \ - --tls-private-key-file="${KUBELET_DIR}"/kubelet.key \ - --bootstrap-kubeconfig="${KUBELET_DIR}"/bootstrap-kubeconfig \ - --cluster-dns=10.43.0.10 \ - --cluster-domain=cluster.local diff --git a/scripts/image-builder/build.sh b/scripts/image-builder/build.sh index 77accf66d2..d50322e4f1 100755 --- a/scripts/image-builder/build.sh +++ b/scripts/image-builder/build.sh @@ -162,6 +162,11 @@ install_caddy_rpm() { fi } +open_repo_permissions() { + find "$1" -type f -exec chmod a+r {} \; + find "$1" -type d -exec chmod a+rx {} \; +} + # Parse the command line while [ $# -gt 0 ] ; do case $1 in @@ -279,6 +284,7 @@ if [ "$(find microshift-local -name '*.rpm' | wc -l)" -eq 0 ] ; then exit 1 fi createrepo microshift-local >/dev/null +open_repo_permissions microshift-local # Determine the image version from the RPM contents RELEASE_INFO_FILE=$(find . -name 'microshift-release-info-*.rpm' | tail -1) @@ -309,6 +315,7 @@ if [ "$(find openshift-local -name '*.rpm' | wc -l)" -eq 0 ] ; then exit 1 fi createrepo openshift-local >/dev/null +open_repo_permissions openshift-local # Install prometheus process exporter install_prometheus_rpm @@ -325,6 +332,7 @@ if [ -n "${CUSTOM_RPM_FILES}" ] ; then cp "${rpm}" custom-rpms done createrepo custom-rpms >/dev/null + open_repo_permissions custom-rpms fi title "Loading sources for OpenShift and MicroShift" @@ -358,23 +366,21 @@ if ${EMBED_CONTAINERS} ; then >> blueprint_v0.0.1.toml fi -# Add open firewall ports -if [ -n "${OPEN_FIREWALL_PORTS}" ] ; then - for port in ${OPEN_FIREWALL_PORTS//,/ } ; do - cat >> blueprint_v0.0.1.toml <> blueprint_v0.0.1.toml < " + exit 1 +} + +function configure_system() { + # Disable selinux + # TODO: remove once selinux is working properly again + sudo setenforce 0 || true + + # TODO: Edit firewall rules instead of stopping firewall + sudo systemctl stop firewalld + sudo systemctl disable firewalld + + # Configure the MicroShift host name + sudo hostnamectl hostname "${PRI_NODE_HOST}" + + # Update /etc/hosts to resolve primary and secondary host names if not already resolved + if [ "$(getent ahostsv4 "${PRI_NODE_HOST}" | head -1 | awk '{print $1}')" != "${PRI_NODE_ADDR}" ] ; then + echo "${PRI_NODE_ADDR} ${PRI_NODE_HOST}" | sudo tee -a /etc/hosts &>/dev/null + fi + if [ "$(getent ahostsv4 "${SEC_NODE_HOST}" | head -1 | awk '{print $1}')" != "${SEC_NODE_ADDR}" ] ; then + echo "${SEC_NODE_ADDR} ${SEC_NODE_HOST}" | sudo tee -a /etc/hosts &>/dev/null + fi +} + +function configure_microshift() { + # Clean the current MicroShift configuration and stop the service + echo 1 | sudo microshift-cleanup-data --all + + # Run OVN initialization script + sleep 5 + sudo systemctl start microshift-ovs-init.service + + # Configure MicroShift to allow API server access by an IP address + cat </dev/null +apiServer: + subjectAltNames: + - "10.43.0.1" +EOF +} + +function wait_for_pod_ready() { + local pod_namespace=$1 + local pod_name=$2 + local service_ready=false + + echo "Waiting for MicroShift ${pod_name}@${pod_namespace} pod to be ready" + for _ in $(seq 300) ; do + if ${OC_CMD} wait --timeout=0s --for=condition=ready pod -n "${pod_namespace}" -l app="${pod_name}" &>/dev/null ; then + service_ready=true + break + fi + echo -n "." + sleep 1 + done + echo + + if ! ${service_ready} ; then + echo "Error: timed out waiting for MicroShift ${pod_name}@${pod_namespace} pod to be ready" + exit 1 + fi +} + +function generate_service_certs() { + local -r cfssl=$(mktemp /tmp/cfssl.XXXXXXXXXX) + local -r cfssl_json=$(mktemp /tmp/cfssl_json.XXXXXXXXXX) + local -r cfssl_sha=$(mktemp /tmp/cfssl_sha.XXXXXXXXXX) + local -r csr_file=$(mktemp /tmp/csrfile.XXXXXXXXXX) + local -r kubelet_csr=$(mktemp /tmp/kubelet.XXXXXXXXXX) + local cfssl_arch="" + + # Cleanup temporary files on exit + # shellcheck disable=SC2064 + trap "rm -f ${cfssl} ${cfssl_json} ${cfssl_sha} ${csr_file} ${kubelet_csr}*" EXIT + + # Install cfssl utilities + declare -A cfssl_map + case "$(uname -m)" in + x86_64) + cfssl_arch="amd64" + cfssl_map[cfssl]="b947d073e677189f8533704c44b2b1eae4042f5cefd2b8347d4d9b4c6a5008cf" + cfssl_map[cfssl_json]="d7c52a815f96ebd4fc857b012cee70b44751edabb55ae60c4b743ee09e67f4de" + ;; + aarch64) + cfssl_arch="arm64" + cfssl_map[cfssl]="453495690f9b4e811d195d1f214ae58ad281e2e50e6dc3ffb19a8c58ddbc8a51" + cfssl_map[cfssl_json]="4f68110f5a88a8b60382ff6b96008f55714636c31e5a06ab9c60855a1bd9bf47" + ;; + *) + echo "Unsupported cfssl architecture $(uname -m)" + exit 1 + esac + + curl -s -L -o "${cfssl}" "https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_${cfssl_arch}" + curl -s -L -o "${cfssl_json}" "https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_${cfssl_arch}" + cat < "${cfssl_sha}" +${cfssl_map[cfssl]} ${cfssl} +${cfssl_map[cfssl_json]} ${cfssl_json} +EOF + sha256sum --check "${cfssl_sha}" + chmod +x "${cfssl}" "${cfssl_json}" + + # Generate serving certs for secondary node's kubelet + cat < "${csr_file}" +{ +"CN": "system:node:${SEC_NODE_ADDR}", +"key": { + "algo": "rsa", + "size": 2048 +}, +"hosts": [ + "${SEC_NODE_HOST}", + "${SEC_NODE_ADDR}" +], +"names": [ + { + "O": "system:nodes" + } +] +} +EOF + + # Generate and apply the certificate + ${cfssl} genkey "${csr_file}" | ${cfssl_json} -bare "${kubelet_csr}" + + ${OC_CMD} apply -f - < "${KUBELET_HOME}/kubelet-${SEC_NODE_HOST}.crt" + cp "${kubelet_csr}-key.pem" "${KUBELET_HOME}/kubelet-${SEC_NODE_HOST}.key" + + # Copy the bootstrap kube configuration files + sudo cp "/var/lib/microshift/resources/kubeadmin/${PRI_NODE_HOST}/kubeconfig" "${KUBELET_HOME}/kubeconfig-${PRI_NODE_HOST}" + sudo chown "$(whoami)." "${KUBELET_HOME}/kubeconfig-${PRI_NODE_HOST}" +} + +# +# Main function +# +if [ $# -ne 4 ] ; then + usage +fi +PRI_NODE_HOST=$1 +PRI_NODE_ADDR=$2 +SEC_NODE_HOST=$3 +SEC_NODE_ADDR=$4 + +KUBELET_HOME="${HOME}" +if [ ! -w "${KUBELET_HOME}" ] ; then + echo "The ${KUBELET_HOME} directory is not writable" + exit 1 +fi + +# Configure system for the multinode environment +configure_system + +# Configure MicroShift for the multinode environment +configure_microshift + +# Run MicroShift in the multinode mode +# shellcheck disable=SC2024 +sudo nohup microshift run --multinode &> "$(mktemp /tmp/microshift-log.XXXXXXXXXX)" & + +# Wait for the service-ca pod to be ready +wait_for_pod_ready "openshift-service-ca" "service-ca" + +# Generate the certificates for the multinode environment +generate_service_certs + +# Print the file names to be copied to the secondary host +echo +echo "Copy the following files to the ${SEC_NODE_HOST} host" +ls -1 "${KUBELET_HOME}/kubeconfig-${PRI_NODE_HOST}" +ls -1 "${KUBELET_HOME}/kubelet-${SEC_NODE_HOST}".{key,crt} + +echo +echo "Done" diff --git a/scripts/multinode/configure-sec.sh b/scripts/multinode/configure-sec.sh new file mode 100755 index 0000000000..a5cfd3166e --- /dev/null +++ b/scripts/multinode/configure-sec.sh @@ -0,0 +1,137 @@ +#!/bin/bash +set -euo pipefail + +PRI_NODE_HOST= +PRI_NODE_ADDR= +SEC_NODE_HOST= +SEC_NODE_ADDR= + +function usage() { + echo "Usage: $(basename "$0") " + exit 1 +} + +function configure_system() { + # Disable selinux + # TODO: remove once selinux is working properly again + sudo setenforce 0 || true + + # TODO: Edit firewall rules instead of stopping firewall + sudo systemctl stop firewalld + sudo systemctl disable firewalld + + # Configure the MicroShift host name + sudo hostnamectl hostname "${SEC_NODE_HOST}" + + # Update /etc/hosts to resolve primary and secondary host names if not already resolved + if [ "$(getent ahostsv4 "${PRI_NODE_HOST}" | head -1 | awk '{print $1}')" != "${PRI_NODE_ADDR}" ] ; then + echo "${PRI_NODE_ADDR} ${PRI_NODE_HOST}" | sudo tee -a /etc/hosts &>/dev/null + fi + if [ "$(getent ahostsv4 "${SEC_NODE_HOST}" | head -1 | awk '{print $1}')" != "${SEC_NODE_ADDR}" ] ; then + echo "${SEC_NODE_ADDR} ${SEC_NODE_HOST}" | sudo tee -a /etc/hosts &>/dev/null + fi +} + +function configure_microshift() { + # Clean the current MicroShift configuration and stop the service + echo 1 | sudo microshift-cleanup-data --all + + # Run OVN initialization script + sleep 5 + sudo systemctl start microshift-ovs-init.service + + # Stop and unload the kubelet service, cleaning up its old data + if [ "$(systemctl is-active kubelet.service)" = "active" ] ; then + sudo systemctl stop --now kubelet + fi + sudo systemctl reset-failed kubelet || true + sudo pkill -9 --exact kubelet || true + sudo rm -rf /var/lib/kubelet &> /dev/null || true + + sudo find /run/systemd -name kubelet.service -exec rm -f {} \; + sudo systemctl daemon-reload +} + +function configure_kubelet() { + # Download the kubelet executable + local kube_arch="amd64" + local kube_hash="cb2845fff0ce41c400489393da73925d28fbee54cfeb7834cd4d11e622cbd3a7" + + case $(uname -m) in + x86_64) + ;; + aarch64) + kube_arch="arm64" + kube_hash="dbb09d297d924575654db38ed2fc627e35913c2d4000c34613ac6de4995457d0" + ;; + *) + echo "Unsupported kubelet architecture $(uname -m)" + exit 1 + esac + + curl -sLO "https://dl.k8s.io/release/v1.27.1/bin/linux/${kube_arch}/kubelet" --output-dir "${KUBELET_HOME}" + cat < "${KUBELET_HOME}/kubelet.sha256" +${kube_hash} ${KUBELET_HOME}/kubelet +EOF + sha256sum --check "${KUBELET_HOME}/kubelet.sha256" + chmod +x "${KUBELET_HOME}/kubelet" + + # OVN requires kubeconfig at this path + # It must be a hard link or copy to be accessed from the container + sudo mkdir -p /var/lib/microshift/resources/kubeadmin + sudo ln "${KUBELET_HOME}/kubeconfig-${PRI_NODE_HOST}" /var/lib/microshift/resources/kubeadmin/kubeconfig + # Remove the old kubelet configuration file so that it is recreated + sudo rm -f "${KUBELET_HOME}/kubeconfig" + + # Start crio & kubelet + sudo systemctl enable --now crio + sudo systemd-run --unit=kubelet --description="Kubelet" \ + --property=Environment="PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/bin" \ + "${KUBELET_HOME}"/kubelet \ + --container-runtime-endpoint=/var/run/crio/crio.sock \ + --resolv-conf=/etc/resolv.conf \ + --rotate-certificates=true \ + --kubeconfig="${KUBELET_HOME}/kubeconfig" \ + --lock-file=/var/run/lock/kubelet.lock \ + --exit-on-lock-contention \ + --fail-swap-on=false \ + --max-pods=250 \ + --cgroup-driver=systemd \ + --tls-cert-file="${KUBELET_HOME}/kubelet-${SEC_NODE_HOST}.crt" \ + --tls-private-key-file="${KUBELET_HOME}/kubelet-${SEC_NODE_HOST}.key" \ + --bootstrap-kubeconfig="${KUBELET_HOME}/kubeconfig-${PRI_NODE_HOST}" \ + --cluster-dns=10.43.0.10 \ + --cluster-domain=cluster.local +} + +# +# Main function +# +if [ $# -ne 4 ] ; then + usage +fi +PRI_NODE_HOST=$1 +PRI_NODE_ADDR=$2 +SEC_NODE_HOST=$3 +SEC_NODE_ADDR=$4 + +# Verify input file existence +KUBELET_HOME="${HOME}" +for file in "kubeconfig-${PRI_NODE_HOST}" "kubelet-${SEC_NODE_HOST}".{key,crt} ; do + if [ ! -e "${file}" ] ; then + echo "The kubelet input file '${file}' is missing" + exit 1 + fi +done + +# Configure system for the multinode environment +configure_system + +# Configure MicroShift for the multinode environment +configure_microshift + +# Configure kubelet for the multinode environment +configure_kubelet + +echo +echo "Done"