Skip to content

USHIFT-1280: Document multinode environment setup and runtime #1844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 1 commit into from
Jun 8, 2023
Merged

USHIFT-1280: Document multinode environment setup and runtime #1844

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ To build MicroShift from source and contribute to its development, refer to the
- [MicroShift Host Networking Overview](./docs/network/host_networking.md)
- [MicroShift Traffic Flows Overview](./docs/network/ovn_kubernetes_traffic_flows.md)
- [Testing MicroShift Integration with Greenboot](./docs/greenboot_dev.md)
- [Multinode Testing Environment for MicroShift](./docs/multinode/setup.md)

## Community
Community documentation sources are managed at <https://github.com/redhat-et/microshift-documentation> and published on <https://microshift.io>.
Expand Down
16 changes: 13 additions & 3 deletions docs/config/microshift-starter.ks
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ text
reboot

# Configure network to use DHCP and activate on boot
network --bootproto=dhcp --device=link --activate --onboot=on --hostname=microshift-starter --noipv6
network --bootproto=dhcp --device=link --activate --onboot=on

# Partition disk with a 1GB boot XFS partition and a 10GB LVM volume containing system root
# The remainder of the volume will be used by the CSI driver for storing data
Expand All @@ -30,14 +30,17 @@ cdrom
# Post install configuration
%post --log=/var/log/anaconda/post-install.log --erroronfail

# Allow the default user to run sudo commands with password
echo -e 'redhat\tALL=(ALL)\tPASSWD: ALL' > /etc/sudoers.d/microshift
# Allow the default user to run sudo commands without password
echo -e 'redhat\tALL=(ALL)\tNOPASSWD: ALL' > /etc/sudoers.d/microshift

# Import Red Hat public keys to allow RPM GPG check (not necessary if a system is registered)
if ! subscription-manager status >& /dev/null ; then
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-*
fi

# Make the KUBECONFIG from MicroShift directly available for the root user
echo -e 'export KUBECONFIG=/var/lib/microshift/resources/kubeadmin/kubeconfig' >> /root/.bash_profile

# Configure systemd journal service to persist logs between boots and limit their size to 1G
sudo mkdir -p /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/microshift.conf <<EOF
Expand All @@ -47,4 +50,11 @@ SystemMaxUse=1G
RuntimeMaxUse=1G
EOF

# Make sure all the Ethernet network interfaces are connected automatically
for uuid in $(nmcli -f uuid,type,autoconnect connection | awk '$2 == "ethernet" && $3 == "no" {print $1}') ; do
# Remove autoconnect option from the configuration file to keep it enabled after reboot
file=$(nmcli -f uuid,filename connection | awk -v uuid=${uuid} '$1 == uuid' | sed "s/${uuid}//g" | xargs)
sed -i '/autoconnect=.*/d' "${file}"
done

%end
4 changes: 2 additions & 2 deletions docs/getting_started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Download the OpenShift pull secret from the https://console.redhat.com/openshift

## Bootstrap MicroShift

Run the following commands to initiate the creation process of the `microshift-starter` virtual machine with 2 CPU cores, 2GB RAM and 20GB storage.
Run the following commands to initiate the creation process of the `microshift-starter` virtual machine with 2 CPU cores, 3GB RAM and 20GB storage.

```bash
VMNAME=microshift-starter
Comment thread
ggiguash marked this conversation as resolved.
Outdated
Expand All @@ -36,7 +36,7 @@ cd /var/lib/libvirt/images && \
virt-install \
--name ${VMNAME} \
--vcpus 2 \
--memory 2048 \
--memory 3072 \
Comment thread
dhellmann marked this conversation as resolved.
Outdated
--disk path=./${VMNAME}.qcow2,size=20 \
--network network=default,model=virtio \
--events on_reboot=restart \
Expand Down
63 changes: 63 additions & 0 deletions docs/multinode/config_r4e.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# MicroShift on RHEL for Edge
Review the [Install MicroShift on RHEL for Edge](../rhel4edge_iso.md)
document to understand the process of building MicroShift ISO images in the
MicroShift development environment.

## Prerequisites
A `development host` (either virtual or physical) configured according to the
instructions in the [MicroShift Development Environment](../devenv_setup.md)
document.

> While it is possible to configure the MicroShift development environment on
> your hypervisor host, it is recommended to use a dedicated virtual machine
> to simplify the installation and configuration process.

## Create ISO Image
Log into the `development host` and run the following commands to download
the released MicroShift RPMs to be included in the ISO image.
```
MICROSHIFT_VERSION=4.14
MICROSHIFT_RPMS_DIR=$(mktemp -d /tmp/microshift-rpms.XXXXXXXXXX)
OCP_REPO_NAME="rhocp-${MICROSHIFT_VERSION}-for-rhel-9-$(uname -m)-rpms"

sudo subscription-manager repos --enable ${OCP_REPO_NAME}
sudo dnf download --downloaddir "${MICROSHIFT_RPMS_DIR}" microshift\*
chmod go+rx "${MICROSHIFT_RPMS_DIR}"
```

> Until the MicroShift 4.14 software is released, it is necessary to compile the
Copy link
Copy Markdown
Contributor

@dhellmann dhellmann Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If these are the developer instructions, why don't we have them always build the RPMs instead of downloading them?

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two reasons I see for keeping the existing RPM option:

  • I'd like ostree and RPM instructions to be as symmetric as possible. If we only instruct using built RPMs, it's implying that we "require" a devenv for this work. In fact, only ostree requires devenv, and even that can be worked around if we use remote server
  • The note with go away as soon as we release v4.14. The current instructions can be reused for CNCF certification if necessary.

> MicroShift RPMs from the latest sources on the `development host` and copy them
> into the `${MICROSHIFT_RPMS_DIR}` directory.
>
> See the [RPM Packages](../devenv_setup.md#rpm-packages) documentation
> for more information on building MicroShift RPMs.

Proceed by building the ISO image with your pull secret, the downloaded MicroShift
RPMS, your SSH authorized keys and the firewall configuration necessary for the
multiple nodes to access each other.
```
cd ~/microshift/
./scripts/image-builder/build.sh \
-pull_secret_file ~/.pull-secret.json \
-microshift_rpms "${MICROSHIFT_RPMS_DIR}" \
-authorized_keys_file ~/.ssh/authorized_keys \
-open_firewall_ports 6443:tcp,9642:tcp
rm -rf "${MICROSHIFT_RPMS_DIR}"
```

> The multinode configuration procedure uses SSH for copying files among the
> nodes and remotely running scripts on the primary and secondary hosts.
> Having your SSH keys authorized in the image would make the node configuration
> procedure more convenient and streamlined.

## Install Virtual Machines
Log into the `hypervisor host` and follow the instructions described in the
[Install MicroShift for Edge](../rhel4edge_iso.md#install-microshift-for-edge)
section to create the `microshift-pri` and `microshift-sec` virtual machines
for running primary and secondary instances.

After the virtual machines are up and running, use the `virsh` command to determine their IP addresses.
```
sudo virsh domifaddr microshift-pri
sudo virsh domifaddr microshift-sec
```
37 changes: 37 additions & 0 deletions docs/multinode/config_rpm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# RHEL with MicroShift RPMs
Log into the `hypervisor host` and follow the instructions described in the
[Getting Started with MicroShift](../getting_started.md) document to create
and configure the `microshift-pri` and `microshift-sec` virtual machines for
running primary and secondary instances.

> Until the MicroShift 4.14 software is released, it is necessary to compile the
> MicroShift RPMs from the latest sources on the `development host` and copy
> them to the `microshift-pri` and `microshift-sec` virtual machines.
>
> See the [RPM Packages](../devenv_setup.md#rpm-packages) documentation
Copy link
Copy Markdown
Member

@jerpeter1 jerpeter1 Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this part of the documentation needs some fleshing out. After following the instructions up to this point, I had two working microshift systems running an older build of microshift. First, I wiped away the old microshift via echo 1 | sudo /usr/bin/microshift-cleanup-data --all". Next I built the microshift rpms from this branch on a fresh microshift-dev VM, copied them over to both microshift-pri and microshift-sec, and installed them (rpm -Uvh *.rpm - also had to dnf install greenboot to satisfy that dependency), and then rebooted both. Afterward, I re-enabled microshift and refreshed the kubeconfig as described in getting_started.md. The end result of all this I now I have the two VMs both running the new microshift build.

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I will add more information here

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the instruction - please see if this is better now.
Note that it's not necessary to clean the data before upgrading MicroShift and/or reboot afterwards.
Also, if you use dnf localinstall command instead of the rpm, the dependencies are pulled automatically.

> for more information on building MicroShift RPMs.
>
> Run the following command on the `microshift-pri` and `microshift-sec`
> virtual machines to upgrade the software.
> ```
> sudo dnf localinstall -y microshift*.rpm
> ```

After the virtual machines are up and running, use the `virsh` command to determine their IP addresses.
```
sudo virsh domifaddr microshift-pri
sudo virsh domifaddr microshift-sec
```

The multinode configuration procedure uses SSH for copying files among the
nodes and remotely running scripts on the primary and secondary hosts.
Having your SSH keys authorized in the nodes would make their configuration
procedure more convenient and streamlined.
```
PRI_ADDR=192.168.122.118
SEC_ADDR=192.168.122.70

# Use 'redhat' without quotes as a password when prompted
ssh-copy-id redhat@${PRI_ADDR}
ssh-copy-id redhat@${SEC_ADDR}
```
77 changes: 77 additions & 0 deletions docs/multinode/setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Multinode Testing Environment for MicroShift
This document describes an opinionated, non-production setup of a two-node MicroShift
setup to facilitate running tests.

MicroShift does **not** support a multinode mode of operation, while some of
the tests require it to run successfully. The instructions in this document are
the recommended way to configure a non-production two-node setup to be used
**only** for running tests.

## Prerequisites
A physical `hypervisor host` with the [libvirt](https://libvirt.org/) virtualization
platform to be used for bootstrapping MicroShift primary and secondary hosts for
running tests.

Depending on the desired MicroShift installation method, follow the instructions
to set up one of the configurations for the primary and secondary hosts:
* [RHEL with MicroShift RPMs](./config_rpm.md)
* [MicroShift on RHEL for Edge](./config_r4e.md)

Log into the `hypervisor host` and set the environment variables that denote
primary and secondary host names and IP addresses to be used in the subsequent
commands.
```
PRI_HOST=microshift-pri
PRI_ADDR=192.168.122.118

SEC_HOST=microshift-sec
SEC_ADDR=192.168.122.70
```

## Configure Primary and Secondary Hosts
Run the following commands to copy the configuration script to the primary host
and run it remotely.
```
git clone https://github.com/openshift/microshift.git ~/microshift
cd ~/microshift/
scp -o StrictHostKeyChecking=no ./scripts/multinode/configure-pri.sh redhat@${PRI_ADDR}:
ssh redhat@${PRI_ADDR} ./configure-pri.sh "${PRI_HOST}" "${PRI_ADDR}" "${SEC_HOST}" "${SEC_ADDR}"
Copy link
Copy Markdown
Member

@jerpeter1 jerpeter1 Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command fails for me:

[jerpeter@fedora microshift]$ ssh redhat@${PRI_ADDR} ./configure-pri.sh "${PRI_HOST}" "${PRI_ADDR}" "${SEC_HOST}" "${SEC_ADDR}"
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

Need to add -t to allow prompt for sudo password for redhat user.

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because you used the merged getting started kickstart, and not the one from the PR.
Please, try using https://raw.githubusercontent.com/ggiguash/microshift/cncf_docs/docs/config/microshift-starter.ks for your tests and the problem should go away.

```

If the configuration script runs successfully, it prints the list of the
`kubelet` configuration files that need to be copied to the secondary host.

Copy the `kubelet` configuration files from the primary to the secondary host.
```
scp -3 -o StrictHostKeyChecking=no \
redhat@${PRI_ADDR}:/home/redhat/kubelet-${SEC_HOST}.{key,crt} \
redhat@${PRI_ADDR}:/home/redhat/kubeconfig-${PRI_HOST} \
redhat@${SEC_ADDR}:
```

Run the following commands to copy the configuration script to the secondary host
and run it remotely.
```
cd ~/microshift/
scp -o StrictHostKeyChecking=no ./scripts/multinode/configure-sec.sh redhat@${SEC_ADDR}:
ssh redhat@${SEC_ADDR} ./configure-sec.sh "${PRI_HOST}" "${PRI_ADDR}" "${SEC_HOST}" "${SEC_ADDR}"
Copy link
Copy Markdown
Member

@jerpeter1 jerpeter1 Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs -t too.

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because you used the merged getting started kickstart, and not the one from the PR.
Please, try using https://raw.githubusercontent.com/ggiguash/microshift/cncf_docs/docs/config/microshift-starter.ks for your tests and the problem should go away.

```

## Run Tests
Before running tests, make sure that the `microshift-pri` host name is resolved
and accessible from the `hypervisor host`.

Set the `KUBECONFIG` variable using the configuration file from the primary host.
```
export KUBECONFIG=$(mktemp /tmp/microshift-kubeconfig.XXXXXXXXXX)
scp redhat@${PRI_ADDR}:/home/redhat/kubeconfig-${PRI_HOST} ${KUBECONFIG}
```

Verify that the cluster has **two** nodes in the `Ready` status and wait until
all the pods are in the `Running` status.
```
oc get nodes
Copy link
Copy Markdown
Member

@jerpeter1 jerpeter1 Jun 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accessing the microshift cluster from the VM host fails if microshift-pri is not resolvable from the VM host.

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify what you mean by "VM host"?
The current document describes access from the hypervisor. Also, the primary and secondary VM hosts should have gone through the configure-pri.sh and configure-sec.sh procedures, which make sure both names are resolvable locally on VMs.

Copy link
Copy Markdown
Contributor Author

@ggiguash ggiguash Jun 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the hypervisor, these names should be resolved by libvirt.
Anyway, there is a comment here asking the user to take care of the name resolution.

watch oc get pods -A
```

Run your test suite of choice using the primary and secondary MicroShift instances.
Loading