diff --git a/README.md b/README.md index 0b24027e6d..47148c1576 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ To build MicroShift from source and contribute to its development, refer to the - [MicroShift Design](./docs/design.md) - [MicroShift Development Environment on RHEL 8.x](./docs/devenv_rhel8.md) +- [Install MicroShift on RHEL for Edge](./docs/rhel4edge_iso.md) - [OpenShift CI for MicroShift](./docs/openshift_ci.md) ## Community diff --git a/docs/devenv_rhel8.md b/docs/devenv_rhel8.md index 8171451e16..94f44e2766 100644 --- a/docs/devenv_rhel8.md +++ b/docs/devenv_rhel8.md @@ -1,3 +1,5 @@ +# MicroShift Development Environment on RHEL 8.x + ## Create Development Virtual Machine Start by downloading the RHEL 8.6 or above ISO image from the https://developers.redhat.com/products/rhel/download location. > RHEL 9.x operating system is not currently supported. @@ -129,119 +131,12 @@ Note that this command only stops the MicroShift executable. To perform full cle ./hack/cleanup.sh ``` -## Build RHEL for Edge Installer ISO -Log into the development virtual machine with the `microshift` user credentials. - -Follow the instructions in the RPM Packages section to build the binary and create MicroShift RPM packages. - -The scripts for building the installer are located in the `scripts/image-builder` subdirectory. - -### Prerequisites -Execute the `configure.sh` script to install the tools necessary for building the installer image. - -Download the OpenShift pull secret from the https://console.redhat.com/openshift/downloads#tool-pull-secret page and save it into the `~microshift/pull-secret.txt` file. - -Make sure there is more than 20GB of free disk space necessary for the build artifacts. Run the following command to free the space if necessary. -```bash -./scripts/image-builder/cleanup.sh -full -``` - -Note that the command deletes various user and system data, including: -- The `scripts/image-builder/_builds` directory containing image build artifacts -- MicroShift `ostree` server container and all the unused container images -- All the Image Builder jobs are canceled and deleted -- Project-specific Image Builder sources are deleted -- The user `~/.cache` directory is deleted to clean Golang cache -- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files - -### Building Installer -It is recommended to execute the partial cleanup procedure (without `-full` argument) before each build to reclaim cached disk space from previous builds. -```bash -./scripts/image-builder/cleanup.sh -``` -Note that the command deletes various user and system data, including: -- MicroShift `ostree` server container -- All the Image Builder jobs are canceled -- Project-specific Image Builder sources are deleted -- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files - -Run the build script without arguments to see its usage. -```bash -./scripts/image-builder/build.sh -Usage: build.sh <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] - -pull_secret_file Path to a file containing the OpenShift pull secret - -ostree_server_name Name or IP address of the OS tree server (default: ) - -Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret. -``` - -Continue by running the build script with the pull secret file argument and wait until build process is finished. It may take over 30 minutes to complete a full build cycle. -```bash -./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt -``` -The script performs the following tasks: -- Check for minimum 10GB of available disk space -- Set up local MicroShift RPM repository using `packaging/rpm/_rpmbuild` directory contents as a source -- Set up local OpenShift RPM repository using public `rhocp-${OCP_VERSION}-for-rhel-8-${ARCH}-rpms` repository necessary for CRI-O and OpenShift client package installation -- Configure Image Builder to use the local MicroShift and OpenShift RPM repositories for image builds -- Run Image Builder to create an edge container image containing all the MicroShift binaries and dependencies -- Start a local `ostree` server with the above image -- Run Image Builder to create an edge installer image comprised of RHEL for Edge OS and MicroShift binaries -- Rebuild the installer image with the `kickstart.ks` file for performing various OS setup when the host is first booted -- Perform partial cleanup procedure to reclaim cached disk space - -The artifact of the build is the `scripts/image-builder/_builds/microshift-installer.${ARCH}.iso` bootable RHEL for Edge OS image. - -## Install MicroShift for Edge -Log into the host machine using your user credentials. The remainder of this section describes how to install a virtual machine running RHEL for Edge OS containing MicroShift binaries. - -Start by copying the installer image from the development virtual machine to the host file system. -```bash -sudo scp microshift@microshift-dev:/home/microshift/microshift/scripts/image-builder/_builds/microshift-installer.*.iso /var/lib/libvirt/images/ -``` - -Run the following commands to create a virtual machine using the installer image. -```bash -VMNAME="microshift-edge" -sudo -b bash -c " \ -cd /var/lib/libvirt/images/ && \ -virt-install \ - --name ${VMNAME} \ - --vcpus 2 \ - --memory 3096 \ - --disk path=./${VMNAME}.qcow2,size=20 \ - --network network=default,model=virtio \ - --os-type generic \ - --events on_reboot=restart \ - --cdrom ./microshift-installer.$(uname -i).iso \ -" -``` - -Watch the OS console to see the progress of the installation, waiting until the machine is rebooted and the login prompt appears. - -Note that it may be more convenient to access the machine using SSH. Run the following command to get its IP address and use it to remotely connect to the system. -```bash -sudo virsh domifaddr microshift-edge -``` - -Log into the system using `redhat:redhat` credentials and run the following commands to configure the MicroShift access. -```bash -mkdir ~/.kube -sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config -``` - -Finally, check if the MicroShift is up and running by executing `oc` commands. -```bash -oc get cs -oc get pods -A -``` - ## Quick Development and Edge Testing Cycle -In the development environment, it is practical to build and run MicroShift executable as described in the [Build MicroShift](#build-microshift) and [Run MicroShift Executable](#run-microshift-executable) sections above. However, it is also necessary to have a convenient technique to occasionally test the system in a RHEL for Edge environment. +During the development cycle, it is practical to build and run MicroShift executable as demonstrated in the [Build MicroShift](#build-microshift) and [Run MicroShift Executable](#run-microshift-executable) sections above. However, it is also necessary to have a convenient technique for testing the system in a setup resembling the production environment. Such an environment can be created in a virtual machine as described in the [Install MicroShift on RHEL for Edge](./rhel4edge_iso.md) document. -The MicroShift Installer ISO build procedure takes over 30 minutes to complete. Once a RHEL for Edge virtual machine is created, it is running a version of MicroShift with the latest changes. +Once a RHEL for Edge virtual machine is created, it is running a version of MicroShift with the latest changes. When MicroShift code is updated and the executable file is rebuilt with the new changes, the updates need to be installed on RHEL for Edge OS. -The remainder of this section describes a simple procedure for replacing the MicroShift executable file on an existing RHEL for Edge OS installation. +Since it takes a long time to create a new RHEL for Edge installer ISO and deploy it on a virtual machine, the remainder of this section describes a simple technique for replacing the MicroShift executable file on an existing RHEL for Edge OS installation. ### Configuring ostree Log into the RHEL for Edge machine using `redhat:redhat` credentials. Run the following command for configuring the ostree to allow transient overlays on top of the /usr directory. @@ -260,7 +155,7 @@ ssh-keygen ssh-copy-id redhat@microshift-edge ``` -Rebuild the MicroShift executable as described in the [Build MicroShift](#build-microshift) section and run the following commands to copy, cleanup and restart the new service on the RHEL for Edge system. +Rebuild the MicroShift executable as described in the [Build MicroShift](#build-microshift) section and run the following commands to copy, cleanup, replace and restart the new service on the RHEL for Edge system. ```bash scp ~/microshift/microshift redhat@microshift-edge: ssh redhat@microshift-edge ' \ diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md new file mode 100644 index 0000000000..8330829bd7 --- /dev/null +++ b/docs/rhel4edge_iso.md @@ -0,0 +1,212 @@ +# Install MicroShift on RHEL for Edge +To test MicroShift in a setup similar to the production environment, it is necessary to create a RHEL for Edge ISO installer with all the necessary components preloaded on the image. + +## Build RHEL for Edge Installer ISO +Log into the development virtual machine with the `microshift` user credentials. +> The development machine configuration guidelines can be found in the [MicroShift Development Environment on RHEL 8.x](./devenv_rhel8.md) document. + +Follow the instructions in the [RPM Packages](./devenv_rhel8.md#rpm-packages) section to create MicroShift RPM packages. + +The scripts for building the installer are located in the `scripts/image-builder` subdirectory. + +### Prerequisites +Execute the `scripts/image-builder/configure.sh` script to install the tools necessary for building the installer image. + +Download the OpenShift pull secret from the https://console.redhat.com/openshift/downloads#tool-pull-secret page and save it into the `~microshift/pull-secret.txt` file. + +Make sure there is more than 20GB of free disk space necessary for the build artifacts. Run the following command to free the space if necessary. +```bash +./scripts/image-builder/cleanup.sh -full +``` + +Note that the command deletes various user and system data, including: +- The `scripts/image-builder/_builds` directory containing image build artifacts +- MicroShift `ostree` server container and all the unused container images +- All the Image Builder jobs are canceled and deleted +- Project-specific Image Builder sources are deleted +- The user `~/.cache` and `/tmp/containers` directory contents are deleted to clean various cache files +- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files + +### Building Installer +It is recommended to execute the partial cleanup procedure (without `-full` argument) before each build to reclaim cached disk space from previous builds. +```bash +./scripts/image-builder/cleanup.sh +``` +Note that the command deletes various user and system data, including: +- MicroShift `ostree` server container +- All the Image Builder jobs are canceled +- Project-specific Image Builder sources are deleted +- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files + +Run the build script without arguments to see its usage. +```bash +./scripts/image-builder/build.sh +Usage: build.sh <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-custom_rpms /path/to/file1.rpm,...,/path/to/fileN.rpm] + -pull_secret_file Path to a file containing the OpenShift pull secret + -ostree_server_name Name or IP address of the OS tree server (default: ) + -custom_rpms Path to one or more comma-separated RPM packages to be included in the image + +Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret. +``` + +Continue by running the build script with the pull secret file argument and wait until build process is finished. It may take over 30 minutes to complete a full build cycle. +```bash +./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt +``` +The script performs the following tasks: +- Check for minimum 10GB of available disk space +- Set up local MicroShift RPM repository using `packaging/rpm/_rpmbuild` directory contents as a source +- Set up local OpenShift RPM repository using public `rhocp-${OCP_VERSION}-for-rhel-8-${ARCH}-rpms` repository necessary for CRI-O and OpenShift client package installation +- Configure Image Builder to use the local MicroShift and OpenShift RPM repositories for image builds +- Run Image Builder to create an edge container image containing all the MicroShift binaries and dependencies +- Start a local `ostree` server with the above image +- Run Image Builder to create an edge installer image comprised of RHEL for Edge OS and MicroShift binaries +- Rebuild the installer image with the `kickstart.ks` file for performing various OS setup when the host is first booted +- Perform partial cleanup procedure to reclaim cached disk space + +The artifact of the build is the `scripts/image-builder/_builds/microshift-installer.${ARCH}.iso` bootable RHEL for Edge OS image. + +### Offline Containers +The `scripts/image-builder/build.sh` script supports a special mode for including user-specific RPM files into the generated ISO. The remainder of this section demonstrates how to generate container image RPMs for MicroShift and include them in the installation ISO. + +When the container images required by MicroShift are preinstalled, `CRI-O` does not attempt to pull them when MicroShift service is first started, saving network bandwidth and avoiding external network connections. +>If user workloads depend on additional container images, the respective RPMs need to be created by the user separately. + +Start by running the `packaging/rpm/make-microshift-images-rpm.sh` script without arguments to see its usage. +```bash +$ ./packaging/rpm/make-microshift-images-rpm.sh +Usage: + make-microshift-images-rpm.sh rpm + make-microshift-images-rpm.sh srpm + make-microshift-images-rpm.sh copr + +pull_secret: Path to a file containing the OpenShift pull secret +architectures: One or more RPM architectures +rpm_mock_target: Target for building RPMs inside a chroot (e.g. 'rhel-8-x86_64') +copr_repo: Target Fedora Copr repository name (e.g. '@redhat-et/microshift-containers') + +Notes: + - The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret + - Use 'x86_64:amd64' or 'aarch64:arm64' as an architecture value + - See /etc/mock/*.cfg for possible RPM mock target values +``` + +Run the script in the `rpm` mode to pull the images required by MicroShift and generate the RPMs including those image data. +```bash +./packaging/rpm/make-microshift-images-rpm.sh rpm ~/pull-secret.txt x86_64:amd64 rhel-8-x86_64 +``` + +If the procedure runs successfully, the RPM artifacts can be found in the `packaging/rpm/paack-result` directory. +```bash +$ ls -1 ~/microshift/packaging/rpm/paack-result/*.rpm +/home/microshift/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.src.rpm +/home/microshift/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm +``` + +Finally, run the build script with the `-custom_rpms` argument to include the specified container image RPMs into the generated ISO. +```bash +./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt -custom_rpms ~/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm +``` +> If user-specific container images need to be included into the ISO, multiple comma-separated RPM files can be specified as the `-custom_rpms` argument value. + +When executed in this mode, the `scripts/image-builder/build.sh` script performs an extra step to set up a custom RPM repository with one or more specified RPM files as a source. These RPMs are then appended to the blueprint so that they are installed when the operating system boots for the first time. + +## Install MicroShift for Edge +Log into the host machine using your user credentials. The remainder of this section describes how to install a virtual machine running RHEL for Edge OS containing MicroShift binaries. + +Start by copying the installer image from the development virtual machine to the host file system. +```bash +sudo scp microshift@microshift-dev:/home/microshift/microshift/scripts/image-builder/_builds/microshift-installer.*.iso /var/lib/libvirt/images/ +``` + +Run the following commands to create a virtual machine using the installer image. +```bash +VMNAME="microshift-edge" +sudo -b bash -c " \ +cd /var/lib/libvirt/images/ && \ +virt-install \ + --name ${VMNAME} \ + --vcpus 2 \ + --memory 3096 \ + --disk path=./${VMNAME}.qcow2,size=20 \ + --network network=default,model=virtio \ + --os-type generic \ + --events on_reboot=restart \ + --cdrom ./microshift-installer.$(uname -i).iso \ +" +``` + +Watch the OS console to see the progress of the installation, waiting until the machine is rebooted and the login prompt appears. + +Note that it may be more convenient to access the machine using SSH. Run the following command to get its IP address and use it to remotely connect to the system. +```bash +sudo virsh domifaddr microshift-edge +``` + +Log into the system using `redhat:redhat` credentials and run the following commands to configure MicroShift access. +```bash +mkdir ~/.kube +sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config +``` + +Finally, check if MicroShift is up and running by executing `oc` commands. +```bash +oc get cs +oc get pods -A +``` + +### Offline Mode +It may sometimes be necessary to install a virtual machine that does not have access to the Internet. For instance, such a configuration can be used for testing the [Offline Containers](#offline-containers) or any other setup that needs to work without the Internet access. + +Create a new isolated `libvirt` network configuration by running the following commands. +```bash +NETCONFIG_FILE=/tmp/isolated-network.xml +cat > $NETCONFIG_FILE < + isolated + + + gateway + + + + + + + + +EOF +sudo virsh net-define $NETCONFIG_FILE +sudo virsh net-start isolated +sudo virsh net-autostart isolated +``` + +Follow the instruction in the [Install MicroShift for Edge](#install-microshift-for-edge) section to install a new virtual machine using the `isolated` network configuration. +> When running the `virt-install` command, specify the `--network network=isolated,model=virtio` option to select the `isolated` network configuration. + +After the virtual machine is created, log into the system and verify that the Internet is not accessible. +```bash +$ curl -I redhat.com +curl: (6) Could not resolve host: redhat.com +``` + +Make sure that `CRI-O` has access to all the container images required by MicroShift. +```bash +$ sudo crictl images +IMAGE TAG IMAGE ID SIZE +k8s.gcr.io/pause 3.6 6270bb605e12e 690kB +quay.io/coreos/flannel v0.14.0 8522d622299ca 68.9MB +quay.io/kubevirt/hostpath-provisioner v0.8.0 7cbc61ff04c89 180MB +quay.io/microshift/flannel-cni v0.14.0 4324dc7a1ffa5 8.12MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev 334363f37666a 401MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev eb9d5c9681cd5 376MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev 60f52af9fc4ba 413MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev ef1c6b04ebe2a 415MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev a538d5965f4fc 458MB +``` + +Finally, check if MicroShift is up and running by executing `oc` commands. +```bash +oc get cs +oc get pods -A +``` diff --git a/scripts/image-builder/build.sh b/scripts/image-builder/build.sh index d92d06dd26..ddd50991cb 100755 --- a/scripts/image-builder/build.sh +++ b/scripts/image-builder/build.sh @@ -10,9 +10,10 @@ STARTTIME=$(date +%s) trap ${ROOTDIR}/cleanup.sh INT usage() { - echo "Usage: $(basename $0) <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip]" + echo "Usage: $(basename $0) <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-custom_rpms /path/to/file1.rpm,...,/path/to/fileN.rpm]" echo " -pull_secret_file Path to a file containing the OpenShift pull secret" echo " -ostree_server_name Name or IP address of the OS tree server (default: ${OSTREE_SERVER_IP})" + echo " -custom_rpms Path to one or more comma-separated RPM packages to be included in the image" echo "" echo "Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret." exit 1 @@ -67,7 +68,7 @@ build_image() { title "Loading ${blueprint} blueprint v${version}" sudo composer-cli blueprints delete ${blueprint} 2>/dev/null || true - sudo composer-cli blueprints push "${ROOTDIR}/config/${blueprint_file}" + sudo composer-cli blueprints push "${ROOTDIR}/_builds/${blueprint_file}" sudo composer-cli blueprints depsolve ${blueprint} 1>/dev/null if [ -n "$parent_version" ]; then @@ -105,6 +106,12 @@ while [ $# -gt 0 ] ; do [ -z "${OCP_PULL_SECRET_FILE}" ] && usage shift ;; + -custom_rpms) + shift + CUSTOM_RPM_FILES="$1" + [ -z "${CUSTOM_RPM_FILES}" ] && usage + shift + ;; *) usage ;; @@ -141,21 +148,47 @@ reposync -n -a x86_64 --download-path openshift-local --repo=rhocp-4.10-for-rhel find openshift-local -name \*coreos\* -exec rm -f {} \; createrepo openshift-local >/dev/null +# Copy user-specific RPM packages +rm -rf custom-rpms 2>/dev/null || true +if [ ! -z ${CUSTOM_RPM_FILES} ] ; then + title "Building User-Specified RPM repository" + mkdir custom-rpms + for rpm in ${CUSTOM_RPM_FILES//,/ } ; do + cp $rpm custom-rpms + done + createrepo custom-rpms >/dev/null +fi + title "Loading sources for OpenShift and MicroShift" -for f in openshift-local microshift-local ; do +for f in openshift-local microshift-local custom-rpms ; do + [ ! -d $f ] && continue cat ../config/${f}.toml.template | sed "s;REPLACE_IMAGE_BUILDER_DIR;${ROOTDIR};g" > ${f}.toml sudo composer-cli sources delete $f 2>/dev/null || true sudo composer-cli sources add ${ROOTDIR}/_builds/${f}.toml done +title "Preparing blueprints" +cp -f ../config/{blueprint_v0.0.1.toml,installer.toml} . +if [ ! -z ${CUSTOM_RPM_FILES} ] ; then + for rpm in ${CUSTOM_RPM_FILES//,/ } ; do + rpm_name=$(basename $rpm | sed 's/.rpm//g') + cat >> blueprint_v0.0.1.toml < kickstart.ks # Run the ISO creation procedure diff --git a/scripts/image-builder/cleanup.sh b/scripts/image-builder/cleanup.sh index 5d41e077ee..09c6a493e6 100755 --- a/scripts/image-builder/cleanup.sh +++ b/scripts/image-builder/cleanup.sh @@ -27,9 +27,13 @@ fi title "Cleaning up local ostree container server" sudo podman rm -f microshift-container-server 2>/dev/null || true -if [ "$FULL_CLEAN" = 1 ] ; then - sudo kill microshift 2>/dev/null || true - sleep 5 +if [ "$FULL_CLEAN" = 1 ] ; then + SRV_PID=$(pidof microshift) || true + if [ ! -z ${SRV_PID} ] ; then + sudo kill ${SRV_PID} 2>/dev/null || true + # Waiting for the server to exit + while sudo kill -0 ${SRV_PID} 2>/dev/null; do sleep 5 ; done + fi sudo podman rmi -af fi @@ -52,10 +56,11 @@ sudo composer-cli sources delete microshift-local 2>/dev/null || true if [ "$FULL_CLEAN" = 1 ] ; then title "Clean up user cache" rm -rf ~/.cache 2>/dev/null || true + sudo rm -rf /tmp/containers/* 2>/dev/null || true fi title "Clean osbuild worker cache" sudo systemctl stop osbuild-composer.socket osbuild-composer.service osbuild-worker@1.service -sleep 3 +sleep 5 sudo rm -rf /var/cache/osbuild-worker/* sudo systemctl start osbuild-composer.socket diff --git a/scripts/image-builder/config/custom-rpms.template b/scripts/image-builder/config/custom-rpms.template new file mode 100644 index 0000000000..bfd75d3bc7 --- /dev/null +++ b/scripts/image-builder/config/custom-rpms.template @@ -0,0 +1,7 @@ +id = "custom-rpms" +name = "MicroShift User-Specific RPM Repo" +type = "yum-baseurl" +url = "file://REPLACE_IMAGE_BUILDER_DIR/_builds/custom-rpms/" +check_gpg = false +check_ssl = false +system = false diff --git a/scripts/image-builder/configure.sh b/scripts/image-builder/configure.sh index 1b259a82b8..45f6052795 100755 --- a/scripts/image-builder/configure.sh +++ b/scripts/image-builder/configure.sh @@ -1,8 +1,15 @@ #!/bin/bash set -exo pipefail -sudo dnf install -y git osbuild-composer composer-cli cockpit-composer bash-completion podman genisoimage createrepo syslinux yum-utils selinux-policy-devel +sudo dnf install -y git osbuild-composer composer-cli \ + cockpit-composer bash-completion podman genisoimage \ + createrepo syslinux yum-utils selinux-policy-devel jq sudo systemctl enable osbuild-composer.socket --now sudo systemctl enable cockpit.socket --now sudo firewall-cmd -q --add-service=cockpit sudo firewall-cmd -q --add-service=cockpit --permanent + +# The mock utility comes from the EPEL repository +sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm +sudo dnf install -y mock +sudo usermod -a -G mock $(whoami)