From ed5639e41f0189d6b82b3dd1b09785ac2f222b17 Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Mon, 27 Jun 2022 06:14:53 -0400 Subject: [PATCH 01/10] Initial implementation and documentation of offline container inclusion in the RHEL4Edge ISO --- README.md | 1 + docs/devenv_rhel8.md | 109 +---------------- docs/rhel4edge_iso.md | 111 ++++++++++++++++++ scripts/image-builder/build.sh | 39 +++++- scripts/image-builder/cleanup.sh | 1 + .../config/container-local.toml.template | 7 ++ 6 files changed, 158 insertions(+), 110 deletions(-) create mode 100644 docs/rhel4edge_iso.md create mode 100644 scripts/image-builder/config/container-local.toml.template diff --git a/README.md b/README.md index 0b24027e6d..10799e75e4 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ For production deployments, MicroShift can be run on bare metal hardware or hype ## User Documentation To install, configure and run MicroShift, refer to the following documentation: +- [Install MicroShift on RHEL for Edge](./docs/rhel4edge_iso.md) - [MicroShift Behind Proxy](./docs/howto_http_proxy.md) - [Load Balancer for User Workloads](./docs/howto_load_balancer.md) diff --git a/docs/devenv_rhel8.md b/docs/devenv_rhel8.md index 8171451e16..9bcdc76d1d 100644 --- a/docs/devenv_rhel8.md +++ b/docs/devenv_rhel8.md @@ -1,3 +1,5 @@ +# MicroShift Development Environment on RHEL 8.x + ## Create Development Virtual Machine Start by downloading the RHEL 8.6 or above ISO image from the https://developers.redhat.com/products/rhel/download location. > RHEL 9.x operating system is not currently supported. @@ -129,113 +131,6 @@ Note that this command only stops the MicroShift executable. To perform full cle ./hack/cleanup.sh ``` -## Build RHEL for Edge Installer ISO -Log into the development virtual machine with the `microshift` user credentials. - -Follow the instructions in the RPM Packages section to build the binary and create MicroShift RPM packages. - -The scripts for building the installer are located in the `scripts/image-builder` subdirectory. - -### Prerequisites -Execute the `configure.sh` script to install the tools necessary for building the installer image. - -Download the OpenShift pull secret from the https://console.redhat.com/openshift/downloads#tool-pull-secret page and save it into the `~microshift/pull-secret.txt` file. - -Make sure there is more than 20GB of free disk space necessary for the build artifacts. Run the following command to free the space if necessary. -```bash -./scripts/image-builder/cleanup.sh -full -``` - -Note that the command deletes various user and system data, including: -- The `scripts/image-builder/_builds` directory containing image build artifacts -- MicroShift `ostree` server container and all the unused container images -- All the Image Builder jobs are canceled and deleted -- Project-specific Image Builder sources are deleted -- The user `~/.cache` directory is deleted to clean Golang cache -- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files - -### Building Installer -It is recommended to execute the partial cleanup procedure (without `-full` argument) before each build to reclaim cached disk space from previous builds. -```bash -./scripts/image-builder/cleanup.sh -``` -Note that the command deletes various user and system data, including: -- MicroShift `ostree` server container -- All the Image Builder jobs are canceled -- Project-specific Image Builder sources are deleted -- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files - -Run the build script without arguments to see its usage. -```bash -./scripts/image-builder/build.sh -Usage: build.sh <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] - -pull_secret_file Path to a file containing the OpenShift pull secret - -ostree_server_name Name or IP address of the OS tree server (default: ) - -Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret. -``` - -Continue by running the build script with the pull secret file argument and wait until build process is finished. It may take over 30 minutes to complete a full build cycle. -```bash -./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt -``` -The script performs the following tasks: -- Check for minimum 10GB of available disk space -- Set up local MicroShift RPM repository using `packaging/rpm/_rpmbuild` directory contents as a source -- Set up local OpenShift RPM repository using public `rhocp-${OCP_VERSION}-for-rhel-8-${ARCH}-rpms` repository necessary for CRI-O and OpenShift client package installation -- Configure Image Builder to use the local MicroShift and OpenShift RPM repositories for image builds -- Run Image Builder to create an edge container image containing all the MicroShift binaries and dependencies -- Start a local `ostree` server with the above image -- Run Image Builder to create an edge installer image comprised of RHEL for Edge OS and MicroShift binaries -- Rebuild the installer image with the `kickstart.ks` file for performing various OS setup when the host is first booted -- Perform partial cleanup procedure to reclaim cached disk space - -The artifact of the build is the `scripts/image-builder/_builds/microshift-installer.${ARCH}.iso` bootable RHEL for Edge OS image. - -## Install MicroShift for Edge -Log into the host machine using your user credentials. The remainder of this section describes how to install a virtual machine running RHEL for Edge OS containing MicroShift binaries. - -Start by copying the installer image from the development virtual machine to the host file system. -```bash -sudo scp microshift@microshift-dev:/home/microshift/microshift/scripts/image-builder/_builds/microshift-installer.*.iso /var/lib/libvirt/images/ -``` - -Run the following commands to create a virtual machine using the installer image. -```bash -VMNAME="microshift-edge" -sudo -b bash -c " \ -cd /var/lib/libvirt/images/ && \ -virt-install \ - --name ${VMNAME} \ - --vcpus 2 \ - --memory 3096 \ - --disk path=./${VMNAME}.qcow2,size=20 \ - --network network=default,model=virtio \ - --os-type generic \ - --events on_reboot=restart \ - --cdrom ./microshift-installer.$(uname -i).iso \ -" -``` - -Watch the OS console to see the progress of the installation, waiting until the machine is rebooted and the login prompt appears. - -Note that it may be more convenient to access the machine using SSH. Run the following command to get its IP address and use it to remotely connect to the system. -```bash -sudo virsh domifaddr microshift-edge -``` - -Log into the system using `redhat:redhat` credentials and run the following commands to configure the MicroShift access. -```bash -mkdir ~/.kube -sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config -``` - -Finally, check if the MicroShift is up and running by executing `oc` commands. -```bash -oc get cs -oc get pods -A -``` - ## Quick Development and Edge Testing Cycle In the development environment, it is practical to build and run MicroShift executable as described in the [Build MicroShift](#build-microshift) and [Run MicroShift Executable](#run-microshift-executable) sections above. However, it is also necessary to have a convenient technique to occasionally test the system in a RHEL for Edge environment. diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md new file mode 100644 index 0000000000..8978f673b9 --- /dev/null +++ b/docs/rhel4edge_iso.md @@ -0,0 +1,111 @@ +# Install MicroShift on RHEL for Edge +To install MicroShift in a production environment, it is necessary to create a RHEL for Edge ISO installer with all the necessary components preloaded on the image. + +## Build RHEL for Edge Installer ISO +Log into the development virtual machine with the `microshift` user credentials. +> Use insructions in [MicroShift Development Environment on RHEL 8.x](./devenv_rhel8.md) document for creating and configuring the machine. + +Follow the instructions in the [RPM Packages](./devenv_rhel8.md#rpm-packages) section to create MicroShift RPM packages. + +The scripts for building the installer are located in the `scripts/image-builder` subdirectory. + +### Prerequisites +Execute the `scripts/image-builder/configure.sh` script to install the tools necessary for building the installer image. + +Download the OpenShift pull secret from the https://console.redhat.com/openshift/downloads#tool-pull-secret page and save it into the `~microshift/pull-secret.txt` file. + +Make sure there is more than 20GB of free disk space necessary for the build artifacts. Run the following command to free the space if necessary. +```bash +./scripts/image-builder/cleanup.sh -full +``` + +Note that the command deletes various user and system data, including: +- The `scripts/image-builder/_builds` directory containing image build artifacts +- MicroShift `ostree` server container and all the unused container images +- All the Image Builder jobs are canceled and deleted +- Project-specific Image Builder sources are deleted +- The user `~/.cache` and `/tmp/containers` directory contents are deleted to clean various cache files +- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files + +### Building Installer +It is recommended to execute the partial cleanup procedure (without `-full` argument) before each build to reclaim cached disk space from previous builds. +```bash +./scripts/image-builder/cleanup.sh +``` +Note that the command deletes various user and system data, including: +- MicroShift `ostree` server container +- All the Image Builder jobs are canceled +- Project-specific Image Builder sources are deleted +- The `/var/cache/osbuild-worker` directory contents are deleted to clean Image Builder cache files + +Run the build script without arguments to see its usage. +```bash +./scripts/image-builder/build.sh +Usage: build.sh <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-offline_containers /path/to/file1.rpm,...,/path/to/fileN.rpm] + -pull_secret_file Path to a file containing the OpenShift pull secret + -ostree_server_name Name or IP address of the OS tree server (default: ) + -offline_containers Path to one or more RPM packages with offline CRI-O container images + +Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret. +``` + +Continue by running the build script with the pull secret file argument and wait until build process is finished. It may take over 30 minutes to complete a full build cycle. +```bash +./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt +``` +The script performs the following tasks: +- Check for minimum 10GB of available disk space +- Set up local MicroShift RPM repository using `packaging/rpm/_rpmbuild` directory contents as a source +- Set up local OpenShift RPM repository using public `rhocp-${OCP_VERSION}-for-rhel-8-${ARCH}-rpms` repository necessary for CRI-O and OpenShift client package installation +- Configure Image Builder to use the local MicroShift and OpenShift RPM repositories for image builds +- Run Image Builder to create an edge container image containing all the MicroShift binaries and dependencies +- Start a local `ostree` server with the above image +- Run Image Builder to create an edge installer image comprised of RHEL for Edge OS and MicroShift binaries +- Rebuild the installer image with the `kickstart.ks` file for performing various OS setup when the host is first booted +- Perform partial cleanup procedure to reclaim cached disk space + +The artifact of the build is the `scripts/image-builder/_builds/microshift-installer.${ARCH}.iso` bootable RHEL for Edge OS image. + +## Install MicroShift for Edge +Log into the host machine using your user credentials. The remainder of this section describes how to install a virtual machine running RHEL for Edge OS containing MicroShift binaries. + +Start by copying the installer image from the development virtual machine to the host file system. +```bash +sudo scp microshift@microshift-dev:/home/microshift/microshift/scripts/image-builder/_builds/microshift-installer.*.iso /var/lib/libvirt/images/ +``` + +Run the following commands to create a virtual machine using the installer image. +```bash +VMNAME="microshift-edge" +sudo -b bash -c " \ +cd /var/lib/libvirt/images/ && \ +virt-install \ + --name ${VMNAME} \ + --vcpus 2 \ + --memory 3096 \ + --disk path=./${VMNAME}.qcow2,size=20 \ + --network network=default,model=virtio \ + --os-type generic \ + --events on_reboot=restart \ + --cdrom ./microshift-installer.$(uname -i).iso \ +" +``` + +Watch the OS console to see the progress of the installation, waiting until the machine is rebooted and the login prompt appears. + +Note that it may be more convenient to access the machine using SSH. Run the following command to get its IP address and use it to remotely connect to the system. +```bash +sudo virsh domifaddr microshift-edge +``` + +Log into the system using `redhat:redhat` credentials and run the following commands to configure the MicroShift access. +```bash +mkdir ~/.kube +sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config +``` + +Finally, check if the MicroShift is up and running by executing `oc` commands. +```bash +oc get cs +oc get pods -A +``` diff --git a/scripts/image-builder/build.sh b/scripts/image-builder/build.sh index d92d06dd26..4a17c6ea77 100755 --- a/scripts/image-builder/build.sh +++ b/scripts/image-builder/build.sh @@ -10,9 +10,10 @@ STARTTIME=$(date +%s) trap ${ROOTDIR}/cleanup.sh INT usage() { - echo "Usage: $(basename $0) <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip]" + echo "Usage: $(basename $0) <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-offline_containers /path/to/file1.rpm,...,/path/to/fileN.rpm]" echo " -pull_secret_file Path to a file containing the OpenShift pull secret" echo " -ostree_server_name Name or IP address of the OS tree server (default: ${OSTREE_SERVER_IP})" + echo " -offline_containers Path to one or more RPM packages with offline CRI-O container images" echo "" echo "Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret." exit 1 @@ -67,7 +68,7 @@ build_image() { title "Loading ${blueprint} blueprint v${version}" sudo composer-cli blueprints delete ${blueprint} 2>/dev/null || true - sudo composer-cli blueprints push "${ROOTDIR}/config/${blueprint_file}" + sudo composer-cli blueprints push "${ROOTDIR}/_builds/${blueprint_file}" sudo composer-cli blueprints depsolve ${blueprint} 1>/dev/null if [ -n "$parent_version" ]; then @@ -105,6 +106,12 @@ while [ $# -gt 0 ] ; do [ -z "${OCP_PULL_SECRET_FILE}" ] && usage shift ;; + -offline_containers) + shift + OFFLINE_CONTAINER_RPMS="$1" + [ -z "${OFFLINE_CONTAINER_RPMS}" ] && usage + shift + ;; *) usage ;; @@ -141,13 +148,39 @@ reposync -n -a x86_64 --download-path openshift-local --repo=rhocp-4.10-for-rhel find openshift-local -name \*coreos\* -exec rm -f {} \; createrepo openshift-local >/dev/null +# Copy offline container RPM packages +rm -rf container-local 2>/dev/null || true +if [ ! -z ${OFFLINE_CONTAINER_RPMS} ] ; then + title "Building MicroShift Offline Container repository" + mkdir container-local + for rpm in ${OFFLINE_CONTAINER_RPMS//,/ } ; do + cp $rpm container-local + done + createrepo container-local >/dev/null +fi + title "Loading sources for OpenShift and MicroShift" -for f in openshift-local microshift-local ; do +for f in openshift-local microshift-local container-local ; do + [ ! -d $f ] && continue cat ../config/${f}.toml.template | sed "s;REPLACE_IMAGE_BUILDER_DIR;${ROOTDIR};g" > ${f}.toml sudo composer-cli sources delete $f 2>/dev/null || true sudo composer-cli sources add ${ROOTDIR}/_builds/${f}.toml done +title "Preparing blueprints" +cp -f ../config/{blueprint_v0.0.1.toml,installer.toml} . +if [ ! -z ${OFFLINE_CONTAINER_RPMS} ] ; then + for rpm in ${OFFLINE_CONTAINER_RPMS//,/ } ; do + rpm_name=$(basename $rpm | sed 's/.rpm//g') + cat >> blueprint_v0.0.1.toml </dev/null || true if [ "$FULL_CLEAN" = 1 ] ; then title "Clean up user cache" rm -rf ~/.cache 2>/dev/null || true + sudo rm -rf /tmp/containers/* 2>/dev/null || true fi title "Clean osbuild worker cache" diff --git a/scripts/image-builder/config/container-local.toml.template b/scripts/image-builder/config/container-local.toml.template new file mode 100644 index 0000000000..0616e6b275 --- /dev/null +++ b/scripts/image-builder/config/container-local.toml.template @@ -0,0 +1,7 @@ +id = "container-local" +name = "MicroShift Container Local Repo" +type = "yum-baseurl" +url = "file://REPLACE_IMAGE_BUILDER_DIR/_builds/container-local/" +check_gpg = false +check_ssl = false +system = false From 571333c1762a3d6305011731105805bcc2d148d5 Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Mon, 27 Jun 2022 09:40:01 -0400 Subject: [PATCH 02/10] Add Offline Containers section to the docs --- docs/rhel4edge_iso.md | 53 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 52 insertions(+), 1 deletion(-) diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md index 8978f673b9..933e0d4482 100644 --- a/docs/rhel4edge_iso.md +++ b/docs/rhel4edge_iso.md @@ -3,7 +3,7 @@ To install MicroShift in a production environment, it is necessary to create a R ## Build RHEL for Edge Installer ISO Log into the development virtual machine with the `microshift` user credentials. -> Use insructions in [MicroShift Development Environment on RHEL 8.x](./devenv_rhel8.md) document for creating and configuring the machine. +> The development machine configuration guidelines can be found in the [MicroShift Development Environment on RHEL 8.x](./devenv_rhel8.md) document. Follow the instructions in the [RPM Packages](./devenv_rhel8.md#rpm-packages) section to create MicroShift RPM packages. @@ -66,6 +66,57 @@ The script performs the following tasks: The artifact of the build is the `scripts/image-builder/_builds/microshift-installer.${ARCH}.iso` bootable RHEL for Edge OS image. +### Offline Containers +The `scripts/image-builder/build.sh` script supports a special mode for including container images into the generated ISO. This allows `CRI-O` not to pull those images when MicroShift service is first started, saving network bandwidth and avoiding external network connections. + +The remainder of this section demonstrates how to generate container image RPMs for MicroShift and include them in the installation ISO. +>If user workloads depend on additional container images, the respective RPMs need to be created separately. + +Install and configure the `mock` utility prerequisites using the following commands. +```bash +sudo dnf install -y mock +sudo usermod -a -G mock $(whoami) +``` + +Run the `packaging/rpm/make-microshift-images-rpm.sh` script without arguments to see its usage. +```bash +$ ./packaging/rpm/make-microshift-images-rpm.sh +Usage: + make-microshift-images-rpm.sh rpm + make-microshift-images-rpm.sh srpm + make-microshift-images-rpm.sh copr + +pull_secret: Path to a file containing the OpenShift pull secret +architectures: One or more RPM architectures +rpm_mock_target: Target for building RPMs inside a chroot (e.g. 'rhel-8-x86_64') +copr_repo: Target Fedora Copr repository name (e.g. '@redhat-et/microshift-containers') + +Notes: + - The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret + - Use 'x86_64:amd64' or 'aarch64:arm64' as an architecture value + - See /etc/mock/*.cfg for possible RPM mock target values +``` + +Run the script in the `rpm` mode to pull the required images and generate the RPMs including container images required by MicroShift. +```bash +./packaging/rpm/make-microshift-images-rpm.sh rpm ~/pull-secret.txt x86_64:amd64 rhel-8-x86_64 +``` + +If the procedure is successful, the RPM artifacts can be found in the `paack-result` directory. +```bash +$ ls -1 ~/microshift/packaging/rpm/paack-result/*.rpm +/home/microshift/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.src.rpm +/home/microshift/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm +``` + +Finally, run the build script with the `-offline_containers` argument to include the specified container image RPMs into the generated ISO. +```bash +./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt -offline_containers ~/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm +``` +> If user-specific container images need to be included into the ISO, multiple comma-separated RPM files can be specified as the `-offline_containers` argument value. + +When executed in this mode, the `scripts/image-builder/build.sh` script performs an extra step to set up local container RPM repository with one or more specified RPM files as a source. These RPMs are then appended to the blueprint so that they are installed when the operating system boots for the first time. + ## Install MicroShift for Edge Log into the host machine using your user credentials. The remainder of this section describes how to install a virtual machine running RHEL for Edge OS containing MicroShift binaries. From 674e5a9f1757455d6bdefb49225d486c20c23da1 Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Tue, 28 Jun 2022 06:27:09 -0400 Subject: [PATCH 03/10] Rename offline-containers option to customer-rpms to denote the generic nature of the mechanism --- docs/rhel4edge_iso.md | 30 +++++++---------- scripts/image-builder/build.sh | 32 +++++++++---------- .../config/container-local.toml.template | 7 ---- .../image-builder/config/custom-rpms.template | 7 ++++ scripts/image-builder/configure.sh | 5 ++- 5 files changed, 39 insertions(+), 42 deletions(-) delete mode 100644 scripts/image-builder/config/container-local.toml.template create mode 100644 scripts/image-builder/config/custom-rpms.template diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md index 933e0d4482..396b34f870 100644 --- a/docs/rhel4edge_iso.md +++ b/docs/rhel4edge_iso.md @@ -41,10 +41,10 @@ Note that the command deletes various user and system data, including: Run the build script without arguments to see its usage. ```bash ./scripts/image-builder/build.sh -Usage: build.sh <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-offline_containers /path/to/file1.rpm,...,/path/to/fileN.rpm] +Usage: build.sh <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-custom_rpms /path/to/file1.rpm,...,/path/to/fileN.rpm] -pull_secret_file Path to a file containing the OpenShift pull secret -ostree_server_name Name or IP address of the OS tree server (default: ) - -offline_containers Path to one or more RPM packages with offline CRI-O container images + -custom_rpms Path to one or more comma-separated RPM packages to be included in the image Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret. ``` @@ -67,18 +67,12 @@ The script performs the following tasks: The artifact of the build is the `scripts/image-builder/_builds/microshift-installer.${ARCH}.iso` bootable RHEL for Edge OS image. ### Offline Containers -The `scripts/image-builder/build.sh` script supports a special mode for including container images into the generated ISO. This allows `CRI-O` not to pull those images when MicroShift service is first started, saving network bandwidth and avoiding external network connections. +The `scripts/image-builder/build.sh` script supports a special mode for including user-specific RPM files into the generated ISO. The remainder of this section demonstrates how to generate container image RPMs for MicroShift and include them in the installation ISO. -The remainder of this section demonstrates how to generate container image RPMs for MicroShift and include them in the installation ISO. ->If user workloads depend on additional container images, the respective RPMs need to be created separately. +When the container images required by MicroShift are preinstalled, `CRI-O` does not attempt to pull them when MicroShift service is first started, saving network bandwidth and avoiding external network connections. +>If user workloads depend on additional container images, the respective RPMs need to be created by the user separately. -Install and configure the `mock` utility prerequisites using the following commands. -```bash -sudo dnf install -y mock -sudo usermod -a -G mock $(whoami) -``` - -Run the `packaging/rpm/make-microshift-images-rpm.sh` script without arguments to see its usage. +Start by running the `packaging/rpm/make-microshift-images-rpm.sh` script without arguments to see its usage. ```bash $ ./packaging/rpm/make-microshift-images-rpm.sh Usage: @@ -97,25 +91,25 @@ Notes: - See /etc/mock/*.cfg for possible RPM mock target values ``` -Run the script in the `rpm` mode to pull the required images and generate the RPMs including container images required by MicroShift. +Run the script in the `rpm` mode to pull the images required by MicroShift and generate the RPMs including those image data. ```bash ./packaging/rpm/make-microshift-images-rpm.sh rpm ~/pull-secret.txt x86_64:amd64 rhel-8-x86_64 ``` -If the procedure is successful, the RPM artifacts can be found in the `paack-result` directory. +If the procedure runs successfully, the RPM artifacts can be found in the `packaging/rpm/paack-result` directory. ```bash $ ls -1 ~/microshift/packaging/rpm/paack-result/*.rpm /home/microshift/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.src.rpm /home/microshift/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm ``` -Finally, run the build script with the `-offline_containers` argument to include the specified container image RPMs into the generated ISO. +Finally, run the build script with the `-custom_rpms` argument to include the specified container image RPMs into the generated ISO. ```bash -./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt -offline_containers ~/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm +./scripts/image-builder/build.sh -pull_secret_file ~/pull-secret.txt -custom_rpms ~/microshift/packaging/rpm/paack-result/microshift-containers-4.10.18-1.x86_64.rpm ``` -> If user-specific container images need to be included into the ISO, multiple comma-separated RPM files can be specified as the `-offline_containers` argument value. +> If user-specific container images need to be included into the ISO, multiple comma-separated RPM files can be specified as the `-custom_rpms` argument value. -When executed in this mode, the `scripts/image-builder/build.sh` script performs an extra step to set up local container RPM repository with one or more specified RPM files as a source. These RPMs are then appended to the blueprint so that they are installed when the operating system boots for the first time. +When executed in this mode, the `scripts/image-builder/build.sh` script performs an extra step to set up a custom RPM repository with one or more specified RPM files as a source. These RPMs are then appended to the blueprint so that they are installed when the operating system boots for the first time. ## Install MicroShift for Edge Log into the host machine using your user credentials. The remainder of this section describes how to install a virtual machine running RHEL for Edge OS containing MicroShift binaries. diff --git a/scripts/image-builder/build.sh b/scripts/image-builder/build.sh index 4a17c6ea77..a98de7a43d 100755 --- a/scripts/image-builder/build.sh +++ b/scripts/image-builder/build.sh @@ -10,10 +10,10 @@ STARTTIME=$(date +%s) trap ${ROOTDIR}/cleanup.sh INT usage() { - echo "Usage: $(basename $0) <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-offline_containers /path/to/file1.rpm,...,/path/to/fileN.rpm]" + echo "Usage: $(basename $0) <-pull_secret_file path_to_file> [-ostree_server_name name_or_ip] [-custom_rpms /path/to/file1.rpm,...,/path/to/fileN.rpm]" echo " -pull_secret_file Path to a file containing the OpenShift pull secret" echo " -ostree_server_name Name or IP address of the OS tree server (default: ${OSTREE_SERVER_IP})" - echo " -offline_containers Path to one or more RPM packages with offline CRI-O container images" + echo " -custom_rpms Path to one or more comma-separated RPM packages to be included in the image" echo "" echo "Note: The OpenShift pull secret can be downloaded from https://console.redhat.com/openshift/downloads#tool-pull-secret." exit 1 @@ -106,10 +106,10 @@ while [ $# -gt 0 ] ; do [ -z "${OCP_PULL_SECRET_FILE}" ] && usage shift ;; - -offline_containers) + -custom_rpms) shift - OFFLINE_CONTAINER_RPMS="$1" - [ -z "${OFFLINE_CONTAINER_RPMS}" ] && usage + CUSTOM_RPM_FILES="$1" + [ -z "${CUSTOM_RPM_FILES}" ] && usage shift ;; *) @@ -148,19 +148,19 @@ reposync -n -a x86_64 --download-path openshift-local --repo=rhocp-4.10-for-rhel find openshift-local -name \*coreos\* -exec rm -f {} \; createrepo openshift-local >/dev/null -# Copy offline container RPM packages -rm -rf container-local 2>/dev/null || true -if [ ! -z ${OFFLINE_CONTAINER_RPMS} ] ; then - title "Building MicroShift Offline Container repository" - mkdir container-local - for rpm in ${OFFLINE_CONTAINER_RPMS//,/ } ; do - cp $rpm container-local +# Copy user-specific RPM packages +rm -rf custom-rpms 2>/dev/null || true +if [ ! -z ${CUSTOM_RPM_FILES} ] ; then + title "Building User-Specified RPM repository" + mkdir custom-rpms + for rpm in ${CUSTOM_RPM_FILES//,/ } ; do + cp $rpm custom-rpms done - createrepo container-local >/dev/null + createrepo custom-rpms >/dev/null fi title "Loading sources for OpenShift and MicroShift" -for f in openshift-local microshift-local container-local ; do +for f in openshift-local microshift-local custom-rpms ; do [ ! -d $f ] && continue cat ../config/${f}.toml.template | sed "s;REPLACE_IMAGE_BUILDER_DIR;${ROOTDIR};g" > ${f}.toml sudo composer-cli sources delete $f 2>/dev/null || true @@ -169,8 +169,8 @@ done title "Preparing blueprints" cp -f ../config/{blueprint_v0.0.1.toml,installer.toml} . -if [ ! -z ${OFFLINE_CONTAINER_RPMS} ] ; then - for rpm in ${OFFLINE_CONTAINER_RPMS//,/ } ; do +if [ ! -z ${CUSTOM_RPM_FILES} ] ; then + for rpm in ${CUSTOM_RPM_FILES//,/ } ; do rpm_name=$(basename $rpm | sed 's/.rpm//g') cat >> blueprint_v0.0.1.toml < Date: Tue, 28 Jun 2022 11:06:02 -0400 Subject: [PATCH 04/10] Add Offline Mode configuration guidelines --- docs/rhel4edge_iso.md | 55 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md index 396b34f870..4f69e10151 100644 --- a/docs/rhel4edge_iso.md +++ b/docs/rhel4edge_iso.md @@ -154,3 +154,58 @@ Finally, check if the MicroShift is up and running by executing `oc` commands. oc get cs oc get pods -A ``` + +### Offline Mode +It may sometimes be necessary to install a virtual machine that does not have access to the Internet. For instance, such a configuration can be used for testing the [Offline Containers](#offline-containers) or any other setup that needs to work without the Internet access. + +Create a new isolated `libvirt` network configuration by running the following commands. +```bash +NETCONFIG_FILE=/tmp/isolated-network.xml +cat > $NETCONFIG_FILE < + isolated + + + gateway + + + + + + + + +EOF +sudo virsh net-define $NETCONFIG_FILE +sudo virsh net-start isolated +sudo virsh net-autostart isolated +``` + +When running `virt-install` command for bootstrapping a new virtual machine, specify the `--network network=isolated,model=virtio` command line argument to have the virtual machine use the `isolated` network configuration. + +After the virtual machine is created, log into the system and verify that the Internet is not accessible. +```bash +$ curl -I redhat.com +curl: (6) Could not resolve host: redhat.com +``` + +Make sure that `CRI-O` has access to all the container images required by MicroShift. +```bash +$ sudo crictl images +IMAGE TAG IMAGE ID SIZE +k8s.gcr.io/pause 3.6 6270bb605e12e 690kB +quay.io/coreos/flannel v0.14.0 8522d622299ca 68.9MB +quay.io/kubevirt/hostpath-provisioner v0.8.0 7cbc61ff04c89 180MB +quay.io/microshift/flannel-cni v0.14.0 4324dc7a1ffa5 8.12MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev 334363f37666a 401MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev eb9d5c9681cd5 376MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev 60f52af9fc4ba 413MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev ef1c6b04ebe2a 415MB +quay.io/openshift-release-dev/ocp-v4.0-art-dev a538d5965f4fc 458MB +``` + +Finally, check if the MicroShift is up and running by executing `oc` commands. +```bash +oc get cs +oc get pods -A +``` From 75e84ca394c008044c4cc075dafc2b1e902a90a2 Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Tue, 28 Jun 2022 11:19:10 -0400 Subject: [PATCH 05/10] Minor text fixes --- docs/rhel4edge_iso.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md index 4f69e10151..907c2e454e 100644 --- a/docs/rhel4edge_iso.md +++ b/docs/rhel4edge_iso.md @@ -181,7 +181,8 @@ sudo virsh net-start isolated sudo virsh net-autostart isolated ``` -When running `virt-install` command for bootstrapping a new virtual machine, specify the `--network network=isolated,model=virtio` command line argument to have the virtual machine use the `isolated` network configuration. +Follow the instruction in the [Install MicroShift for Edge](#install-microshift-for-edge) section to install a new virtual machine using the `isolated` network configuration. +> When running the `virt-install` command, specify the `--network network=isolated,model=virtio` option to select the `isolated` network configuration. After the virtual machine is created, log into the system and verify that the Internet is not accessible. ```bash @@ -204,7 +205,7 @@ quay.io/openshift-release-dev/ocp-v4.0-art-dev ef1c6b04ebe quay.io/openshift-release-dev/ocp-v4.0-art-dev a538d5965f4fc 458MB ``` -Finally, check if the MicroShift is up and running by executing `oc` commands. +Finally, check if MicroShift is up and running by executing `oc` commands. ```bash oc get cs oc get pods -A From c7a50328fe36e4b45c63af75e115acbc2a1c6bab Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Tue, 28 Jun 2022 12:06:05 -0400 Subject: [PATCH 06/10] Update Quick Dev Cycle documentation --- docs/devenv_rhel8.md | 8 ++++---- docs/rhel4edge_iso.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/devenv_rhel8.md b/docs/devenv_rhel8.md index 9bcdc76d1d..94f44e2766 100644 --- a/docs/devenv_rhel8.md +++ b/docs/devenv_rhel8.md @@ -132,11 +132,11 @@ Note that this command only stops the MicroShift executable. To perform full cle ``` ## Quick Development and Edge Testing Cycle -In the development environment, it is practical to build and run MicroShift executable as described in the [Build MicroShift](#build-microshift) and [Run MicroShift Executable](#run-microshift-executable) sections above. However, it is also necessary to have a convenient technique to occasionally test the system in a RHEL for Edge environment. +During the development cycle, it is practical to build and run MicroShift executable as demonstrated in the [Build MicroShift](#build-microshift) and [Run MicroShift Executable](#run-microshift-executable) sections above. However, it is also necessary to have a convenient technique for testing the system in a setup resembling the production environment. Such an environment can be created in a virtual machine as described in the [Install MicroShift on RHEL for Edge](./rhel4edge_iso.md) document. -The MicroShift Installer ISO build procedure takes over 30 minutes to complete. Once a RHEL for Edge virtual machine is created, it is running a version of MicroShift with the latest changes. +Once a RHEL for Edge virtual machine is created, it is running a version of MicroShift with the latest changes. When MicroShift code is updated and the executable file is rebuilt with the new changes, the updates need to be installed on RHEL for Edge OS. -The remainder of this section describes a simple procedure for replacing the MicroShift executable file on an existing RHEL for Edge OS installation. +Since it takes a long time to create a new RHEL for Edge installer ISO and deploy it on a virtual machine, the remainder of this section describes a simple technique for replacing the MicroShift executable file on an existing RHEL for Edge OS installation. ### Configuring ostree Log into the RHEL for Edge machine using `redhat:redhat` credentials. Run the following command for configuring the ostree to allow transient overlays on top of the /usr directory. @@ -155,7 +155,7 @@ ssh-keygen ssh-copy-id redhat@microshift-edge ``` -Rebuild the MicroShift executable as described in the [Build MicroShift](#build-microshift) section and run the following commands to copy, cleanup and restart the new service on the RHEL for Edge system. +Rebuild the MicroShift executable as described in the [Build MicroShift](#build-microshift) section and run the following commands to copy, cleanup, replace and restart the new service on the RHEL for Edge system. ```bash scp ~/microshift/microshift redhat@microshift-edge: ssh redhat@microshift-edge ' \ diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md index 907c2e454e..915aeefac3 100644 --- a/docs/rhel4edge_iso.md +++ b/docs/rhel4edge_iso.md @@ -143,13 +143,13 @@ Note that it may be more convenient to access the machine using SSH. Run the fol sudo virsh domifaddr microshift-edge ``` -Log into the system using `redhat:redhat` credentials and run the following commands to configure the MicroShift access. +Log into the system using `redhat:redhat` credentials and run the following commands to configure MicroShift access. ```bash mkdir ~/.kube sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config ``` -Finally, check if the MicroShift is up and running by executing `oc` commands. +Finally, check if MicroShift is up and running by executing `oc` commands. ```bash oc get cs oc get pods -A From 0b102347e0ee943bb8e699d29fc02ee46b3bd57f Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Wed, 29 Jun 2022 10:20:28 -0400 Subject: [PATCH 07/10] Fix configure.sh to install EPEL repository for mock utility --- scripts/image-builder/configure.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/scripts/image-builder/configure.sh b/scripts/image-builder/configure.sh index 034b89426f..53b15c52be 100755 --- a/scripts/image-builder/configure.sh +++ b/scripts/image-builder/configure.sh @@ -3,9 +3,13 @@ set -exo pipefail sudo dnf install -y git osbuild-composer composer-cli \ cockpit-composer bash-completion podman genisoimage \ - createrepo syslinux yum-utils selinux-policy-devel mock + createrepo syslinux yum-utils selinux-policy-devel sudo systemctl enable osbuild-composer.socket --now sudo systemctl enable cockpit.socket --now sudo firewall-cmd -q --add-service=cockpit sudo firewall-cmd -q --add-service=cockpit --permanent + +# The mock utility comes from the EPEL repository +sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm +sudo dnf install -y mock sudo usermod -a -G mock $(whoami) From 3388f610cd89fd7710227a6c55b585ebcd91383e Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Wed, 29 Jun 2022 10:30:47 -0400 Subject: [PATCH 08/10] Added jq compaction to the pull secret contents to avoid sed errors --- scripts/image-builder/build.sh | 4 ++-- scripts/image-builder/configure.sh | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/image-builder/build.sh b/scripts/image-builder/build.sh index a98de7a43d..ddd50991cb 100755 --- a/scripts/image-builder/build.sh +++ b/scripts/image-builder/build.sh @@ -185,10 +185,10 @@ build_image blueprint_v0.0.1.toml "${IMGNAME}-container" 0.0.1 edge-container build_image installer.toml "${IMGNAME}-installer" 0.0.0 edge-installer "${IMGNAME}-container" 0.0.1 title "Embedding kickstart in the installer image" -# Create a kickstart file from a template +# Create a kickstart file from a template, compacting pull secret contents if necessary cat "../config/kickstart.ks.template" \ | sed "s;REPLACE_OSTREE_SERVER_NAME;${OSTREE_SERVER_NAME};g" \ - | sed "s;REPLACE_OCP_PULL_SECRET_CONTENTS;$(cat $OCP_PULL_SECRET_FILE);g" \ + | sed "s;REPLACE_OCP_PULL_SECRET_CONTENTS;$(cat $OCP_PULL_SECRET_FILE | jq -c);g" \ > kickstart.ks # Run the ISO creation procedure diff --git a/scripts/image-builder/configure.sh b/scripts/image-builder/configure.sh index 53b15c52be..45f6052795 100755 --- a/scripts/image-builder/configure.sh +++ b/scripts/image-builder/configure.sh @@ -3,7 +3,7 @@ set -exo pipefail sudo dnf install -y git osbuild-composer composer-cli \ cockpit-composer bash-completion podman genisoimage \ - createrepo syslinux yum-utils selinux-policy-devel + createrepo syslinux yum-utils selinux-policy-devel jq sudo systemctl enable osbuild-composer.socket --now sudo systemctl enable cockpit.socket --now sudo firewall-cmd -q --add-service=cockpit From b63601e06775491c8b15031c726791dce8bbf43b Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Wed, 29 Jun 2022 11:23:56 -0400 Subject: [PATCH 09/10] Added a wait loop in cleanup.sh to make sure microshift process ends --- scripts/image-builder/cleanup.sh | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/scripts/image-builder/cleanup.sh b/scripts/image-builder/cleanup.sh index 113fbde7ee..09c6a493e6 100755 --- a/scripts/image-builder/cleanup.sh +++ b/scripts/image-builder/cleanup.sh @@ -27,9 +27,13 @@ fi title "Cleaning up local ostree container server" sudo podman rm -f microshift-container-server 2>/dev/null || true -if [ "$FULL_CLEAN" = 1 ] ; then - sudo kill microshift 2>/dev/null || true - sleep 5 +if [ "$FULL_CLEAN" = 1 ] ; then + SRV_PID=$(pidof microshift) || true + if [ ! -z ${SRV_PID} ] ; then + sudo kill ${SRV_PID} 2>/dev/null || true + # Waiting for the server to exit + while sudo kill -0 ${SRV_PID} 2>/dev/null; do sleep 5 ; done + fi sudo podman rmi -af fi @@ -57,6 +61,6 @@ fi title "Clean osbuild worker cache" sudo systemctl stop osbuild-composer.socket osbuild-composer.service osbuild-worker@1.service -sleep 3 +sleep 5 sudo rm -rf /var/cache/osbuild-worker/* sudo systemctl start osbuild-composer.socket From cd1343070763c2026f62c9147e6c9f620d35d603 Mon Sep 17 00:00:00 2001 From: "ggiguash@redhat.com" Date: Mon, 4 Jul 2022 09:54:37 -0400 Subject: [PATCH 10/10] Move RHEL4Edge document to dev section in README --- README.md | 2 +- docs/rhel4edge_iso.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 10799e75e4..47148c1576 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,6 @@ For production deployments, MicroShift can be run on bare metal hardware or hype ## User Documentation To install, configure and run MicroShift, refer to the following documentation: -- [Install MicroShift on RHEL for Edge](./docs/rhel4edge_iso.md) - [MicroShift Behind Proxy](./docs/howto_http_proxy.md) - [Load Balancer for User Workloads](./docs/howto_load_balancer.md) @@ -41,6 +40,7 @@ To build MicroShift from source and contribute to its development, refer to the - [MicroShift Design](./docs/design.md) - [MicroShift Development Environment on RHEL 8.x](./docs/devenv_rhel8.md) +- [Install MicroShift on RHEL for Edge](./docs/rhel4edge_iso.md) - [OpenShift CI for MicroShift](./docs/openshift_ci.md) ## Community diff --git a/docs/rhel4edge_iso.md b/docs/rhel4edge_iso.md index 915aeefac3..8330829bd7 100644 --- a/docs/rhel4edge_iso.md +++ b/docs/rhel4edge_iso.md @@ -1,5 +1,5 @@ # Install MicroShift on RHEL for Edge -To install MicroShift in a production environment, it is necessary to create a RHEL for Edge ISO installer with all the necessary components preloaded on the image. +To test MicroShift in a setup similar to the production environment, it is necessary to create a RHEL for Edge ISO installer with all the necessary components preloaded on the image. ## Build RHEL for Edge Installer ISO Log into the development virtual machine with the `microshift` user credentials.