diff --git a/_topic_map.yml b/_topic_map.yml index 59cd1b38a991..0b69829da00f 100644 --- a/_topic_map.yml +++ b/_topic_map.yml @@ -311,6 +311,8 @@ Name: Post-installation configuration Dir: post_installation_configuration Distros: openshift-origin,openshift-enterprise,openshift-webscale Topics: +- Name: Machine configuration tasks + File: machine-configuration-tasks - Name: Cluster tasks File: cluster-tasks - Name: Node tasks diff --git a/modules/checking-mco-status.adoc b/modules/checking-mco-status.adoc new file mode 100644 index 000000000000..d95f58d65aac --- /dev/null +++ b/modules/checking-mco-status.adoc @@ -0,0 +1,112 @@ +// Module included in the following assemblies: +// +// * post_installation_configuration/machine-configuration-tasks.adoc + +[id="checking-mco-status_{context}"] += Checking Machine Config Pool status + +To see the status of the Machine Config Operator, its sub-components, +and the resources it manages, use the following `oc` commands: + +.Procedure +. To see the number of MCO-managed nodes available on your cluster for each pool, type: ++ +[source,terminal] +---- +$ oc get machineconfigpool +NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE +master rendered-master-dd… True False False 3 3 3 0 4h42m +worker rendered-worker-fde… True False False 3 3 3 0 4h42m +---- ++ +In the previous output, there are three master and three worker nodes. All machines are updated +and none are currently updating. Because all nodes are Updated and Ready and none are Degraded, +you can ell that there are no issues. + +. To see each existing `machineconfig`, type: ++ +[source,terminal] +---- +$ oc get machineconfigs +NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE +00-master 2c9371fbb673b97a6fe8b1c52... 3.1.0 5h18m +00-worker 2c9371fbb673b97a6fe8b1c52... 3.1.0 5h18m +01-master-container-runtime 2c9371fbb673b97a6fe8b1c52... 3.1.0 5h18m +01-master-kubelet 2c9371fbb673b97a6fe8b1c52… 3.1.0 5h18m +... +rendered-master-dde... 2c9371fbb673b97a6fe8b1c52... 3.1.0 5h18m +rendered-worker-fde... 2c9371fbb673b97a6fe8b1c52... 3.1.0 5h18m +---- ++ +Note that the `machineconfigs` listed as `rendered` are not meant to be +changed or deleted. Expect them to be hidden at some point in the future. + +. Check the status of worker (or change to master) to see the status of that pool of nodes: ++ +[source,terminal] +---- +$ oc describe mcp worker +... + Degraded Machine Count: 0 + Machine Count: 3 + Observed Generation: 2 + Ready Machine Count: 3 + Unavailable Machine Count: 0 + Updated Machine Count: 3 +Events: +---- + +. You can view the contents of a particular machineconfig (in this case, +`01-master-kubelet`). The trimmed output from the following `oc describe` +command shows that this `machineconfig` contains both configuration files +(`cloud.conf` and `kubelet.conf`) and a systemd service +(Kubernetes Kubelet): ++ +[source,terminal] +---- +$ oc describe machineconfigs 01-master-kubelet +Name: 01-master-kubelet +... +Spec: + Config: + Ignition: + Version: 3.1.0 + Storage: + Files: + Contents: + Source: data:, + Mode: 420 + Overwrite: true + Path: /etc/kubernetes/cloud.conf + Contents: + Source: data:,kind%3A%20KubeletConfiguration%0AapiVersion%3A%20kubelet.config.k8s.io%2Fv1beta1%0Aauthentication%3A%0A%20%20x509%3A%0A%20%20%20%20clientCAFile%3A%20%2Fetc%2Fkubernetes%2Fkubelet-ca.crt%0A%20%20anonymous... + Mode: 420 + Overwrite: true + Path: /etc/kubernetes/kubelet.conf + Systemd: + Units: + Contents: [Unit] +Description=Kubernetes Kubelet +Wants=rpc-statd.service network-online.target crio.service +After=network-online.target crio.service + +ExecStart=/usr/bin/hyperkube \ + kubelet \ + --config=/etc/kubernetes/kubelet.conf \ ... +---- + +If something goes wrong with a machineconfig that you apply, you can always +back out that change. For example, if you had run `oc create -f ./myconfig.yaml` +to apply a machineconfig, you could remove that machineconfig by typing: ++ +[source,terminal] +---- +$ oc delete -f ./myconfig.yaml +---- ++ +If that was the only problem, the nodes in the affected pool should return to a non-degraded state. +This actually causes the rendered configuration to roll back to its previously rendered state. + +If you add your own MachineConfigs to your cluster, you can use the commands +shown in the previous example to check their status and the related status of +the pool to which they are applied. diff --git a/modules/create-a-containerruntimeconfig-crd.adoc b/modules/create-a-containerruntimeconfig-crd.adoc new file mode 100644 index 000000000000..132ec4ca99fc --- /dev/null +++ b/modules/create-a-containerruntimeconfig-crd.adoc @@ -0,0 +1,115 @@ +// Module included in the following assemblies: +// +// * post_installation_configuration/machine-configuration-tasks.adoc + +[id="create-a-containerruntimeconfig_{context}"] + += Creating a ContainerRuntime CR to edit CRI-O parameters +The ContainerRuntimeConfig custom resource definition (CRD) provides a +structured way of changing settings associated with the {product-title} +CRI-O runtime. Using a ContainerRuntimeConfig custom resource (CR), +you select the configuration values you want and the MCO handles rebuilding +the `crio.conf` and `storage.conf` configuration files. + +Parameters you can set in a ContainerRuntimeConfig CR include: + +* **PIDs limit**: Sets the maximum number of processes allowed in a container. +By default, the limit is set to 1024 (`pids_limit = 1024`). +* **Log level**: Sets the level of verbosity for log messages. The default is +`info` (`log_level = info`). Other options include `fatal`, `panic`, `error`, +`warn`, `debug`, and `trace`. +* **Overlay size**: Sets the maxim size of a container image. The default is +10 GB. +* **Maximum log size**: Sets the maximum size allowed for the container +log file. The default maximum log size is unlimited (`log_size_max = -1`). +If it is set to a positive number, it must be at least 8192 +to not be smaller than `conmon`'s read buffer. Conmon is a program that +monitors communications between a container manager (such as Podman or CRI-O) +and the OCI runtime (such as runc or crun) for a single container. + +The following procedure describes how to change CRI-O settings using the +ContainerRuntimeConfig CR. + +.Procedure + +. To raise the `pidsLimit` to 2048, set the `logLevel` to `debug`, and +set the `overlaySize` to 8 GB, create a CR file +(for example, `overlay-size.yaml`) that contains that setting: ++ +[source,yaml] +---- +$ cat << EOF > /tmp/overlay-size.yaml +apiVersion: machineconfiguration.openshift.io/v1 +kind: ContainerRuntimeConfig +metadata: + name: overlay-size +spec: + machineConfigPoolSelector: + matchLabels: + custom-crio: overlay-size + containerRuntimeConfig: + pidsLimit: 2048 + logLevel: debug + overlaySize: 8G +EOF +---- + +. To apply the ContainerRuntimeConfig settings, run: ++ +[source,terminal] +---- +$ oc create -f /tmp/overlay-size +---- + +. To verify that the settings wer applied, run: ++ +[source,terminal] +---- +$ oc get ContainerRuntimeConfig +NAME AGE +overlay-size 3m19s + +---- + +. To edit a pool of machines, such as `worker`, run the following +command to open a MachineConfigPool: ++ +[source,terminal] +---- +$ oc edit machineconfigpool worker +---- + +. Check that a new containerruntime object has appeared under the machineconfigs: ++ +[source,terminal] +---- +$ oc get machineconfigs | grep containerrun +99-worker-generated-containerruntime 2c9371fbb673b97a6fe8b1c52691999ed3a1bfc2 3.1.0 31s +---- +. Monitor the Machine Config Pool as the changes are rolled into the machines until all are shown as ready: ++ +[source,terminal] +---- +$ oc get mcp worker +NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE +worker rendered-worker-169 False True False 3 1 1 0 9h +---- + +. Open an `oc debug` session to a worker node and run `chroot /host`. + +. Verify the changes by running: ++ +[source,terminal] +---- +$ crio config | egrep 'log_level|pids_limit' +pids_limit = 2048 +log_level = "debug" +$ head -n 7 /etc/containers/storage.conf +[storage] + driver = "overlay" + runroot = "/var/run/containers/storage" + graphroot = "/var/lib/containers/storage" + [storage.options] + additionalimagestores = [] + size = "8G" +---- diff --git a/modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc b/modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc index 4ff2c910a268..183102fe518c 100644 --- a/modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc +++ b/modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc @@ -2,6 +2,7 @@ // // * scalability_and_performance/recommended-host-practices.adoc // * post_installation_configuration/node-tasks.adoc +// * post_installation_configuration/machine-configuration-tasks.adoc [id="create-a-kubeletconfig-crd-to-edit-kubelet-parameters_{context}"] = Creating a KubeletConfig CRD to edit kubelet parameters @@ -91,13 +92,6 @@ spec: kubeAPIBurst: kubeAPIQPS: ---- -+ -.. Run: -+ -[source,terminal] ----- -$ oc label machineconfigpool worker custom-kubelet=large-pods ----- .. Run: + diff --git a/modules/installation-special-config-crony.adoc b/modules/installation-special-config-crony.adoc index 22a1f4c53027..926c299e4ce3 100644 --- a/modules/installation-special-config-crony.adoc +++ b/modules/installation-special-config-crony.adoc @@ -1,6 +1,7 @@ // Module included in the following assemblies: // // * installing/install_config/installing-customizing.adoc +// * post_installation_configuration/machine-configuration-tasks.adoc [id="installation-special-config-crony_{context}"] = Configuring chrony time service @@ -68,10 +69,12 @@ EOF . Make a backup copy of the configuration file. -. If the cluster is not up yet, generate manifest files, add this file to the `openshift` -directory, then continue to create the cluster. - -. If the cluster is already running, apply the file as follows: +. Apply the configuration in one of two ways: ++ +* If the cluster is not up yet, generate manifest files, add this file to the `openshift` +directory, and then continue to create the cluster. ++ +* If the cluster is already running, apply the file as follows: + [source,terminal] ---- diff --git a/modules/machine-config-operator.adoc b/modules/machine-config-operator.adoc index 1c5a36ce0998..a29776dd73c8 100644 --- a/modules/machine-config-operator.adoc +++ b/modules/machine-config-operator.adoc @@ -1,6 +1,7 @@ // Module included in the following assemblies: // -// * operators/operator-reference.adoc +// * operators/operator-reference.adoc +// * post_installation_configuration/machine-configuration-tasks.adoc [id="machine-config-operator_{context}"] = Machine Config Operator @@ -8,7 +9,7 @@ [discrete] == Purpose -The Machine Congig Operator manages and applies configuration and updates of the +The Machine Config Operator manages and applies configuration and updates of the base operating system and container runtime, including everything between the kernel and kubelet. diff --git a/modules/machine-config-overview.adoc b/modules/machine-config-overview.adoc new file mode 100644 index 000000000000..d6cba9d15bcd --- /dev/null +++ b/modules/machine-config-overview.adoc @@ -0,0 +1,109 @@ +// Module included in the following assemblies: +// +// * operators/operator-reference.adoc +// * post_installation_configuration/machine-configuration-tasks.adoc + +[id="machine-config-overview{context}"] += MachineConfig overview + +The Machine Config Operator (MCO) manages updates to systemd, +CRI-O and Kubelet, the kernel, Network Manager and other +system features. +It also offers a MachineConfig CRD that can write configuration files +onto the host (see +link:https://github.com/openshift/machine-config-operator#machine-config-operator[machine-config-operator]) +Understanding what MCO does and how it interacts with other components +is critical to making advanced, system-level changes to an +{product-title} cluster. Here are some things you should know about MCO, +MachineConfigs, and how they are used: + +* A MachineConfig can make a specific change to a file or service on the +operating system of each system representing a pool of {product-title} nodes. + +* MCO applies changes to operating systems in pools of machines. All {product-title} clusters +start with worker and master node pools. By adding more +role labels, you can configure custom pools of nodes. For example, +you can set up a custom pool of worker nodes that includes particular hardware +features needed by an application. However, examples in this section focus +on changes to the default pool types. + +* Some machine configuration must be in place before {product-title} +is installed to disk. In most cases, this can be accomplished by creating +a MachineConfig that is injected directly into the {product-title} installer +process, instead of running as a post-installation MachineConfig. +In other cases, you might need to do bare metal installation where you +pass kernel arguments at {product-title} installer start-up, to do such +things as setting per-node individual IP addresses or advanced disk partitioning. + +* MCO manages items that are set in MachineConfigs. Manual changes you do to +your systems will not be overwritten by MCO, unless MCO is explicitly told to +manage a conflicting file. In other words, MCO only makes specific updates +you request, it does not claim control over the whole node. + +* Manual changes to nodes are strongly discouraged. If you need to decommission +a node and start a new one, those direct changes would be lost. + +* MCO is only supported for writing to files in `/etc` and `/var` directories, although +there are symbolic links to some directories that can be writeable by being +symbolically linked to one of those areas. The `/opt` directory is an example. + +* Ignition is the configuration format used in MachineConfigs. See the +link:https://github.com/coreos/ignition/blob/master/docs/configuration-v3_1.md[Ignition Configuration Specification v3.1.0] +for details. + +* Although Ignition config settings can be delivered directly at {product-title} +installation time, and are formatted in the same way that MCO delivers Ignition +configs, MCO has no way of seeing what those original Ignition configs are. So +you should wrap Ignition config settings into a MachineConfig before +deploying them. + +* When a file managed by MCO changes outside of MCO, the Machine +Config Daemon (MCD) sets the node as `degraded`. It will not overwrite the +offending file, however, and should continue to operate in a `degraded` state. + +* A key reason for using a MachineConfig is that it will be applied when +you spin up new nodes for a pool in your {product-title} cluster. The `machine-api-operator` +provisions a new machine and MCO configures it. + +MCO uses link:https://github.com/coreos/ignition[CoreOS Ignition] as the +configuration format {product-title} 4.6 moved from Ignition version 2 to +Ignition version 3 format. + +== What can you change with MachineConfigs? +The kinds of components that MCO can change include: + +* **config**: Create Ignition config objects (see the +link:https://github.com/coreos/ignition/blob/master/docs/configuration-v3_1.md[Ignition configuration specification]) +to do things like modify files, systemd services, and other features on {product-title} machines, including: +- **Configuration files**: Create or overwrite files in the `/var` or `/etc` directory. +- **systemd units**: Create and set the status of a systemd service or add to an existing systemd service by dropping in additional settings. +- **users and groups**: Change ssh keys in the passwd section post-installation. +* **kernelArguments**: Add arguments to the kernel command line when {product-title} nodes boot. +* **kernelType**: Optionally identify a non-standard kernel to use instead of the standard kernel. Use `realtime` +to use the RT kernel (for RAN). This is only supported on select platforms. +* **fips**: Enable link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/security_hardening/index#using-the-system-wide-cryptographic-policies_security-hardening[FIPS] +mode. FIPS should be set at +installation-time setting and not a post-installation procedure. +* **extensions**: Extend {op-system} features by adding selected pre-packaged software. +For this feature (new in {product-title} 4.6), available extensions include +link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/security_hardening/index#protecting-systems-against-intrusive-usb-devices_security-hardening[usbguard] and kernel modules. +* **Custom resources (for ContainerRuntime and Kubelet)**: Outside of +MachineConfigs, MCO manages two special custom resources for modifying +CRI-O container runtime settings (ContainerRuntime CR) and the +Kubelet service (Kubelet CR). + +The MCO is not the only Operator that can change operating +system components on {product-title} nodes. Other Operators +can modify operating system-level features as well. One example +is the link:https://docs.openshift.com/container-platform/4.5/operators/operator-reference.html#about-node-tuning-operator_red-hat-operators[Node Tuning Operator], +which allows you to do node-level tuning through Tuned daemon profiles. + +Tasks for the MCO configuration that can be done post-installation +are included in the following procedures. See descriptions of +{op-system} bare metal installation for system configuration +tasks that must be done during or before {product-title} installation. + +== Project + +See the link:https://github.com/openshift/machine-config-operator[openshift-machine-config-operator] +GitHub site for details. diff --git a/modules/machineconfig-modify-journald.adoc b/modules/machineconfig-modify-journald.adoc new file mode 100644 index 000000000000..dcbac4952b28 --- /dev/null +++ b/modules/machineconfig-modify-journald.adoc @@ -0,0 +1,115 @@ +// Module included in the following assemblies: +// +// * installing/post_installation_configuration/machine-configuration-tasks.adoc +// * post_installation_configuration/machine-configuration-tasks.adoc + +[id="machineconfig-modify-journald_{context}"] += Configuring journald settings + +If you need to configure settings for the `journald` service on {product-title} +nodes, you can do that by modifying the appropriate configuration file and passing the +file to the appropriate pool of nodes as a MachineConfig. + +This procedure describes how to modify `journald` rate limiting settings +in the `/etc/systemd/journald.conf` file and apply them to worker nodes. +See the `journald.conf` man page for information on how to use that file. + +.Prerequisites +* Have a running {product-title} cluster (version 4.4 or later). +* Log in to the cluster as a user with administrative privileges. + +.Procedure + +. Create the contents of the `/etc/systemd/journald.conf` file and encode it as base64. For example: ++ +[source,terminal] +---- +$ cat > /tmp/jrnl.conf < /tmp/40-worker-custom-journald.yaml < /tmp/myregistry.yaml < 80-extensions.yaml +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 80-worker-extensions +spec: + config: + ignition: + version: 3.1.0 + extensions: + - usbguard +EOF +---- + +. Add the MachineConfig to the cluster. Type the following to add the MachineConfig +to the cluster: ++ +[source,terminal] +---- +$ oc create -f 80-extensions.yaml +---- ++ +This sets all worker nodes to have rpm packages for `usbguard` +installed. + +. Check that the extensions were applied: ++ +[source,terminal] +---- +$ oc get machineconfig 80-worker-extensions +NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE +80-worker-extensions 3.1.0 57s +---- + +. Check that the new MachineConfig has been applied and that the nodes +are not in a degraded state. It may take a few minutes. +The worker pool will show the updates in progress, as each machine successfully +has the new MachineConfig applied: ++ +[source,terminal] +---- +$ oc get machineconfigpool +NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE +master rendered-master-35 True False False 3 3 3 0 34m +worker rendered-worker-d8 False True False 3 1 1 0 34m +---- + + +. Check the extensions. To check that the extension was applied, run: ++ +[source,terminal] +---- +$ oc get node | grep worker +NAME STATUS ROLES AGE VERSION +ip-10-0-169-2.us-east-2.compute.internal Ready worker 102m v1.18.3 +$ oc debug node/ip-10-0-169-2.us-east-2.compute.internal +... +To use host binaries, run `chroot /host` +sh-4.4# chroot /host +sh-4.4# rpm -q usbguard +usbguard-0.7.4-4.el8.x86_64.rpm +---- diff --git a/post_installation_configuration/machine-configuration-tasks.adoc b/post_installation_configuration/machine-configuration-tasks.adoc new file mode 100644 index 000000000000..f1a1c34fe4f8 --- /dev/null +++ b/post_installation_configuration/machine-configuration-tasks.adoc @@ -0,0 +1,60 @@ +:context: post-install-machine-configuration-tasks +[id="post-install-machine-configuration-tasks"] += Post-installation machine configuration tasks +include::modules/common-attributes.adoc[] +toc::[] + +There are times when you need to make changes to the +operating systems running on {product-title} nodes. This can include +changing settings for network time service, adding kernel arguments, +or configuring journaling in a specific way. + +Aside from a few specialized features, most changes to +operating systems on {product-title} nodes can be done by creating +what are referred to as MachineConfig objects that are managed by the +Machine Config Operator. + +Tasks in this section describe how to use features of +the Machine Config Operator to configure operating system +features on {product-title} nodes. + +[id="understanding-the-machine-config-operator"] +== Understanding the Machine Config Operator + +include::modules/machine-config-operator.adoc[leveloffset=+2] +include::modules/machine-config-overview.adoc[leveloffset=+2] +include::modules/checking-mco-status.adoc[leveloffset=+2] + +[id="using-machineconfigs-to-change-machines"] +== Using MachineConfigs to configure nodes +Tasks in this section let you create MachineConfig objects to modify files, systemd unit files, and other operating system features running on {product-title} nodes. For more ideas on working with MachineConfigs, see + content related to link:https://access.redhat.com/solutions/5307301[changing MTU network settings], link:https://access.redhat.com/solutions/5096731[adding] or +link:https://access.redhat.com/solutions/4510281[updating] SSH authorized keys, , link:https://access.redhat.com/solutions/4518671[replacing DNS nameservers], link:https://access.redhat.com/verify-images-ocp4[verifying image signatures], link:https://access.redhat.com/solutions/4727321[enabling SCTP], and link:https://access.redhat.com/solutions/5170251[configuring iSCSI initiatornames] for {product-title}. + +MachineConfigs + +{product-title} version 4.6 supports +link:https://github.com/coreos/ignition/blob/master/docs/configuration-v3_1.md[Ignition specification version 3.1]. +All new MachineConfigs you create going forward should be based on +Ignition specification version 3.1. If you are upgrading your {product-title} cluster, +any existing Ignition specification version 2.x MachineConfigs will be translated automatically to +specification version 3.1. + +include::modules/installation-special-config-crony.adoc[leveloffset=+2] +include::modules/nodes-nodes-kernel-arguments.adoc[leveloffset=+2] +include::modules/nodes-nodes-rtkernel-arguments.adoc[leveloffset=+2] +include::modules/machineconfig-modify-journald.adoc[leveloffset=+2] +include::modules/machineconfig-modify-registry.adoc[leveloffset=+2] +include::modules/rhcos-add-extensions.adoc[leveloffset=+2] + +Use the "Configuring crony time service" section as a model for how to go about adding +other configuration files to {product-title} nodes. + +[id="configuring-machines-with-custom-resources"] +== Configuring MCO-related custom resources +Besides managing MachineConfigs, the MCO manages two custom resources (CRs): KubeletConfig and ContainerRuntimeConfig. +Those CRs let you change node-level settings impacting how +the Kubelet and CRI-O container runtime services behave. + +include::modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc[leveloffset=+2] +include::modules/create-a-containerruntimeconfig-crd.adoc[leveloffset=+2]