[ShiftStack] SR-IOV compute machines for IPI installations

_topic_map.yml

@@ -265,6 +265,8 @@ Topics:
     File: installing-openstack-installer-custom
   - Name: Installing a cluster on OpenStack with Kuryr
     File: installing-openstack-installer-kuryr
+  - Name: Installing a cluster that supports SR-IOV compute machines on OpenStack
+    File: installing-openstack-installer-sr-iov
   - Name: Installing a cluster on OpenStack on your own infrastructure
     File: installing-openstack-user
   - Name: Installing a cluster on OpenStack with Kuryr on your own infrastructure

installing/installing_openstack/installing-openstack-installer-sr-iov.adoc

@@ -0,0 +1,66 @@
+[id="installing-openstack-installer-sr-iov"]
+= Installing a cluster on OpenStack that supports SR-IOV-connected compute machines
+include::modules/common-attributes.adoc[]
+:context: installing-openstack-installer-sr-iov
+
+toc::[]
+
+In {product-title} version {product-version}, you can install a cluster on {rh-openstack-first} that can use compute machines with single-root I/O virtualization (SR-IOV) technology.
+
+== Prerequisites
+
+* Review details about the
+xref:../../architecture/architecture-installation.adoc#architecture-installation[{product-title} installation and update]
+processes.
+** Verify that {product-title} {product-version} is compatible with your {rh-openstack} version by using the "Supported platforms for OpenShift clusters" section. You can also compare platform support across different versions by viewing the link:https://access.redhat.com/articles/4679401[{product-title} on {rh-openstack} support matrix].
+// Statement seems somewhat obsolete, though not sure about assembly placement.
+* Verify that your network configuration does not rely on a provider network. Provider networks are not supported.
+
+* Have a storage service installed in {rh-openstack}, like block storage (Cinder) or object storage (Swift). Object storage is the recommended storage technology for {product-title} registry cluster deployment. For more information, see xref:../../scalability_and_performance/optimizing-storage.adoc#optimizing-storage[Optimizing storage].
+
+* Have metadata service enabled in {rh-openstack}
+
+include::modules/installation-osp-default-deployment.adoc[leveloffset=+1]
+include::modules/installation-osp-control-compute-machines.adoc[leveloffset=+2]
+include::modules/installation-osp-bootstrap-machine.adoc[leveloffset=+2]
+include::modules/cluster-entitlements.adoc[leveloffset=+1]
+include::modules/installation-osp-enabling-swift.adoc[leveloffset=+1]
+include::modules/installation-osp-verifying-external-network.adoc[leveloffset=+1]
+include::modules/installation-osp-describing-cloud-parameters.adoc[leveloffset=+1]
+include::modules/installation-obtaining-installer.adoc[leveloffset=+1]
+include::modules/installation-initializing.adoc[leveloffset=+1]
+include::modules/installation-configure-proxy.adoc[leveloffset=+2]
+include::modules/installation-configuration-parameters.adoc[leveloffset=+1]
+include::modules/installation-osp-custom-subnet.adoc[leveloffset=+2]
+include::modules/installation-osp-deploying-bare-metal-machines.adoc[leveloffset=+2]
+include::modules/installation-osp-config-yaml.adoc[leveloffset=+2]
+include::modules/ssh-agent-using.adoc[leveloffset=+1]
+include::modules/installation-osp-accessing-api.adoc[leveloffset=+1]
+include::modules/installation-osp-accessing-api-floating.adoc[leveloffset=+2]
+include::modules/installation-osp-accessing-api-no-floating.adoc[leveloffset=+2]
+include::modules/installation-osp-configuring-sr-iov.adoc[leveloffset=+1]
+include::modules/installation-launching-installer.adoc[leveloffset=+1]
+include::modules/installation-osp-verifying-cluster-status.adoc[leveloffset=+1]
+include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1]
+The cluster is operational. Before you can add SR-IOV compute machines though, you must perform additional tasks.
+
+include::modules/networking-osp-preparing-for-sr-iov.adoc[leveloffset=+1]
+include::modules/networking-osp-enabling-metadata.adoc[leveloffset=+2]
+include::modules/networking-osp-enabling-vfio-noiommu.adoc[leveloffset=+2]
+
+//Tweak copied statement, but same gist as UPI. "Day 1 is done. Now, day 2."
+The cluster is installed and prepared for SR-IOV configuration. Complete the post-installation SR-IOV tasks that are listed in the "Next steps" section.
+
+== Next steps
+
+* To complete SR-IOV configuration for your cluster:
+** xref:../../scalability_and_performance/cnf-performance-addon-operator-for-low-latency-nodes.adoc#installing-the-performance-addon-operator_cnf-master[Install the Performance Addon Operator].
+** xref:../../scalability_and_performance/what-huge-pages-do-and-how-they-are-consumed-by-apps.adoc#what-huge-pages-do_huge-pages[Configure the Performance Addon Operator with huge pages support].
+** xref:../../networking/hardware_networks/installing-sriov-operator.adoc#installing-sr-iov-operator_installing-sriov-operator[Install the SR-IOV Operator].
+** xref:../../networking/hardware_networks/configuring-sriov-device.adoc#nw-sriov-networknodepolicy-object_configuring-sriov-device[Configure your SR-IOV network device].
+** xref:../../machine_management/creating_machinesets/creating-machineset-osp.adoc#machineset-yaml-osp-sr-iov_creating-machineset-osp[Add an SR-IOV compute machine set].
+* xref:../../post_installation_configuration/cluster-tasks.adoc#available_cluster_customizations[Customize your cluster].
+* If necessary, you can
+xref:../../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[opt out of remote health reporting].
+* If you need to enable external access to node ports, xref:../../networking/configuring_ingress_cluster_traffic/configuring-ingress-cluster-traffic-nodeport.adoc#nw-using-nodeport_configuring-ingress-cluster-traffic-nodeport[configure ingress cluster traffic by using a node port].
+* If you did not configure {rh-openstack} to accept application traffic over floating IP addresses, xref:../../post_installation_configuration/network-configuration.adoc#installation-osp-configuring-api-floating-ip_post-install-network-configuration[configure {rh-openstack} access with floating IP addresses].

machine_management/creating_machinesets/creating-machineset-osp.adoc

@@ -11,7 +11,15 @@ include::modules/machine-api-overview.adoc[leveloffset=+1]
 
 include::modules/machineset-yaml-osp.adoc[leveloffset=+1]
 
+include::modules/machineset-yaml-osp-sr-iov.adoc[leveloffset=+1]
+
+.Additional resources
+
+* xref:../../installing/installing_openstack/installing-openstack-installer-sr-iov.adoc#installing-openstack-installer-sr-iov[Installing a cluster on OpenStack that supports SR-IOV-connected compute machines]
+
+include::modules/machineset-yaml-osp-sr-iov-port-security.adoc[leveloffset=+1]
+
 include::modules/machineset-creating.adoc[leveloffset=+1]
 
 // Mothballed - re-add when available
-// include::modules/machineset-osp-adding-bare-metal.adoc[leveloffset=+1]
+// include::modules/machineset-osp-adding-bare-metal.adoc[leveloffset=+1]

modules/installation-osp-control-compute-machines.adoc

@@ -2,10 +2,14 @@
 //
 // * installing/installing_openstack/installing-openstack-installer-custom.adoc
 // * installing/installing_openstack/installing-openstack-installer-kuryr.adoc
+// * installing/installing_openstack/installing-openstack-installer-sr-iov.adoc
 
 ifeval::["{context}" == "installing-openstack-user-sr-iov"]
 :osp-sr-iov:
 endif::[]
+ifeval::["{context}" == "installing-openstack-installer-sr-iov"]
+:osp-sr-iov:
+endif::[]
 
 [id="installation-osp-control-compute-machines_{context}"]
 = Control plane and compute machines
@@ -41,3 +45,6 @@ endif::osp-sr-iov[]
 ifeval::["{context}" == "installing-openstack-user-sr-iov"]
 :!osp-sr-iov:
 endif::[]
+ifeval::["{context}" == "installing-openstack-installer-sr-iov"]
+:!osp-sr-iov:
+endif::[]

modules/machineset-yaml-osp-sr-iov-port-security.adoc

@@ -0,0 +1,125 @@
+// Module included in the following assemblies:
+//
+// * machine_management/creating_machinesets/creating-machineset-osp.adoc
+
+[id="machineset-yaml-osp-sr-iov-port-security_{context}"]
+=  Sample YAML for SR-IOV deployments where port security is disabled
+
+To create single-root I/O virtualization (SR-IOV) ports on a network that has port security disabled, define a machine set that includes the ports as items in the `spec.template.spec.providerSpec.value.ports` list. This difference from the standard SR-IOV machine set is due to the automatic security group and allowed address pair configuration that occurs for ports that are created by using the network and subnet interfaces.
+
+Ports that you define for machines subnets require:
+
+* Allowed address pairs for the API and ingress virtual IP ports
+* The compute security group
+* Attachment to the machines network and subnet
+
+[NOTE]
+====
+Only parameters that are specific to SR-IOV deployments where port security is disabled are described in this sample. To review a more general sample, see Sample YAML for a machine set custom resource that uses SR-IOV on {rh-openstack}".
+====
+
+.An example machine set that uses SR-IOV networks and has port security disabled
+[source,yaml]
+----
+apiVersion: machine.openshift.io/v1beta1
+kind: MachineSet
+metadata:
+  labels:
+    machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
+    machine.openshift.io/cluster-api-machine-role: <node_role>
+    machine.openshift.io/cluster-api-machine-type: <node_role>
+  name: <infrastructure_ID>-<node_role>
+  namespace: openshift-machine-api
+spec:
+  replicas: <number_of_replicas>
+  selector:
+    matchLabels:
+      machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
+      machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
+  template:
+    metadata:
+      labels:
+        machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
+        machine.openshift.io/cluster-api-machine-role: <node_role>
+        machine.openshift.io/cluster-api-machine-type: <node_role>
+        machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
+    spec:
+      metadata: {}
+      providerSpec:
+        value:
+          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
+          cloudName: openstack
+          cloudsSecret:
+            name: openstack-cloud-credentials
+            namespace: openshift-machine-api
+          flavor: <nova_flavor>
+          image: <glance_image_name_or_location>
+          kind: OpenstackProviderSpec
+          ports:
+            - allowedAddressPairs: <1> 
+              - ipAddress: <API_VIP_port_IP>
+              - ipAddress: <ingress_VIP_port_IP>
+              fixedIPs:
+                - subnetID: <machines_subnet_UUID> <2>
+              nameSuffix: nodes
+              networkID: <machines_network_UUID> <2>
+              securityGroups:
+                  - <compute_security_group_UUID> <3>
+            - networkID: <SRIOV_network_UUID>
+              nameSuffix: sriov
+              fixedIPs:
+                - subnetID: <SRIOV_subnet_UUID>
+              tags:
+                - sriov
+              vnicType: direct
+              portSecurity: False
+          primarySubnet: <machines_subnet_UUID>
+          serverMetadata:
+            Name: <infrastructure_ID>-<node_role>
+            openshiftClusterID: <infrastructure_ID>
+          tags:
+          - openshiftClusterID=<infrastructure_ID>
+          trunk: false
+          userDataSecret:
+            name: worker-user-data
+          configDrive: True
+----
+<1> Specify allowed address pairs for the API and ingress ports.
+<2> Specify the machines network and subnet.
+<3> Specify the compute machines security group.
+
+[NOTE]
+====
+Trunking is enabled for ports that are created by entries in the networks and subnets lists. The name of ports that are created from these lists follow the pattern `<machine_name>-<nameSuffix>`. The `nameSuffix` field is required in port definitions.
+
+Trunking is not enabled for ports that are defined in the ports list.
+
+Optionally, you can add tags to ports as part of their `tags` lists.
+====
+
+If your cluster uses Kuryr and the {rh-openstack} SR-IOV network has port security disabled, the primary port for compute machines must have:
+
+* The value of the `spec.template.spec.providerSpec.value.networks.portSecurityEnabled` parameter set to `false`.
+
+* For each subnet, the value of the `spec.template.spec.providerSpec.value.networks.subnets.portSecurityEnabled` parameter set to `false`.
+
+* The value of `spec.template.spec.providerSpec.value.securityGroups` set to empty: `[]`.
+
+.An example section of a machine set for a cluster on Kuryr that uses SR-IOV and has port security disabled
+[source,yaml]
+----
+...
+          networks:
+            - subnets:
+              - uuid: <machines_subnet_UUID>
+                portSecurityEnabled: false
+              portSecurityEnabled: false
+          securityGroups: []
+...
+----
+
+In that case, you can apply the compute security group to the primary VM interface after the VM is created. For example, from a command line:
+[source,terminal]
+----
+$ openstack port set --enable-port-security --security-group <infrastructure_ID>-<node_role> <main_port_ID>
+----

modules/machineset-yaml-osp-sr-iov.adoc

@@ -0,0 +1,111 @@
+// Module included in the following assemblies:
+//
+// * machine_management/creating_machinesets/creating-machineset-osp.adoc
+
+[id="machineset-yaml-osp-sr-iov_{context}"]
+=  Sample YAML for a machine set custom resource that uses SR-IOV on {rh-openstack}
+
+If you configured your cluster for single-root I/O virtualization (SR-IOV), you can create machine sets that use that technology.
+
+This sample YAML defines a machine set that uses SR-IOV networks. The nodes that it creates are labeled with `node-role.openshift.io/<node_role>: ""`
+
+In this sample, `infrastructure_ID` is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and `node_role` is the node label to add.
+
+The sample assumes two SR-IOV networks that are named "radio" and "uplink". The networks are used in port definitions in the `spec.template.spec.providerSpec.value.ports` list.
+
+[NOTE]
+====
+Only parameters that are specific to SR-IOV deployments are described in this sample. To review a more general sample, see "Sample YAML for a machine set custom resource on {rh-openstack}".
+====
+
+.An example machine set that uses SR-IOV networks
+[source,yaml]
+----
+apiVersion: machine.openshift.io/v1beta1
+kind: MachineSet
+metadata:
+  labels:
+    machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
+    machine.openshift.io/cluster-api-machine-role: <node_role>
+    machine.openshift.io/cluster-api-machine-type: <node_role>
+  name: <infrastructure_ID>-<node_role>
+  namespace: openshift-machine-api
+spec:
+  replicas: <number_of_replicas>
+  selector:
+    matchLabels:
+      machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
+      machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
+  template:
+    metadata:
+      labels:
+        machine.openshift.io/cluster-api-cluster: <infrastructure_ID>
+        machine.openshift.io/cluster-api-machine-role: <node_role>
+        machine.openshift.io/cluster-api-machine-type: <node_role>
+        machine.openshift.io/cluster-api-machineset: <infrastructure_ID>-<node_role>
+    spec:
+      metadata:
+      providerSpec:
+        value:
+          apiVersion: openstackproviderconfig.openshift.io/v1alpha1
+          cloudName: openstack
+          cloudsSecret:
+            name: openstack-cloud-credentials
+            namespace: openshift-machine-api
+          flavor: <nova_flavor>
+          image: <glance_image_name_or_location>
+          serverGroupID: <optional_UUID_of_server_group>
+          kind: OpenstackProviderSpec
+          networks:
+            - subnets:
+              - UUID: <machines_subnet_UUID>
+          ports:
+            - networkID: <radio_network_UUID> <1>
+              nameSuffix: radio
+              fixedIPs:
+                - subnetID: <radio_subnet_UUID> <2>
+              tags:
+                - sriov
+                - radio
+              vnicType: direct <3>
+              portSecurity: false <4>
+            - networkID: <uplink_network_UUID> <1>
+              nameSuffix: uplink
+              fixedIPs:
+                - subnetID: <uplink_subnet_UUID> <2>
+              tags:
+                - sriov
+                - uplink
+              vnicType: direct <3>
+              portSecurity: false <4>
+          primarySubnet: <machines_subnet_UUID>
+          securityGroups:
+          - filter: {}
+            name: <infrastructure_ID>-<node_role>
+          serverMetadata:
+            Name: <infrastructure_ID>-<node_role>
+            openshiftClusterID: <infrastructure_ID>
+          tags:
+          - openshiftClusterID=<infrastructure_ID>
+          trunk: true
+          userDataSecret:
+            name: <node_role>-user-data
+          availabilityZone: <optional_openstack_availability_zone>
+          configDrive: true <5>
+----
+<1> Enter a network UUID for each port.
+<2> Enter a subnet UUID for each port.
+<3> The value of the `vnicType` parameter must be `direct` for each port.
+<4> The value of the `portSecurity` parameter must be `false` for each port.
++
+You cannot set security groups and allowed address pairs for ports when port security is disabled. Setting security groups on the instance applies the groups to all ports that are attached to it.
+<5> The value of the `configDrive` parameter must be `true`.
+
+[NOTE]
+====
+Trunking is enabled for ports that are created by entries in the networks and subnets lists. The name of ports that are created from these lists follow the pattern `<machine_name>-<nameSuffix>`. The `nameSuffix` field is required in port definitions.
+
+Trunking is not enabled for ports that are defined in the ports list.
+
+Optionally, you can add tags to ports as part of their `tags` lists.
+====