WIP Add doc for the SR-IOV Network Operator by pliurh · Pull Request #16393 · openshift/openshift-docs

pliurh · 2019-08-27T07:18:20Z

https://jira.coreos.com/browse/SDN-416

zshi-redhat

It might be worth having a few examples of Pod spec for using SR-IOV in different mode, kernel mode, userspace mode and RDMA.

zshi-redhat · 2019-08-27T09:31:15Z

+Resources, which are generated and updated by the operator automatically. 
+
+One CR is created for each worker node, and shares the same name as the node. In
+the interface list, you can the information of the network devices.  


missing word at you can <> the information

zshi-redhat · 2019-08-27T09:34:04Z

+<1> The name of the CR.
+<2> The namespace of the CR, this must be in the same namespace of operator.
+<3> The resource name of SR-IOV device plugin. Prefix `openshift.io/` will be
+add when it's referred in pod annotation. It's allowed to create multiple CRs of `SriovNetworkNodePolicy` for one resource name.


add -> added
pod annotation -> pod spec? or net-attach-def ?

zshi-redhat · 2019-08-27T09:36:56Z

+The SR-IOV Network Operator also introduces a CRD `SriovNetwork` for creating
+the Custom Resource of `NetworkAttachmentDefinition` of SR-IOV CNI plugin. When
+you create a CR of `SriovNetwork`, the operator will created a CR of
+`NetworkAttachmentDefinition` according.


accordingly

zshi-redhat · 2019-08-27T09:38:10Z

+
+[NOTE]
+=====
+You shall not modify or delete a Custom `SriovNetwork`, when it has been used by


Custom 'SriovNetwork' -> 'SriovNetwork' Custom Resource.

zshi-redhat · 2019-08-27T09:38:43Z

+=====
+
+The following is an example of a typical Custom Resource for
+`SriovNetworkNodePolicy`.


'SriovNetworkNodePolicy' -> 'SriovNetwork'

zshi-redhat · 2019-08-27T09:43:32Z

+      "rangeStart": "10.56.217.171",
+      "rangeEnd": "10.56.217.181",
+      "routes": [{
+        "dst": "10.0.0.0/8"


overwrite default route is not possible with current ipam plugin, perhaps add an example of non-default route address.

zshi-redhat · 2019-08-27T09:46:49Z

+Range from 0 to 99.
+<6> The MTU of the virtual functions. Range from 1 to 9000. Leave it blank if
+you don't need to change the MTU.
+<7> The number of the virtual functions for each SR-IOV network device.


for each SR-IOV network device => will be created for each SR-IOV physical network device

zshi-redhat · 2019-08-27T09:54:18Z

+<6> The MTU of the virtual functions. Range from 1 to 9000. Leave it blank if
+you don't need to change the MTU.
+<7> The number of the virtual functions for each SR-IOV network device.
+<8> The vendor hex code of SR-IoV device. Allowed value "8086", "15b3".


might be good to add note that

it's not necessary to fill all nicSelector filters in every policy

some nicSelector filters may conflict with each other if not configured properly, such as pfNames and rootDevices.

zshi-redhat · 2019-08-27T10:04:31Z

+<11> The PCI addresses of SR-IoV physical function.
+<12> The driver type of the virtual functions. Allowed value "netdevice",
+"vfio-pci". Defaults to "netdevice".
+<13> The RDMA mode. Defaults to false.


This may be worth more explanation or notes on

which types of NIC are supported for RDMA mode in current release,

supported RDMA mode is RoCE.

consequence of configuring this option, it should not affect other non-RDMA capable network devices be discovered and used with the same resource name, this just enables SR-IOV device plugin to attach additional RDMA device spec (/dev/infiniband etc) when RDMA device is allocated.

I don' think we shall suggest user to use the same resource name for both RDMA and non-RDMA devices even it is doable, shall we? Since normally the RDMA devices shall be in a separated network.

When RDMA flag is configured to true, it doesn't prevent user from using RDMA enabled VF as a network device. A device can be used in either mode.

Note added.

zshi-redhat · 2019-08-27T10:08:35Z

+* Initialize the supported SR-IOV NIC models on nodes.
+* Provision SR-IOV network device plugin on nodes.
+* Provision SR-IOV CNI plugin executable on nodes.
+* Manage configuration of SR-IOV device plugin.


SR-IOV device plugin -> SR-IOV network device plugin

zshi-redhat · 2019-08-27T10:19:26Z

This PR is created against openshift-docs:master, shall we do that in 4.2 branch? I didn't see the files in this PR included in modules directory on master branch.

jboxman · 2019-08-28T01:33:45Z

Hi @zshi-redhat! For 4.2 features, a PR against master is the correct approach. Then we'll cherry-pick the (squashed) commit into the 4.2 branch.

@pliurh thanks for opening this PR!

jboxman · 2019-08-29T22:24:26Z

@vikram-redhat, what's the process for this? It's contributed content for a 4.2 feature. It's related to Multus, but SR-IOV is more complex so the feature includes an Operator. For docs this epic was considered out of scope.

vikram-redhat · 2019-08-30T02:18:50Z

@jboxman if it passes QE and peer review (from you), then it will be acceptable.

Confirm with @aheslin.

zshi-redhat · 2019-09-02T07:20:48Z

 kind: Namespace
 metadata:
-  name: openshift-sriov
+  name: openshift-sriov-network-operator


This namespace name depends on openshift/sriov-network-operator#70

change namespace back to 'sriov-network-operator' for 4.2. https://bugzilla.redhat.com/show_bug.cgi?id=1746222

zshi-redhat · 2019-09-02T07:30:04Z

+<11> The PCI addresses of SR-IoV physical function.
+<12> The driver type of the virtual functions. Allowed value "netdevice",
+"vfio-pci". Defaults to "netdevice".
+<13> The RDMA mode. Defaults to false.


When RDMA flag is configured to true, it doesn't prevent user from using RDMA enabled VF as a network device. A device can be used in either mode.

zshi-redhat · 2019-09-02T07:37:07Z

 * The SR-IOV CNI plug-in plumbs VF interfaces allocated from the SR-IOV device
 plug-in directly into a Pod.

+You can use the {product-title} console to install SR-IOV, by deploying,


SR-IOV Operator will be the only interface user needs to interact with, perhaps move above two paragraphs about sriov network device plugin and sriov cni after introducing all the operator features. For example, maybe have detailed introduction of each component that managed by SR-IOV operator after listing the operator features.

zshi-redhat · 2019-09-02T07:38:21Z

+if a NetworkAttachmentDefinition CR of SRIOV CNI plugin is referred in the Pod
+annotation.
+=====
+


Perhaps adding pod examples for both dpdk and rdma mode.

jboxman · 2019-09-12T19:21:36Z

@pliurh

If you enable Allow edits from maintainers. I can edit this PR directly:

https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork

(If it's enabled already, I can't tell. GitHub doesn't say.)

pliurh · 2019-09-13T05:27:55Z

@pliurh

If you enable Allow edits from maintainers. I can edit this PR directly:

https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork

(If it's enabled already, I can't tell. GitHub doesn't say.)

It's enabled already.

jboxman · 2019-09-13T17:45:48Z

Perfect. Then I'll push any updates back to the PR for your review.

jboxman · 2019-09-17T12:18:57Z

Ended up with #16679 which is not what I intended; I may ultimately create a clean PR with the proposed changes. GitHub makes it inconvenient to push to someone else's PR.

vikram-redhat · 2020-01-06T05:17:08Z

@jboxman can this be closed?

openshift-ci-robot · 2020-01-06T05:17:14Z

@pliurh: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jboxman · 2020-01-06T15:54:40Z

Merged in #16679.

[WIP]Add doc for the SR-IOV Network Operator

067a94e

https://jira.coreos.com/browse/SDN-416

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 27, 2019

zshi-redhat reviewed Aug 27, 2019

View reviewed changes

Comment thread modules/nw-multinetwork-sriov.adoc

jboxman added this to the Future Release milestone Aug 28, 2019

jboxman added the branch/enterprise-4.2 label Aug 28, 2019

Update based on the comments

99ae42f

openshift-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 29, 2019

zshi-redhat reviewed Sep 2, 2019

View reviewed changes

Add RDMA and DPDK example, change namespace to sriov-network-operator

f9f2340

pliurh changed the title ~~[WIP]Add doc for the SR-IOV Network Operator~~ Add doc for the SR-IOV Network Operator Sep 9, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2019

jboxman self-assigned this Sep 13, 2019

jboxman changed the title ~~Add doc for the SR-IOV Network Operator~~ WIP Add doc for the SR-IOV Network Operator Sep 20, 2019

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 20, 2019

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 6, 2020

jboxman closed this Jan 6, 2020

vikram-redhat removed the branch/enterprise-4.2 label Jan 23, 2020

vikram-redhat removed this from the Future Release milestone Jan 23, 2020

Conversation

pliurh commented Aug 27, 2019

Uh oh!

zshi-redhat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zshi-redhat commented Aug 27, 2019

Uh oh!

jboxman commented Aug 28, 2019

Uh oh!

jboxman commented Aug 29, 2019

Uh oh!

vikram-redhat commented Aug 30, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jboxman commented Sep 12, 2019

Uh oh!

pliurh commented Sep 13, 2019

Uh oh!

jboxman commented Sep 13, 2019

Uh oh!

jboxman commented Sep 17, 2019