SR-IOV Network Operator configuration by jboxman · Pull Request #16679 · openshift/openshift-docs

jboxman · 2019-09-17T11:56:52Z

So this ended up becoming the working PR for this.

This PR adds content for using the SR-IOV Network Operator.

openshift-docs-preview-bot · 2019-09-17T11:59:47Z

The preview will be available shortly at:

https://pliurh-sriov-network-operator--ocpdocs.netlify.com/

pliurh · 2019-09-18T04:36:36Z

s/Following/following/

zshi-redhat · 2019-09-19T01:07:47Z

Do we have an estimated time on how long the installation would take? perhaps give a number here so that user has expectation on how long they shall wait.

It mainly depends on how long the operator images can be downloaded from Internet, which it hard to predict in customer's environment.

zshi-redhat · 2019-09-19T01:14:16Z

Shall we have a explanation for interfaces field, followed by this example?
for example, all the properties are for physical functions.
We have vendorID and deviceID mentioned in several places in the doc, but they may refer to PF or VF IDs depending on whether they are in a status field of SriovNetworkNodeState CR or in the spec field of SriovNetworkNodePolicy CR.

The field under interfaces is quite straightforward. User shall be able to know what it is through the name.

zshi-redhat · 2019-09-19T01:20:56Z

Might be good to explicitly mention VF in SR-IoV device, for example, use SR-IoV VF device

The device ID should be PF's not VF's.
s/device hex/SR-IOV PF device hex/

zshi-redhat · 2019-09-19T01:31:36Z

Is there a simple command to query which SriovNetwork are in use by a particular pod?

I don't think there is any.

zshi-redhat · 2019-09-19T01:36:44Z

Is there any pre-requisite to meet before installing SR-IOV Operator?
For example, we only support SR-IOV deployment on baremetal environment, might want to call out this as a pre-requisite in a separate section. Other I can think of is when using SR-IOV in userspace mode (DPDK), user might want to configure hugepages which is not supported by SR-IOV Operator itself, worth a link to MCO doc for how to configure kernel args etc.

zshi-redhat · 2019-09-19T01:38:29Z

Since we are giving an pod example here, might be good to also include the example of net-attach-def CR(sriov-rdma-mlnx) used here. This applies to dpdk example below as well.

zshi-redhat · 2019-09-19T01:39:46Z

Might be good to add cpu in resources requests and limits because dpdk application always need to bind cpu.

resources: limits: memory: "1Gi" cpu: "2" requests: memory: "1Gi" cpu: "2"

I updated to this:

resources: limits: memory: "1Gi" cpu: "2" hugepages-1Gi: "4Gi" requests: memory: "1Gi" cpu: "2" hugepages-1Gi: "4Gi"

Is that okay? Or should hugepages be removed?

I updated to this:

resources: limits: memory: "1Gi" cpu: "2" hugepages-1Gi: "4Gi" requests: memory: "1Gi" cpu: "2" hugepages-1Gi: "4Gi"

Is that okay? Or should hugepages be removed?

I think it's ok. DPDK application requires hugepages.

Please add a note with link to kernel argument configurations (1G hugepage needs to be configured via kernel arguments, this config is supported via MCO). for example: https://github.com/openshift/openshift-docs/blob/enterprise-4.2/modules/nodes-nodes-kernel-arguments.adoc

Please also add a note for application using 2Mi hugepages (this doesn't require changing of kernel argument and can be configured via tuned Operator), link to the doc: https://github.com/openshift/openshift-docs/blob/enterprise-4.2/modules/configuring-huge-pages.adoc

Adding kernel args: https://docs.openshift.com/container-platform/4.2/nodes/nodes/nodes-nodes-working.html#nodes-nodes-kernel-arguments_nodes-nodes-working

Tuned: https://docs.openshift.com/container-platform/4.2/scalability_and_performance/using-node-tuning-operator.html

zhaozhanqi · 2019-09-23T06:47:16Z

I did not see isRdma option in the SriovNetworkNodePolicy CR in my env.

isRdma is empty if not set : https://github.com/openshift/sriov-network-operator/blob/master/pkg/apis/sriovnetwork/v1/sriovnetworknodepolicy_types.go#L36
If you take a look at the device plugin configMap generated by operator, it shall be configured as False if not set.

@zhaozhanqi is this still an issue in your test?

@zshi-redhat yes, as you said, it's the false by default.
but I think we should add this one to template, here is the default template for now:

apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: policy-1 namespace: sriov-network-operator spec: resourceName: intelnics nodeSelector: feature.node.kubernetes.io/sriov-capable: 'true' priority: 99 mtu: 9000 numVfs: 6 nicSelector: vendor: '8086' rootDevices: - '0000:01:00.1' pfNames: - eth1 deviceType: vfio-pci

zshi-redhat · 2019-09-25T06:47:48Z

This should be VF device IDs according to my testing, otherwise device plugin will expose PF as resource.

And we may also want to list a mapping of VF IDs to supported NICs so that user knows what to configure in node policy custom resource.

This should be VF device IDs according to my testing, otherwise device plugin will expose PF as resource.

It's a bug, the deviceID shall always be set with the PF's ID. The operator shall convert it to the VF device ID when generating the device plugin configuration, if the numVFs is set. cc @zhaozhanqi

@zhaozhanqi please help to confirm if that's the case in your testing.

this should be PF deviceID since we do not know the vf device id yet before the nodenetworkpolicy is created.

- Vfs: - deviceID: 154c driver: iavf mtu: 1500 pciAddress: '0000:3b:02.0' vendor: '8086' - deviceID: 154c driver: iavf mtu: 1500 pciAddress: '0000:3b:02.1' vendor: '8086' - deviceID: 154c driver: iavf mtu: 1500 pciAddress: '0000:3b:02.2' vendor: '8086' - deviceID: 154c driver: iavf mtu: 1500 pciAddress: '0000:3b:02.3' vendor: '8086' - deviceID: 154c driver: iavf mtu: 1500 pciAddress: '0000:3b:02.4' vendor: '8086' - deviceID: 154c driver: iavf mtu: 1500 pciAddress: '0000:3b:02.5' vendor: '8086' driver: i40e vendor: '8086' name: ens1f0 mtu: 1500 deviceID: 158b numVfs: 6 pciAddress: '0000:3b:00.0' totalvfs: 64

The deviceID do not be set in the dpconfig if not set 'deviceID' in the policy. and the numVFs is also not be set
oc get configmap -n sriov-network-operator device-plugin-config -o yaml apiVersion: v1 data: config.json: '{"resourceList":[{"resourceName":"intelnetdevice","IsRdma":false,"selectors":{"vendors":["8086"],"drivers":["iavf","mlx5_core","i40evf","ixgbevf"],"pfNames":["ens1f0"]}}]}' kind: ConfigMap

zshi-redhat · 2019-09-28T01:56:20Z

@jboxman this PR may need another commit to fix opening comments. can you help on it?

jboxman · 2019-10-09T19:44:25Z

@zshi-redhat, I'm going to work through the comments here; I may open a new PR with the updated content after for a clean review.

zhaozhanqi · 2019-10-18T03:19:46Z

hi, @jboxman
I saw the 4.2 docs for sriov is still using old way to setup sriov. https://docs.openshift.com/container-platform/4.2/networking/multiple-networks/configuring-sr-iov.html
Why this PR did not been merged?

jboxman · 2019-10-18T14:53:29Z

@zhaozhanqi, it doesn't conform to our contributing guide, so I've been rewriting it; the work for that will be complete in a few days. Thanks for your patience.

jboxman · 2019-10-22T18:56:38Z

@pliurh

What is a resourceName in this context?

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
spec:
  resourceName: <sriov_resource_name>

jboxman · 2019-10-25T17:40:04Z

@pliurh @zshi-redhat @zhaozhanqi

I've edited and reorganized the content. Can you take a look? Thanks!

zshi-redhat

Thanks @jboxman for the update!
some minor comments inline.

pliurh · 2019-10-29T15:04:57Z

@pliurh

What is a resourceName in this context?

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
spec:
  resourceName: <sriov_resource_name>

The resourceName is a logic name for a group of sriov VFs. It can be referenced by the manifests of pod and net-att-def, if user want to allocate the VF to a pod.

jboxman · 2019-10-29T20:34:18Z

@zshi-redhat thanks! I've made those changes.

@zhaozhanqi, can you confirm the content for this PR is okay? Thanks!

pliurh · 2019-10-31T06:48:30Z

Add, "For Intel NIC, this value shall not be larger than the TotalVFs. For Mellanox NIC, this value shall not be larger than 128."

@pliurh, what is TotalVFs in this context? Is it a configuration parameter in the CR? Or some fixed value for Intel?

TotalVFs is the maximum number VF can be configured. For Intel NICs, it’s a static value. For Mellanox NICs it is variable with max value 128 for the NICs we claim to support, so it can be changed by the sriov operator when user specify the numVFs.

jboxman · 2019-11-07T20:52:29Z

@zshi-redhat @pliurh @zhaozhanqi

I've made the requested changes; PTAL. Thanks!

pliurh · 2019-11-08T02:33:20Z

LGTM

zshi-redhat · 2019-11-08T04:56:21Z

/lgtm

jboxman · 2019-11-08T20:38:00Z

I'm going to merge this, at long last! We can iterate on it more if necessary under cover of a new PR.

cc @aburdenthehand @lamek

jboxman · 2019-11-08T20:44:23Z

/cherry-pick enterprise-4.3

openshift-cherrypick-robot · 2019-11-08T20:44:28Z

@jboxman: #16679 failed to apply on top of branch "enterprise-4.3":

.git/rebase-apply/patch:455: trailing whitespace.
<6> Specify a value for the maximum transmission unit (MTU) of the virtual function. The value for MTU must be in the range from `1` to `9000`. If you do not need to specify the MTU, specify a value of `''`. 
.git/rebase-apply/patch:715: new blank line at EOF.
+
warning: 2 lines add whitespace errors.
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	modules/nw-multinetwork-sriov.adoc
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): modules/nw-multinetwork-sriov.adoc deleted in Add content for SR-IOV Network Operator and modified in HEAD. Version HEAD of modules/nw-multinetwork-sriov.adoc left in tree.
Patch failed at 0001 Add content for SR-IOV Network Operator

Details

In response to this:

/cherry-pick enterprise-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jboxman · 2019-11-08T20:45:05Z

/cherry-pick enterprise-4.2

openshift-cherrypick-robot · 2019-11-08T20:45:09Z

@jboxman: #16679 failed to apply on top of branch "enterprise-4.2":

.git/rebase-apply/patch:455: trailing whitespace.
<6> Specify a value for the maximum transmission unit (MTU) of the virtual function. The value for MTU must be in the range from `1` to `9000`. If you do not need to specify the MTU, specify a value of `''`. 
.git/rebase-apply/patch:715: new blank line at EOF.
+
warning: 2 lines add whitespace errors.
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	modules/nw-multinetwork-sriov.adoc
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): modules/nw-multinetwork-sriov.adoc deleted in Add content for SR-IOV Network Operator and modified in HEAD. Version HEAD of modules/nw-multinetwork-sriov.adoc left in tree.
Patch failed at 0001 Add content for SR-IOV Network Operator

Details

In response to this:

/cherry-pick enterprise-4.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 17, 2019

jboxman mentioned this pull request Sep 17, 2019

WIP Add doc for the SR-IOV Network Operator #16393

Closed

pliurh reviewed Sep 18, 2019

View reviewed changes

zshi-redhat reviewed Sep 19, 2019

View reviewed changes

zhaozhanqi reviewed Sep 23, 2019

View reviewed changes

zshi-redhat reviewed Sep 25, 2019

View reviewed changes

jboxman mentioned this pull request Sep 26, 2019

OSDOCS-475: Update content for multiple networks #16349

Merged

zhaozhanqi reviewed Sep 30, 2019

View reviewed changes

Comment thread modules/nw-multinetwork-sriov.adoc Outdated

zhaozhanqi reviewed Sep 30, 2019

View reviewed changes

Comment thread modules/nw-multinetwork-sriov.adoc Outdated

zhaozhanqi reviewed Sep 30, 2019

View reviewed changes

Comment thread modules/nw-multinetwork-sriov.adoc Outdated

jboxman force-pushed the pliurh-sriov-network-operator branch from 46112d9 to 9e2c742 Compare October 9, 2019 22:25

jboxman self-assigned this Oct 9, 2019

jboxman force-pushed the pliurh-sriov-network-operator branch from 9e2c742 to b790a73 Compare October 9, 2019 22:43

jboxman added branch/enterprise-4.2 branch/enterprise-4.3 labels Oct 16, 2019

jboxman changed the title ~~WIP Pliurh sriov network operator~~ WIP SR-IOV Network Operator configuration Oct 20, 2019

jboxman force-pushed the pliurh-sriov-network-operator branch from b790a73 to 4a6c27e Compare October 25, 2019 17:36

jboxman changed the title ~~WIP SR-IOV Network Operator configuration~~ SR-IOV Network Operator configuration Oct 25, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 25, 2019

zshi-redhat reviewed Oct 29, 2019

View reviewed changes

Comment thread modules/nw-sriov-configuring-device.adoc Outdated

Comment thread modules/nw-sriov-configuring-device.adoc Outdated

Comment thread modules/nw-sriov-network-attachment.adoc Outdated

Comment thread networking/multiple-networks/configuring-sr-iov.adoc Outdated

jboxman force-pushed the pliurh-sriov-network-operator branch from 4a6c27e to 9119712 Compare October 29, 2019 20:19

zhaozhanqi reviewed Oct 30, 2019

View reviewed changes

Comment thread modules/nw-sriov-network-attachment.adoc Outdated

pliurh reviewed Oct 31, 2019

View reviewed changes

Add content for SR-IOV Network Operator

ce83395

jboxman force-pushed the pliurh-sriov-network-operator branch from 9119712 to ce83395 Compare November 7, 2019 20:48

openshift-ci-robot assigned zshi-redhat Nov 8, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 8, 2019

jboxman merged commit 3d05385 into openshift:master Nov 8, 2019

This was referenced Nov 8, 2019

[enterprise-4.3] Add content for SR-IOV Network Operator #18029

Merged

[enterprise-4.2] Add content for SR-IOV Network Operator #18030

Merged

jboxman deleted the pliurh-sriov-network-operator branch November 8, 2019 20:55

Conversation

jboxman commented Sep 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-docs-preview-bot commented Sep 17, 2019

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jboxman Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jboxman Oct 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhaozhanqi Sep 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhaozhanqi Sep 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pliurh Sep 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhaozhanqi Sep 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jboxman commented Sep 17, 2019 •

edited

Loading

jboxman Sep 20, 2019 •

edited

Loading

jboxman Oct 9, 2019 •

edited

Loading

zhaozhanqi Sep 23, 2019 •

edited

Loading

zhaozhanqi Sep 29, 2019 •

edited

Loading

pliurh Sep 27, 2019 •

edited

Loading

zhaozhanqi Sep 29, 2019 •

edited

Loading

zhaozhanqi Sep 29, 2019 •

edited

Loading

zshi-redhat commented Sep 28, 2019 •

edited

Loading