Device Network data in Status is not updated correctly

[aojea]

It seems only one device gets updated

kubectl get resourceclaims -o yaml
apiVersion: v1
items:
- apiVersion: resource.k8s.io/v1beta1
  kind: ResourceClaim
  metadata:
    annotations:
      resource.kubernetes.io/pod-claim-name: rdma-net-interface
    creationTimestamp: "2025-06-17T22:59:48Z"
    finalizers:
    - resource.kubernetes.io/delete-protection
    generateName: nccl-gib-test-0-rdma-net-interface-
    name: nccl-gib-test-0-rdma-net-interface-8dj8b
    namespace: default
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Pod
      name: nccl-gib-test-0
      uid: 0c344e45-9866-4c3f-bc3f-44e8b664bbbd
    resourceVersion: "1750201254435023024"
    uid: 18317a3d-300d-41e3-ba99-4fe78f425d21
  spec:
    devices:
      requests:
      - allocationMode: ExactCount
        count: 8
        deviceClassName: dranet
        name: rdma-net-interface
        selectors:
        - cel:
            expression: device.attributes["dra.net"].rdma == true
  status:
    allocation:
      devices:
        results:
        - adminAccess: null
          device: gpu0rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu1rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu2rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu3rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu4rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu5rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu6rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
        - adminAccess: null
          device: gpu7rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
          request: rdma-net-interface
      nodeSelector:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
    devices:
    - conditions:
      - lastTransitionTime: "2025-06-17T23:00:54Z"
        message: ""
        reason: NetworkDeviceReady
        status: "True"
        type: Ready
      - lastTransitionTime: "2025-06-17T23:00:54Z"
        message: ""
        reason: NetworkReady
        status: "True"
        type: NetworkReady
      device: gpu7rdma0
      driver: dra.net
      networkData:
        hardwareAddress: 62:55:7a:a9:36:16
        interfaceName: gpu7rdma0
        ips:
        - 192.168.8.4/32
      pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-lp53
    reservedFor:
    - name: nccl-gib-test-0
      resource: pods
      uid: 0c344e45-9866-4c3f-bc3f-44e8b664bbbd
- apiVersion: resource.k8s.io/v1beta1
  kind: ResourceClaim
  metadata:
    annotations:
      resource.kubernetes.io/pod-claim-name: rdma-net-interface
    creationTimestamp: "2025-06-17T23:00:55Z"
    finalizers:
    - resource.kubernetes.io/delete-protection
    generateName: nccl-gib-test-1-rdma-net-interface-
    name: nccl-gib-test-1-rdma-net-interface-j5gzp
    namespace: default
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Pod
      name: nccl-gib-test-1
      uid: 09bdeedc-9981-4b04-9dc6-0fb687b07a2f
    resourceVersion: "1750201286238063000"
    uid: 5eb63421-14fe-4ec8-8903-c74fb2c00143
  spec:
    devices:
      requests:
      - allocationMode: ExactCount
        count: 8
        deviceClassName: dranet
        name: rdma-net-interface
        selectors:
        - cel:
            expression: device.attributes["dra.net"].rdma == true
  status:
    allocation:
      devices:
        results:
        - adminAccess: null
          device: gpu0rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu1rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu2rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu3rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu4rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu5rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu6rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
        - adminAccess: null
          device: gpu7rdma0
          driver: dra.net
          pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
          request: rdma-net-interface
      nodeSelector:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
    devices:
    - conditions:
      - lastTransitionTime: "2025-06-17T23:01:26Z"
        message: ""
        reason: NetworkDeviceReady
        status: "True"
        type: Ready
      - lastTransitionTime: "2025-06-17T23:01:26Z"
        message: ""
        reason: NetworkReady
        status: "True"
        type: NetworkReady
      device: gpu6rdma0
      driver: dra.net
      networkData:
        hardwareAddress: 92:77:04:06:a2:13
        interfaceName: gpu6rdma0
        ips:
        - 192.168.7.5/32
      pool: gke-dranet-maspinwal-dranet-maspinwal-d3003787-zwtv
    reservedFor:
    - name: nccl-gib-test-1
      resource: pods
      uid: 09bdeedc-9981-4b04-9dc6-0fb687b07a2f
- apiVersion: resource.k8s.io/v1beta1
  kind: ResourceClaim
  metadata:
    annotations:
      resource.kubernetes.io/pod-claim-name: rdma-net-interface
    creationTimestamp: "2025-06-17T23:15:16Z"
    generateName: nccl-gib-test-4w-0-rdma-net-interface-
    name: nccl-gib-test-4w-0-rdma-net-interface-7f9gq
    namespace: default
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Pod
      name: nccl-gib-test-4w-0
      uid: e93f4e36-ae83-4701-85f6-dfeb25bdfe38
    resourceVersion: "1750202116244175024"
    uid: c7b82de3-bb74-484d-b769-eb3d1a3edcb2
  spec:
    devices:
      requests:
      - allocationMode: ExactCount
        count: 4
        deviceClassName: dranet
        name: rdma-net-interface
        selectors:
        - cel:
            expression: |
              device.attributes["dra.net"].rdma == true &&
              (
                (device.attributes["dra.net"].ifName.startsWith("gpu") &&
                 device.attributes["dra.net"].ifName.endsWith("rdma0") &&
                 int(device.attributes["dra.net"].ifName.substring(3, 4)) < 4)
              ||
                (device.attributes["dra.net"].ifName.startsWith("gpu") &&
                 device.attributes["dra.net"].ifName.endsWith("rdma0") &&
                 int(device.attributes["dra.net"].ifName.substring(3, 4)) >= 4)
              )
  status: {}
kind: List
metadata:

[michaelasp]

Looks like some issue with SSA not merging the lists

https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/resource/types.go#L1136-L1143 shows that the keys of the list should include the device name

However testing locally k apply -f test.yaml --subresource=status --server-side caused my values to get overwritten despite changing the keys. This seems like a bug, unless I'm misunderstanding the above?

[gauravkghildiyal]

I believe this is working as intended. See https://kubernetes.io/docs/reference/using-api/server-side-apply/#merge-strategy, with special emphasis on "applied by the same user"

And you would also be correct to think that the problem here is that we do multiple status "apply" for each device concurrently with the end result being that only the last to apply persists:

dranet/pkg/driver/nri_hooks.go

Lines 200 to 213 in 66de5dd

	// do not block the handler to update the status
	go func() {
	ctxStatus, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()
	_, err = np.kubeClient.ResourceV1beta1().ResourceClaims(config.Claim.Namespace).ApplyStatus(ctxStatus,
	resourceClaimApply,
	metav1.ApplyOptions{FieldManager: np.driverName, Force: true},
	)
	if err != nil {
	klog.Infof("failed to update status for claim %s/%s : %v", config.Claim.Namespace, config.Claim.Name, err)
	} else {
	klog.V(4).Infof("update status for claim %s/%s", config.Claim.Namespace, config.Claim.Name)
	}
	}()

[aojea]

maybe we just need to build the final status and update entirely, there should not be other owner

[gauravkghildiyal]

maybe we just need to build the final status and update entirely, there should not be other owner

Sounds fair to me

[aojea]

any taker?

[gauravkghildiyal]

I'm guessing @michaelasp has already begun working on this?

[aojea]

you can sync offline :)

[michaelasp]

Ah gotcha, that makes sense, I didn't get the nuance of having the same owner. Yeah I was already working on a solution along those lines, building the status in one go. I can take a look at getting the PR out

[gauravkghildiyal]

Sounds good, thanks Michael!