[enterprise-3.11] Added section on Setting Resource Quota for Extended Resources

admin_guide/quota.adoc

@@ -159,6 +159,126 @@ $ oc create quota <name> --hard=count/<resource>.<group>=<quota> <1>
 <1> `<resource>` is the name of the resource, and `<group>` is the API group, if applicable.
 Use the `kubectl api-resources` command for a list of resources and their associated API groups.
 
+[[extended-resources]]
+=== Setting Resource Quota for Extended Resources
+
+Overcommitment of resources is not allowed for extended resources, so you must
+specify `requests` and `limits` for the same extended resource in a quota.
+Currently, only quota items with the prefix `requests.` are allowed for extended
+resources. The following is an example scenario of how to set resource quota for
+the GPU resource `nvidia.com/gpu`.
+
+.Procedure
+
+. Determine how many GPUs are available on a node in your cluster. For example:
++
+----
+# oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'
+                    openshift.com/gpu-accelerator=true
+Capacity:
+ nvidia.com/gpu:  2
+Allocatable:
+ nvidia.com/gpu:  2
+  nvidia.com/gpu  0           0
+----
++
+In this example, 2 GPUs are available.
+
+. Set a quota in the namespace `nvidia`. In this example, the quota is `1`:
++
+----
+# cat gpu-quota.yaml
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+  name: gpu-quota
+  namespace: nvidia
+spec:
+  hard:
+    requests.nvidia.com/gpu: 1
+----
+
+. Create the quota:
++
+----
+# oc create -f gpu-quota.yaml
+resourcequota/gpu-quota created
+----
+
+. Verify that the namespace has the correct quota set:
++
+----
+# oc describe quota gpu-quota -n nvidia
+Name:                    gpu-quota
+Namespace:               nvidia
+Resource                 Used  Hard
+--------                 ----  ----
+requests.nvidia.com/gpu  0     1
+----
+
+. Run a pod that asks for a single GPU:
++
+----
+# oc create pod gpu-pod.yaml
+----
++
+----
+apiVersion: v1
+kind: Pod
+metadata:
+  generateName: gpu-pod-
+  namespace: nvidia
+spec:
+  restartPolicy: OnFailure
+  containers:
+  - name: rhel7-gpu-pod
+    image: rhel7
+    env:
+      - name: NVIDIA_VISIBLE_DEVICES
+        value: all
+      - name: NVIDIA_DRIVER_CAPABILITIES
+        value: "compute,utility"
+      - name: NVIDIA_REQUIRE_CUDA
+        value: "cuda>=5.0"
+
+    command: ["sleep"]
+    args: ["infinity"]
+
+    resources:
+      limits:
+        nvidia.com/gpu: 1
+----
+
+. Verify that the pod is running:
++
+----
+# oc get pods
+NAME              READY     STATUS      RESTARTS   AGE
+gpu-pod-s46h7     1/1       Running     0          1m
+----
+
+. Verify that the quota `Used` counter is correct:
++
+----
+# oc describe quota gpu-quota -n nvidia
+Name:                    gpu-quota
+Namespace:               nvidia
+Resource                 Used  Hard
+--------                 ----  ----
+requests.nvidia.com/gpu  1     1
+----
+
+. Attempt to create a second GPU pod in the `nvidia` namespace. This is
+technically available on the node because it has 2 GPUs:
++
+----
+# oc create -f gpu-pod.yaml
+Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1
+----
++
+This *Forbidden* error message is expected because you have a quota of 1 GPU and
+this pod tried to allocate a second GPU, which exceeds its quota.
+
 [[quota-scopes]]
 == Quota Scopes