Skip to main content
Version: v1.4

GPU Workloads with Nova

Overview​

Nova supports running GPU-accelerated workloads using both NVIDIA and AMD GPUs. Nova also respects nodeSelector constraints which can be used to specify GPU characteristics. In addition, Nova supports both simple and mixed mode NVIDIA MIG.

Prerequisites​

  • Nova workload cluster with GPU-enabled nodes
  • Appropriate GPU drivers, such as the gpu-operator installed on the workload clusters

GPU Resource Types​

Nova recognizes the following GPU resource types:

  • NVIDIA GPUs: nvidia.com/gpu
  • AMD GPUs: amd.com/gpu
  • NVIDIA mixed mode MIG GPUs: nvidia.com/mig-*

GPU Resource Monitoring​

Available GPU resources on Nova workload clusters can be monitored using the novactl command (or nova kubectl plugin):

kubectl  --context=nova get clusters -o wide

This will display GPU, CPU and Memory resources:

NAME    K8S-VERSION   K8S-CLUSTER    NOVA-CREATED   PROVIDER   REGION   ZONE       AVAIL-CPU   AVAIL-MEM     AVAIL-NVIDIAGPU   AVAIL-AMDGPU   READY   IDLE    STANDBY
wlc-1   1.33          worklc-12232   false          azure      eastus   eastus-2   16019m      102957284Ki   3                 0              True    False   False
wlc-2   1.33          worklc-30337   false          azure      eastus   eastus-2   12516m      91274704Ki    3                 0              True    False   False