Fill-and-Spill Scheduling

Fill-and-spill scheduling is a type of scheduling technique in Nova that enables Kubernetes workloads to be placed on a prioritized list of workload clusters connected to the Nova Control Plane. In the prioritized list, clusters can be referred to by their names or by labels to specify a group of workload clusters.

Fill-and-Spill scheduling can be specified within a Schedule Policy using the field OrderedClusterSelector. Let's take a look at the different ways in which this field can be used to capture cluster ordering or priority.

Fill and Spill Ordering by Cluster Name

In this type of scheduling, Nova will try to place workloads on the first (i.e. the highest priority) cluster in the user-provided list. In case, this target cluster's remaining capacity is not sufficient to place the incoming Kubernetes object (or group of Kubernetes objects, ScheduleGroup, as the case maybe), then Nova will try to place the workload on the next lower priority workload cluster in the user-provided list, and so on.

The Nova SchedulePolicy to capture this, used the OrderedClusterSelector field as well as the matchExpression field along with the In operator. In the example below of a fill-and-spill policy, workloads will first be attempted to be placed on the higher priority workload cluster onprem-cluster-oregon-A. In case of insufficient capacity, workload cluster eks-cluster-us-east-1 will be considered as the target cluster.

    spec:
      OrderedClusterSelector:
        matchExpressions:
          - key: kubernetes.io/metadata.name
            operator: In
            values:
              - onprem-cluster-oregon-A
              - eks-cluster-us-east-1

Fill and Spill Ordering by Cluster Group Name

Fill-and-spill scheduling can also be used to prioritize groups of workload clusters. This is done by leveraging Nova-specific or user-specified cluster labels. Let's look at this capability via an example where clusters are prioritized based on the cloud provider via label nova.elotl.co/cluster.provider.

Let's consider a case where Nova is managing workload clusters from two different cloud providers, Amazon EKS and Google GKE, and the user would like to prioritize GKE Kubernetes clusters over EKS clusters. The YAML snippet below shows how this can be specified in a SchedulePolicy.

    spec:
      OrderedClusterSelector:
      matchExpressions:
       - key: nova.elotl.co/cluster.provider
         operator: In
         values:
          - gce
          - aws

The label nova.elotl.co/cluster.provider is automatically added by the Nova Control Plane to the Cluster custom resource at the time a Nova workload cluster is added to the Nova CP.