Skip to main content
Version: v1.5

Fill-and-Spill Scheduling

Fill-and-spill scheduling places workloads on a prioritized list of workload clusters.

Nova attempts to place workloads on the highest-priority eligible workload cluster first. If that workload cluster does not have enough available capacity, Nova tries the next workload cluster in the priority order, and so on.

When to Use This​

Use fill-and-spill scheduling when:

  • one workload cluster should be preferred over others
  • on-premises capacity should be used before cloud capacity
  • lower-cost workload clusters should be filled before higher-cost workload clusters
  • secondary workload clusters should only be used when preferred workload clusters are full

How It Works​

Fill-and-spill scheduling is configured in a SchedulePolicy using orderedClusterSelector.

The order of values determines priority.

Ordering by Workload Cluster Name​

This example attempts placement on onprem-cluster-oregon-A first. If that workload cluster does not have enough capacity, Nova considers eks-cluster-us-east-1.

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: fill-and-spill-by-name
spec:
  orderedClusterSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - onprem-cluster-oregon-A
          - eks-cluster-us-east-1

Ordering by Workload Cluster Group​

Fill-and-spill scheduling can also prioritize groups of workload clusters using labels.

This example prioritizes GKE workload clusters before EKS workload clusters using the Nova-managed provider label:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: fill-and-spill-by-provider
spec:
  orderedClusterSelector:
    matchExpressions:
      - key: nova.elotl.co/cluster.provider
        operator: In
        values:
          - gce
          - aws

The label nova.elotl.co/cluster.provider is automatically added by the Nova control plane to Cluster resources when workload clusters are added.

Example Output​

You can verify the available workload cluster capacity before placement:

kubectl --context=nova get clusters -o wide

Example (abridged) output:

NAME      PROVIDER   REGION       AVAIL-CPU   AVAIL-MEM   AVAIL-NVIDIAGPU   AVAIL-AMDGPU  READY
wlc-1     gce        us-central1  16000m      102Gi       1                 0             True
wlc-2     aws        us-east-1    12000m      90Gi        0                 0             True

If both workload clusters are eligible, Nova evaluates them in the order defined by orderedClusterSelector.

Relationship to Grouping​

If the matched workload is grouped, Nova evaluates capacity for the full group rather than for each resource independently.