Policy-based Scheduling

Policy-based scheduling is the primary way to control workload placement in Nova.

A SchedulePolicy defines:

• which workloads the policy applies to
• which workload clusters are eligible
• how placement decisions should be made

This allows placement behavior to be defined declaratively and updated without modifying the workload manifests themselves.

What a SchedulePolicy Controls

A SchedulePolicy can control several aspects of placement.

Area	Purpose
Namespace selection	Selects namespaces whose resources may be scheduled by the policy
Resource selection	Selects Kubernetes resources matched by labels or expressions
Cluster selection	Defines which workload clusters are eligible
Placement behavior	Defines grouping, spreading, or prioritization behavior

Namespace Selector

A namespace selector specifies which namespaces contain resources that should be considered by the policy.

This example selects resources in the microsvc-demo namespace:

spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: microsvc-demo

This example selects resources in either namespace-1 or namespace-2:

spec:
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - namespace-1
          - namespace-2

Supported operators include:

• In
• NotIn
• Exists
• DoesNotExist

Resource Selector

A resource selector defines which Kubernetes resources are matched by the policy.

This example matches resources with the label microServicesDemo: "yes":

spec:
  resourceSelectors:
    labelSelectors:
      - matchLabels:
          microServicesDemo: "yes"

This example matches resources that have the label key app.kubernetes.io/name:

spec:
  resourceSelectors:
    labelSelectors:
      - matchExpressions:
          - key: app.kubernetes.io/name
            operator: Exists
            values: []

Cluster Selector

A cluster selector defines the workload clusters that are eligible for placement.

If clusterSelector is not included, Nova considers all workload clusters.

This example selects workload clusters in us-east-1:

spec:
  clusterSelector:
    matchLabels:
      nova.elotl.co/cluster.region: "us-east-1"

This example excludes two specific workload clusters:

spec:
  clusterSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: NotIn
        values:
          - prod-eks-us-west-1
          - prod-eks-us-west-2

This example selects workload clusters running a specific Kubernetes version:

spec:
  clusterSelector:
    matchLabels:
      nova.elotl.co/cluster.version: "v1.33.0"

This example selects a specific workload cluster by name:

spec:
  clusterSelector:
    matchLabels:
      kubernetes.io/metadata.name: "cluster-one"

This example selects workload clusters that are not in us-east-1:

spec:
  clusterSelector:
    matchExpressions:
      - key: nova.elotl.co/cluster.region
        operator: NotIn
        values:
          - us-east-1

This example selects workload clusters that have a custom on-prem: "true" label:

spec:
  clusterSelector:
    matchLabels:
      on-prem: "true"

Complete Example

The following example selects resources in the microsvc-demo namespace that have app.kubernetes.io/part-of: web-app, and places them on eligible workload clusters in us-east-1.

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: web-app-policy
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: microsvc-demo
  resourceSelectors:
    labelSelectors:
      - matchLabels:
          app.kubernetes.io/part-of: web-app
  clusterSelector:
    matchLabels:
      nova.elotl.co/cluster.region: "us-east-1"

Example Output

You can inspect the policy from the Nova control plane:

kubectl --context=nova get schedulepolicy web-app-policy -o yaml

Example output:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: web-app-policy
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: microsvc-demo
  resourceSelectors:
    labelSelectors:
      - matchLabels:
          app.kubernetes.io/part-of: web-app
  clusterSelector:
    matchLabels:
      nova.elotl.co/cluster.region: us-east-1

Common Placement Patterns

After a SchedulePolicy selects the workloads and eligible workload clusters, additional policy fields can be used to control how those workloads are placed.

Common patterns include:

• Grouping and co-location – place related resources together on the same workload cluster
• Spread scheduling – distribute workloads across multiple workload clusters
• Fill-and-spill scheduling – prioritize preferred workload clusters and use others only when needed
• Workload movement – move workloads when policy, capacity, or workload cluster availability changes

Policy-based scheduling is also the basis for more advanced placement behavior:

• Grouping and Co-location
• Spread Scheduling
• Fill-and-Spill Scheduling
• Workload Movement