Skip to main content
Version: v0.7.0

Policy-based Scheduling

Overview​

Nova currently supports three ways to schedule a workload - annotation-based scheduling, policy based scheduling, and smart scheduling based on resource availability

Policy Based Scheduling Testing Example​

Policy based scheduling is done via scheduling is through Nova's SchedulePolicy CRD. A schedule policy contains one or more resource selectors, and a placement to tell how the scheduling should happen for matching resources.

Both workload clusters are connected to the Nova Control Plane. We will use the names of those clusters in the SchedulePolicy. You can check how your clusters are named in the Nova Control Plane:

kubectl --context=nova get clusters
NAME                    K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE   STANDBY
my-workload-cluster-1   1.22          workload-1                    True    True   False
my-workload-cluster-2   1.22          workload-2                    True    True   False

Manifests used in this tutorial can be found in examples directory in try-nova repository. If you installed Nova from release tarball, you should have those manifests already.

  1. Open examples/sample-policy/policy.yaml in text editor and edit line:
  clusterSelector:
    matchLabels:
      kubernetes.io/metadata.name: my-workload-cluster-1 # change it to the name of one of your workload clusters

  1. kubectl --context=nova apply -f examples/sample-policy/policy.yaml. This policy is saying, for any objects with label app: redis or app: guestbook, schedule them to cluster my-workload-cluster-1 (in your case, workload cluster name will be likely different).

  2. kubectl --context=nova apply -f examples/sample-policy/guestbook-all-in-one.yaml -n guestbook. This schedules the guestbook stateless application into my-workload-cluster-1.

  3. kubectl --context=nova get all -n guestbook. You should be able to see something like the following:

    NAME                     TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
    service/frontend         LoadBalancer   10.96.25.97    35.223.90.60   80:31528/TCP   82s
    service/redis-follower   ClusterIP      10.96.251.47   <none>         6379/TCP       83s
    service/redis-leader     ClusterIP      10.96.27.169   <none>         6379/TCP       83s

    NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/frontend         3/3     3            3           83s
    deployment.apps/redis-follower   2/2     2            2           83s
    deployment.apps/redis-leader     1/1     1            1           83s

The external-ip of the frontend service should lead you to the main page of the guestbook application.

Workload migration​

Now let's say your my-workload-cluster-1 will go through some maintenance and you want to migrate your guestbook application to my-workload-cluster-2. You can achieve this by editing the schedulePolicy:

  1. kubectl --context=nova edit schedulepolicy app-guestbook -n guestbook. Update my-workload-cluster-1 to my-workload-cluster-2.
  2. You should be able to see your workload deleted from my-workload-cluster-1 and recreated inmy-workload-cluster-2.

Policy configuration​

SchedulePolicy Custom Resource selects objects from their labels and namespace via .spec.resourceSelectors.labelSelectors and .spec.namespaceSelector. The selected objects are dispatched to the clusters specified in .spec.clusterSelector.

Matching Kubernetes Objects​

To match objects from default namespace, define a SchedulePolicy like this:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  ...

To match objects from a multiple namespaces define a SchedulePolicy like this:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:
      - namespace-one
      - namespace-two
      - namespace-three
  ...

To match objects from all namespaces define a SchedulePolicy like this:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: Exists
  ...

Only the objects selected by .spec.namespaceSelector and .spec.resourceSelectors.labelSelectors are selected. To match all object in multiple namespaces define a resource selector like this:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
  resourceSelectors:
    labelSelectors:
    - matchExpressions: []
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:
      - namespace-one
      - namespace-two
      ...
  ...

Specifying target cluster(s)​

When Nova is scheduling objects, it finds matching SchedulePolicy. By default, if no .spec.clusterSelector is set, Nova will try to pick any workload cluster connected, based on available resources (see Smart scheduling). You can narrow down a list of cluster, using .spec.clusterSelector. You can also set target cluster explicitly, similar as it was described in Annotation based scheduling.

You can get a list of clusters connected to Nova, using kubectl:

kubectl get clusters

We can use cluster properties (such as name, cluster provider, kubernetes version, cluster region, zone, etc.) exposed by Nova as Cluster object labels, to instrument where the workloads should run.

To schedule all objects matching a SchedulePolicy to one of the clusters which is kubernetes version v1.25.0:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
   clusterSelector:
      matchLabels:
         nova.elotl.co/cluster.version: "v1.25.0"
   resourceSelectors:
      labelSelectors:
      ...
   namespaceSelector:
      ...
   ...

To schedule all objects matching a SchedulePolicy to the cluster named cluster-one:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
   clusterSelector:
      matchLabels:
         kubernetes.io/metadata.name: "cluster-one"
   resourceSelectors:
      labelSelectors:
      ...
   namespaceSelector:
      ...
   ...

To schedule all objects matching a SchedulePolicy to one of the clusters which is not running in region us-east-1:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
   clusterSelector:
      matchExpressions:
         - key: nova.elotl.co/cluster.region
           operator: NotIn
           values:
              - us-east-1
   resourceSelectors:
      labelSelectors:
      ...
   namespaceSelector:
      ...
   ...

You can also set custom labels to the Cluster objects and use them. For example if you mix on-prem and cloud clusters, and want to schedule workloads in on-prem clusters only, you can label all on-prem clusters objects with on-prem: true and use it in SchedulePolicy:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
   ...
spec:
   clusterSelector:
      matchLabels:
         on-prem: "true"
   resourceSelectors:
      labelSelectors:
      ...
   namespaceSelector:
      ...
   ...