Annotation-based scheduling

Overview

In Annotation-based scheduling, Kubernetes workloads can be scheduled to run on any one of the workload clusters managed by the Nova control plane simply by adding an annotation to the workload manifest.

The annotation to be added to the workload manifest will specify the destination workload cluster. Here is an example of an annotation that will schedule the corresponding Kubernetes object to a workload cluster named ${NOVA_WORKLOAD_CLUSTER_1}:

metadata:
  annotations:
     nova.elotl.co/cluster: wlc-1

Example

Nova comes with sample workloads that can be used to try out annotation-based scheduling. These workloads are available in the examples/sample-workloads folder in the try-nova repository. Let's use the nginx workload available in try-nova repo in the: examples/sample-workloads/nginx.yaml to try out annotation-based scheduling.

We will first export these environment variables so that subsequent steps in this tutorial can be easily followed.

export NOVA_NAMESPACE=elotl
export NOVA_CONTROLPLANE_CONTEXT=nova
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2

Export these additional environment variables if you installed Nova using the tarball.

export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2

Alternatively export these environment variables if you installed Nova using setup scripts provided in the try-nova repository.

export K8S_HOSTING_CLUSTER_CONTEXT=kind-hosting-cluster
export K8S_CLUSTER_CONTEXT_1=kind-wlc-1
export K8S_CLUSTER_CONTEXT_2=kind-wlc-2

1. Let's begin by checking the workload clusters connected to Nova using kubectl:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters
NAME    K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE   STANDBY
wlc-1   1.28          wlc-1                         True    True   False
wlc-2   1.28          wlc-2                         True    True   False

2. To start running the workload:

In the workload manifest file: examples/sample-workloads/nginx.yaml, you can manually edit annotation nova.elotl.co/cluster: ${NOVA_WORKLOAD_CLUSTER_1} replacing ${NOVA_WORKLOAD_CLUSTER_1} with the name the workload cluster on which you would like to run the nginx workload. Or you can use the following command to replace the environment variable using the envsubstr command.

envsubst < "examples/sample-workloads/nginx.yaml" > "./nginx.yaml"
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} apply -f ./nginx.yaml
rm -f ./nginx.yaml

3. To verify that the deployment has begun:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get deployments

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   2/2     2            2           28s

4. To check that the deployment's pods have begun running in the workload cluster:

kubectl --context ${K8S_CLUSTER_CONTEXT_1} get pods

NAME                     READY   STATUS    RESTARTS   AGE
nginx-66b6c48dd5-g6nrg   1/1     Running   0          106s
nginx-66b6c48dd5-r2twr   1/1     Running   0          106s

5. Note that there will be no pods running in the Nova control plane cluster:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get pods

No resources found in default namespace.

Updating and Deleting Kubernetes resources through Nova

You can also modify or delete a workload through Nova and Nova will automatically update the corresponding objects in the workload cluster. Let's use the nginx deployment as an example:

1. To change number of replicas to 3, run:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} scale deployment nginx --replicas=3

2. In your workload cluster, there should be 3 nginx pods running.
3. Run

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get deployments

and you should be able to see 3 replicas running.

Deleting a workload in nova will result in the workload being deleted from the workload cluster too:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete deployment nginx

You should be able to see the nginx deployment deleted both from nova control plane and your workload cluster.