Annotation-based scheduling

In Annotation-based scheduling, Kubernetes workloads can be scheduled to run on any one of the workload clusters managed by the Nova control plane simply by adding an annotation to the workload manifest.

The annotation to be added to the workload manifest will specify the destination workload cluster. Here is an example of an annotation that will schedule the corresponding Kubernetes object to a workload cluster named workload-dev-1:

metadata:
  annotations:
     nova.elotl.co/cluster: workload-dev-1

Example

Nova comes with sample workloads that can be used to try out annotation-based scheduling. These workloads are available in the sample-workloads folder. Let's use the nginx workload available here: ./sample-workloads/nginx.yaml to try out annotation-based scheduling.

1. In the workload manifest file: ./sample-workloads/nginx.yaml, edit annotation nova.elotl.co/cluster: my-workload-cluster-1 replacing my-workload-cluster-1 with the name the workload cluster on which you would like to run the nginx workload. If you would like to retrieve the list of workload clusters being managed by Nova, run this command:

    $ kubectl --context=nova get clusters
    NAME              K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE   STANDBY  
    workload-dev-1    1.22          workload-dev-1                True    True   False    
    workload-dev-2    1.22          workload-dev-2                True    True   False   

2. To start running the workload: kubectl --context=nova apply -f sample-workloads/nginx.yaml

    $ kubectl --context=nova apply -f sample-workloads/nginx.yaml
    deployment.apps/nginx created

3. To verify that the deployment has begun: kubectl --context=nova get deployments

    $ kubectl --context=nova get deployments
    NAME    READY   UP-TO-DATE   AVAILABLE   AGE
    nginx   2/2     2            2           28s

4. To check that the deployment's pods have begun running in the workload cluster: kubectl --context my-workload-cluster-1 get pods

    $ kubectl --context my-workload-dev-1 get pods                                
    NAME                     READY   STATUS    RESTARTS   AGE
    nginx-66b6c48dd5-g6nrg   1/1     Running   0          106s
    nginx-66b6c48dd5-r2twr   1/1     Running   0          106s

5. Note that there will be no pod running in the Nova control plane cluster: kubectl --context=nova get pods should show no pod.

    $ kubectl --context=nova get pods       
    No resources found in default namespace.

Updating and Deleting Kubernetes resources through Nova

You can also modify or delete a workload through Nova and Nova will automatically update the corresponding objects in the workload cluster. Let's use the nginx deployment as an example:

1. Run kubectl --context=nova edit deployment nginx, and change the replica from 2 to 3.
2. In your workload cluster, there should be 3 nginx pods running.
3. Run kubectl --context=nova get deployments, and you should be able to see 3 replicas running.

Deleting a workload in nova will result in the workload being deleted from the workload cluster too:

1. Run kubectl --context=nova delete deployment nginx.
2. You should be able to see the nginx deployment deleted both from nova control plane and your workload cluster.