Spread scheduling with overrides
export NOVA_NAMESPACE=elotl
export NOVA_CONTROLPLANE_CONTEXT=nova
export K8S_CLUSTER_CONTEXT_1=k8s-cluster-1
export K8S_CLUSTER_CONTEXT_2=k8s-cluster-2
export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2
Overview
Nova supports spreading a group of workloads onto multiple clusters. This means that the whole group can be cloned and run in the multiple workload clusters.
User also has an option to override fields in the managed objects per cluster, using .spec.spreadConstraints.overrides
field in SchedulePolicy. In this tutorial we will learn how to do this for Namespace kind of object, but it can be done for any kind which can be scheduled by Nova.
This is useful for cases when we want to ensure that the given workload runs in a set of cluster, but we want to tweak its configuration in each workload cluster.
In this tutorial we will deploy istio-system
namespace via Nova to two workload cluster, and we will override a value of topology.istio.io/network
label in each cluster.
Let's start by listing our workload clusters connected to the Nova Control Plane:
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters
NAME K8S-VERSION K8S-CLUSTER REGION ZONE READY IDLE STANDBY
wlc-1 1.28 wlc-1 True True False
wlc-2 1.28 wlc-2 True True False
We want to ensure that this namespace:
apiVersion: v1
kind: Namespace
metadata:
name: istio-system
labels:
release: istio
will be present in each workload cluster with a topology.istio.io/network
label but with different value in each cluster.
Let's take a look at the SchedulePolicy manifest we will use in this tutorial:
apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
name: spread-namespace-policy
spec:
namespaceSelector: {}
resourceSelectors:
labelSelectors:
- matchLabels:
release: istio # this makes sure we will match our istio-system namspace, because it has release=istio label
groupBy:
labelKey: release
clusterSelector: # this cluster selector selects two workload clusters: wlc-1 & wlc-2
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- ${NOVA_WORKLOAD_CLUSTER_1} # change it to your workload cluster name
- ${NOVA_WORKLOAD_CLUSTER_2} # change it to your workload cluster name
spreadConstraints:
spreadMode: Duplicate # this means that each cluster selected in clusterSelector will get a duplicate of the object(s) matched by this policy
# our topologyKey is kubernetes.io/metadata.name label, which will have unique value for each workload cluster.
# This means that each workload cluster will get a duplicated of matched object(s).
topologyKey: kubernetes.io/metadata.name
# here we specify overrides per cluster
overrides:
# for a workload cluster which has topologyKey=topologyValue (in this case kubernetes.io/metadata.name=wlc-1),
# we will apply following workloads overrides:
- topologyValue: ${NOVA_WORKLOAD_CLUSTER_1} # change it to your workload cluster name
resources:
# we will override v1/Namespace object named istio-system
- kind: Namespace
apiVersion: v1
name: istio-system
# in istio-system namespace, we can override multiple fields, in this case we will override only one
override:
# here we select which field we want to override. In this case it is "topology.istio.io/network" label.
- fieldPath: metadata.labels['topology.istio.io/network']
value:
# we will override it with "west-network" value
staticValue: west-network
# for a workload cluster which has topologyKey=topologyValue (in this case kubernetes.io/metadata.name=wlc-2),
# we will apply following workloads overrides:
- topologyValue: wlc-2 # change it to your workload cluster name
resources:
# we will override v1/Namespace object named istio-system
- kind: Namespace
apiVersion: v1
name: istio-system
# in istio-system namespace, we can override multiple fields, in this case we will override only one
override:
# here we select which field we want to override. In this case it is "topology.istio.io/network" label.
- fieldPath: metadata.labels['topology.istio.io/network']
value:
# we will override it with "central-network" value
staticValue: central-network
Now, we can create a SchedulePolicy and istio-system
namespace in the Nova Control Plane:
envsubst < "examples/sample-spread-scheduling/policy-override.yaml" > "./policy-override.yaml"
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f ./policy-override.yaml
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f examples/sample-spread-scheduling/namespace-override.yaml
We expect to see istio-system
namespace being present in both workload clusters:
- in the first workload cluster (in my case
wlc-1
, it's also a name of the kube config context)istio-system
namespace should havewest-network
label value. - in the second workload cluster (in my case
wlc-2
, it's also a name of the kube config context)istio-system
namespace should havecentral-network
label value.
If we have kube contexts available for these two workload clusters, we can check it with kubectl
.
kubectl --context=${K8S_CLUSTER_CONTEXT_1} wait '--for=jsonpath={.metadata.labels.topology\.istio\.io/network}'=west-network namespace istio-system --timeout=90s
kubectl --context=${K8S_CLUSTER_CONTEXT_2} wait '--for=jsonpath={.metadata.labels.topology\.istio\.io/network}'=central-network namespace istio-system --timeout=90s
Overrides were applied correctly!
noteNumber of replicas for Deployments, ReplicaSets, StatefulSets and paralellism for Jobs cannot be overridden using overrides. Please use
.spec.spreadConstraints.percentageSplit
instead.
Cleanup
To delete all resources created in this tutorial, delete the namespace in Nova Control Plane:
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f examples/sample-spread-scheduling/namespace-override.yaml
and then delete schedule policy:
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f ./policy-override.yaml
rm -f ./policy-override.yaml