Version: v1.0

Argo Rollouts in multi-cluster environment

Prerequisites

argo rollouts kubectl plugins installed Link to install instructions
Nova Control Plane installed with 2 workload clusters connected
access to the Nova Control Plane hosting cluster

Installing Argo Rollouts

Typically, in the single cluster scenario, both Argo Rollouts Custom Resource Definitions (CRDs) and the controller would be created in a single cluster. In a multi-cluster environment managed by Nova, we need to split it: CRDs will be created in the Nova Control Plane, but argo-rollouts controller will run next to the Nova Control Plane components in the hosting cluster. Argo-rollouts controller manifest is modified, so it can observe Rollouts in the Nova Control Plane (instead of doing this in the hosting cluster).

We will first export these environment variables so that subsequent steps in this tutorial can be easily followed.

export NOVA_NAMESPACE=elotl
export NOVA_CONTROLPLANE_CONTEXT=nova
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2

Export these additional environment variables if you installed Nova using the tarball.

export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2

Alternatively export these environment variables if you installed Nova using setup scripts provided in the try-nova repository.

export K8S_HOSTING_CLUSTER_CONTEXT=kind-hosting-cluster
export K8S_CLUSTER_CONTEXT_1=kind-wlc-1
export K8S_CLUSTER_CONTEXT_2=kind-wlc-2

Let's create Argo Rollouts CRDs in the Nova Control Plane first:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f examples/argo-rollouts/crds.yaml

and then the Argo Rollouts controller in the hosting cluster (see the kube context in the command below. Please modify the context if your hosting cluster context is different). We need to install Argo Rollouts controller in the same namespace where Nova Control Plane components were installed. By default, it's elotl namespace.

kubectl --context=${K8S_HOSTING_CLUSTER_CONTEXT} create -f examples/argo-rollouts/argo-rollouts.yaml -n ${NOVA_NAMESPACE}

Now, we need to wait until the argo-rollouts controller is up and running:

kubectl --context=${K8S_HOSTING_CLUSTER_CONTEXT} wait --for=condition=available -n ${NOVA_NAMESPACE} deployment argo-rollouts --timeout=210s

Our goal in this tutorial is to do canary rollout of our application to two workload clusters at the same time. We will deploy 10 replicas of our application and instruct Nova, to spread it equally on two workload clusters (5 replicas each). The Rollout, that we will use will have 5 steps:

First, we will replace 20% of 10 replicas with a new version (so 2 replicas, 1 replica in each workload cluster)
We will pause, that will give us an opportunity to verify if 2 new replicas are spread correctly on two clusters. Then, we will move to the next step
We will run automated analysis of the new version: It will be a kubernetes Job, which will run 1 pod in each workload cluster, load testing the new version and checking the error rate.
If the analysis run was successful, we scale up to 6 replicas of the new version (3 in each workload cluster) and scale down old version to 4 replicas (2 in each workload cluster).
Then we will pause to verify that replicas are spread correctly. After that, we will promote the rollout again After those steps, new version will be scaled up to replicas (10, 5 in each workload cluster) and the old version will be scaled to 0. We will call it a successful Rollout in the multi cluster environment.

All these steps are defined in the canary strategy steps part of the Rollout manifest:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-demo
spec:
  replicas: 10
  revisionHistoryLimit: 1
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: rollouts-demo
  strategy:
    canary:
      canaryMetadata:
        labels:
          role: canary
      stableMetadata:
        labels:
          role: stable
      canaryService: rollouts-demo-canary
      steps:
        - setWeight: 20
        - pause: {}
        - analysis:
            templates:
              - templateName: http-benchmark
            args:
              - name: host
                value: rollouts-demo-canary
        - setWeight: 60
        - pause: {}

as you see, we specify a reference to the deployment that we will use to get the spec of the ReplicaSets created by argo rollouts controller. Here is the Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rollouts-demo
  namespace: default
  labels:
    base-template: "yes"
spec:
  replicas: 10
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
        role: placeholder
    spec:
      containers:
        - name: main
          image: argoproj/rollouts-demo:green
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              memory: 32Mi
              cpu: 5m

in the 3rd step of the Rollout canary strategy, we specify the reference to yet another Argo Rollout Custom Resource, which is AnalysisTemplate. We will need to have it created in the Nova Control Plane for rollout to proceed. Here is AnalysisTemplate manifest:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: http-benchmark
spec:
  args:
    - name: host
  metrics:
    - name: http-benchmark
      failureLimit: 1
      interval: 5s
      count: 1
      provider:
        job:
          metadata:
            labels:
              role: placeholder
              app: rollouts-demo
          spec:
            parallelism: 2
            template:
              spec:
                containers:
                  - name: load-tester
                    image: argoproj/load-tester:latest # we will use load-test image provided by argo project to test a new version
                    command: [sh, -xec]
                    # we will send requests to the new version of our app, collect the responses and then check whether the error rate is below acceptable threshold.
                    args:
                      - |
                        wrk -t1 -c1 -d5 -s report.lua http://{{args.host}}/color
                        jq -e '.errors_ratio <= 0.05' report.json
                restartPolicy: Never
            backoffLimit: 0

Let's create AnalysisTemplate and base Deployment in the Nova Control Plane first.

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f examples/argo-rollouts/analysis_template.yaml

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f examples/argo-rollouts/base-deploy.yaml

As we mentioned before, we want to spread services and replicasets for our Rollout across two workload clusters using Nova. Let's check the workload clusters connected to the Nova Control Plane:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters

NAME    K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE   STANDBY
wlc-1   1.28          wlc-1                         True    True   False
wlc-2   1.28          wlc-2                         True    True   False

In my case, they are named wlc-1 & wlc-2. We will use it in the SchedulePolicy's clusterSelector as well as spread constraints:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: rollouts-demo-policy
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  clusterSelector: # here we select both clusters
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - ${NOVA_WORKLOAD_CLUSTER_1} # if your workload clusters are named differently, please modify the value here
          - ${NOVA_WORKLOAD_CLUSTER_2} # if your workload clusters are named differently, please modify the value here
  spreadConstraints:
    # in spread constraints, we ask Nova to divide our workload (ReplicaSet) equally across two clusters selected in clusterSelector section.
    # practically it means that ReplicaSet with 10 replicas will run 5 replicas in each cluster.
    topologyKey: kubernetes.io/metadata.name 
    spreadMode: Divide
  resourceSelectors:
    labelSelectors:
      - matchExpressions:
          # this selector will match stable and canary Services, ReplicaSets created by argo-rollouts controller
          # as well as AnalysisRun Job.
          - key: app
            operator: In
            values:
              - rollouts-demo
          - key: role
            operator: Exists
          # this is to ensure that our "base" Deployment won't be scheduled by Nova to workload clusters
          - key: base-template
            operator: DoesNotExist
  groupBy:
    labelKey: role

Now, we can create this SchedulePolicy.

envsubst < "examples/argo-rollouts/schedule_policy_canary.yaml" > "./schedule_policy_canary.yaml"
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f ./schedule_policy_canary.yaml

Next step is creating services: one will point out to the stable replicaset pods, another to the canary pods.

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f examples/argo-rollouts/services-canary.yaml

Finally, we can create the Rollout spec and wait until it reached 10 available replicas:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} create -f examples/argo-rollouts/rollout_prod_canary.yaml

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} wait '--for=jsonpath={.status.availableReplicas}'=10 replicaset -l app=rollouts-demo --timeout=180s

We can now verify if each workload cluster runs 5 replicas:

kubectl --context=${K8S_CLUSTER_CONTEXT_1} get pods -l app=rollouts-demo

NAME                            READY   STATUS    RESTARTS   AGE
rollout-demo-f6597fc9d-7z78m   1/1     Running   0          3m
rollout-demo-f6597fc9d-c8fhj   1/1     Running   0          3m
rollout-demo-f6597fc9d-c8pnn   1/1     Running   0          3m
rollout-demo-f6597fc9d-ntkcj   1/1     Running   0          3m
rollout-demo-f6597fc9d-s2f2n   1/1     Running   0          3m

kubectl --context=${K8S_CLUSTER_CONTEXT_2} get pods -l app=rollouts-demo

NAME                            READY   STATUS    RESTARTS   AGE
rollout-demo-f6597fc9d-7t2cx   1/1     Running   0          3m
rollout-demo-f6597fc9d-kbdf7   1/1     Running   0          3m
rollout-demo-f6597fc9d-mk6sl   1/1     Running   0          3m
rollout-demo-f6597fc9d-p7p76   1/1     Running   0          3m
rollout-demo-f6597fc9d-rbvl5   1/1     Running   0          3m

Great, our initial deployment was successful, now we can try to deploy a new version of application image and see the progressive delivery on multi cluster environment in action.

Deploy a new version

We will update the image of our rollout from green to blue tag, using Argo Rollouts kubectl plugin. This can be also done without the plugin, by updating the manifest and applying it to cluster with kubectl.

kubectl argo rollouts --context=${NOVA_CONTROLPLANE_CONTEXT} set image rollout-demo main=argoproj/rollouts-demo:blue

Our rollout should complete step 1 and then pause. We can verify it using Argo Rollouts kubectl plugin command:

kubectl argo rollouts --context=${NOVA_CONTROLPLANE_CONTEXT} get rollout rollout-demo

Name:            rollout-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/5
  SetWeight:     20
  ActualWeight:  20
Images:          argoproj/rollouts-demo:blue (canary)
                 argoproj/rollouts-demo:green (stable)
Replicas:
  Desired:       10
  Current:       10
  Updated:       2
  Ready:         10
  Available:     10

NAME                             KIND        STATUS     AGE    INFO
⟳ rollout-demo                   Rollout     ॥ Paused   3m11s  
├──# revision:2                                                
│  └──⧉ rollout-demo-5669659bb5  ReplicaSet  ✔ Healthy  2m39s  canary
└──# revision:1                                                
   └──⧉ rollout-demo-f6597fc9d  ReplicaSet  ✔ Healthy  3m11s  stable

It seems that the new replicaset (called rollout-demo-5669659bb5) was created with 2 replicas. Let's verify if each workload cluster got a 1 replica of the new replicaset:

kubectl --context=${K8S_CLUSTER_CONTEXT_1} get pods

rollout-demo-f6597fc9d-4bdvw   1/1     Running   0          2m5s
rollout-demo-f6597fc9d-4zfvb   1/1     Running   0          2m5s
rollout-demo-f6597fc9d-9qtst   1/1     Running   0          2m5s
rollout-demo-f6597fc9d-brfdw   1/1     Running   0          2m5s
rollout-demo-5669659bb5-lzww7   1/1     Running   0          103s

kubectl --context=${K8S_CLUSTER_CONTEXT_2} get pods

NAME                            READY   STATUS    RESTARTS   AGE
rollout-demo-f6597fc9d-49dpj   1/1     Running   0          2m53s
rollout-demo-f6597fc9d-jfpvm   1/1     Running   0          2m53s
rollout-demo-f6597fc9d-ltbjf   1/1     Running   0          2m53s
rollout-demo-f6597fc9d-sdqdv   1/1     Running   0          2m53s
rollout-demo-5669659bb5-xqjc8   1/1     Running   0          2m11s

Looks like everything worked as expected. We still have 8 replicas of stable version (4 in each workload cluster) and 2 of the new one (1 in each workload cluster). We can promote rollout, and continue progressing:

kubectl argo rollouts --context=${NOVA_CONTROLPLANE_CONTEXT} promote rollout-demo

this will continue rollout until we reach step 4, which is another pause.

kubectl argo rollouts --context=${NOVA_CONTROLPLANE_CONTEXT} get rollout rollout-demo

Name:            rollout-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          4/5
  SetWeight:     60
  ActualWeight:  60
Images:          argoproj/rollouts-demo:blue (canary)
                 argoproj/rollouts-demo:green (stable)
Replicas:
  Desired:       10
  Current:       10
  Updated:       6
  Ready:         10
  Available:     10

NAME                                                              KIND         STATUS        AGE    INFO
⟳ rollout-demo                                                    Rollout      ॥ Paused      4m48s  
├──# revision:2                                                                                     
│  ├──⧉ rollout-demo-5669659bb5                                   ReplicaSet   ✔ Healthy     4m16s  canary
│  └──α rollout-demo-5669659bb5-2-2                               AnalysisRun  ✔ Successful  40s    ✔ 1
│     └──⊞ e6f5f5d6-7518-421b-b672-b47701e67f6c.http-benchmark.1  Job          ✔ Successful  40s    
└──# revision:1                                                                                     
   └──⧉ rollout-demo-f6597fc9d                                   ReplicaSet   ✔ Healthy     4m48s  stable

Seems that AnalysisRun (step 3) was successful. We can verify it with a following command:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} wait --for='jsonpath={.status.phase}=Successful' analysisrun -l app=rollout-demo -l step-index=2 --timeout=180s

Each workload cluster should now have: 1 completed pod of the AnalysisRun, 2 replicas of the stable version and 3 replicas of the canary version. Let's check that in both workload clusters:

kubectl --context=${K8S_CLUSTER_CONTEXT_1} get pods

NAME                                                          READY   STATUS      RESTARTS   AGE
e6f5f5d6-7518-421b-b672-b47701e67f6c.http-benchmark.1-wlsmk   0/1     Completed   0          2m14s
rollout-demo-f6597fc9d-4zfvb                                 1/1     Running     0          6m22s
rollout-demo-f6597fc9d-brfdw                                 1/1     Running     0          6m22s
rollout-demo-5669659bb5-hkdxl                                 1/1     Running     0          119s
rollout-demo-5669659bb5-lzww7                                 1/1     Running     0          6m
rollout-demo-5669659bb5-rcv9k                                 1/1     Running     0          119s

kubectl --context=${K8S_CLUSTER_CONTEXT_2} get pods

NAME                                                          READY   STATUS      RESTARTS   AGE
e6f5f5d6-7518-421b-b672-b47701e67f6c.http-benchmark.1-vh6ln   0/1     Completed   0          2m31s
rollout-demo-f6597fc9d-49dpj                                 1/1     Running     0          6m49s
rollout-demo-f6597fc9d-ltbjf                                 1/1     Running     0          6m49s
rollout-demo-5669659bb5-4g647                                 1/1     Running     0          2m16s
rollout-demo-5669659bb5-gm6qg                                 1/1     Running     0          2m16s
rollout-demo-5669659bb5-xqjc8                                 1/1     Running     0          6m7s

Everything looks ok, so we can promote rollout. After that, a new version will reach 10 replicas and the old one will be scaled down to 0 replicas.

kubectl argo rollouts --context=${NOVA_CONTROLPLANE_CONTEXT} promote rollout-demo

We can verify the new version running 10 replicas:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} wait '--for=jsonpath={.status.availableReplicas}'=10 replicaset -n default rollout-demo-5669659bb5 --timeout=180s

or check the Rollout status using Argo Rollouts kubectl plugin command:

kubectl argo rollouts --context=${NOVA_CONTROLPLANE_CONTEXT} get rollout rollout-demo

Name:            rollout-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          5/5
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       10
  Current:       10
  Updated:       10
  Ready:         10
  Available:     10

NAME                                                              KIND         STATUS        AGE    INFO
⟳ rollout-demo                                                    Rollout      ✔ Healthy     10m    
├──# revision:2                                                                                     
│  ├──⧉ rollout-demo-5669659bb5                                   ReplicaSet   ✔ Healthy     9m51s  stable
│  └──α rollout-demo-5669659bb5-2-2                               AnalysisRun  ✔ Successful  6m15s  ✔ 1
│     └──⊞ e6f5f5d6-7518-421b-b672-b47701e67f6c.http-benchmark.1  Job          ✔ Successful  6m15s  
└──# revision:1                                                                                     
   └──⧉ rollout-demo-f6597fc9d                                   ReplicaSet   • ScaledDown  10m

We can verify if each workload cluster has 5 pods of the new version:

kubectl --context=${K8S_CLUSTER_CONTEXT_1} get pods               
NAME                                                          READY   STATUS      RESTARTS   AGE
e6f5f5d6-7518-421b-b672-b47701e67f6c.http-benchmark.1-wlsmk   0/1     Completed   0          7m2s
rollout-demo-5669659bb5-d4gp9                                 1/1     Running     0          72s
rollout-demo-5669659bb5-hkdxl                                 1/1     Running     0          6m47s
rollout-demo-5669659bb5-lvbdd                                 1/1     Running     0          72s
rollout-demo-5669659bb5-lzww7                                 1/1     Running     0          10m
rollout-demo-5669659bb5-rcv9k                                 1/1     Running     0          6m47s

kubectl --context=${K8S_CLUSTER_CONTEXT_2} get pods
NAME                                                          READY   STATUS      RESTARTS   AGE
e6f5f5d6-7518-421b-b672-b47701e67f6c.http-benchmark.1-vh6ln   0/1     Completed   0          7m38s
rollout-demo-5669659bb5-4g647                                 1/1     Running     0          7m23s
rollout-demo-5669659bb5-4gtt6                                 1/1     Running     0          108s
rollout-demo-5669659bb5-gm6qg                                 1/1     Running     0          7m23s
rollout-demo-5669659bb5-tsm2k                                 1/1     Running     0          108s
rollout-demo-5669659bb5-xqjc8                                 1/1     Running     0          11m

Cleanup

Let's first delete Rollout, Services, AnalysisTemplate and Deployment created for this tutorial:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f examples/argo-rollouts/rollout_prod_canary.yaml
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f examples/argo-rollouts/services-canary.yaml
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f examples/argo-rollouts/analysis_template.yaml
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f examples/argo-rollouts/base-deploy.yaml

then, we can delete SchedulePolicy:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f ./schedule_policy_canary.yaml
rm -f schedule_policy_canary.yaml

finally, we can uninstall Argo Rollouts CRDs and the Rollouts controller:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} delete -f examples/argo-rollouts/crds.yaml

kubectl --context=${K8S_HOSTING_CLUSTER_CONTEXT} delete -f examples/argo-rollouts/argo-rollouts.yaml -n ${NOVA_NAMESPACE}

Additional resources

This tutorial is written based on our presentation made on DevOps Day Madrid 2023. You can watch the presentation recording here or get the slides here

Argo Rollout documentation

Argo Rollouts in multi-cluster environment

Prerequisites​

Installing Argo Rollouts​

Deploy a new version​

Cleanup​

Additional resources​

Prerequisites

Installing Argo Rollouts

Deploy a new version

Cleanup

Additional resources