Skip to main content
Version: v1.5

Grouping and Co-location

Applications are often made up of multiple Kubernetes resources that must be placed together.

For example, an application might include:

  • a Deployment
  • a ConfigMap
  • a Secret
  • a Service
  • a ServiceAccount
  • one or more supporting services, such as Redis, Postgres, Kafka, or RabbitMQ

In a multi-cluster environment, related resources often need to run on the same workload cluster so they can communicate using cluster-local networking and DNS.

Nova supports this through grouping and co-location.

How Grouping Works​

Grouping is defined through the groupBy field in a SchedulePolicy.

groupBy tells Nova to treat resources that share the same label value as a single placement unit.

Nova automatically manages the resulting groups. Users generally define grouping behavior through SchedulePolicy; they do not usually modify the generated group objects directly.

Example SchedulePolicy​

This policy groups resources by the kubernetes.io/part-of label:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: web-app-policy
spec:
  resourceSelectors:
    labelSelectors:
      - matchExpressions:
          - key: kubernetes.io/part-of
            operator: Exists
            values: []
  groupBy:
    labelKey: kubernetes.io/part-of

Example Workload Resources​

The following resources share labels that identify which application group they belong to:

apiVersion: v1
kind: ConfigMap
metadata:
  name: backend-app-config
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-app
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-db
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  labels:
    kubernetes.io/part-of: observability-stack

Resulting Behavior​

Nova creates two placement groups from these resources:

  1. web-app, containing:

    • backend-app-config
    • backend-app
    • redis
    • web-db

  2. observability-stack, containing:

    • prometheus

Each group is treated as one placement unit. Nova places the resources in a group on the same workload cluster when sufficient resources are available.

Capacity-aware Group Placement​

When grouped workloads are updated, Nova evaluates whether the current workload cluster still has sufficient CPU, memory, and, optionally, GPU resources for the entire group.

If the current workload cluster no longer has sufficient capacity, Nova will reschedule the group to another eligible workload cluster unless capacity-based relocation is disabled.

Multi-cluster Group Placement​

By default, Nova places a group on a single workload cluster.

If Nova cannot find one workload cluster with enough available resources and multi-cluster capacity placement is enabled, Nova may consider placing replica-count-based members of the group across multiple workload clusters.

This mode should only be used when workload clusters share a network layer that allows pods and services to span clusters, such as with Cisco Cilium Cluster Mesh.

When using Cilium Cluster Mesh, services fronting pods across clusters are expected to include:

metadata:
  annotations:
    service.cilium.io/global: "true"

For headless services, also include:

metadata:
  annotations:
    service.cilium.io/global-sync-endpoint-slices: "true"

Cilium Cluster Mesh must also be configured with endpoint slice synchronization enabled.