Skip to main content
Version: v1.5

Grouping and Co-location

Applications are often made up of multiple Kubernetes resources that must be placed together.

For example, an application might include:

  • a Deployment
  • a ConfigMap
  • a Secret
  • a Service
  • a ServiceAccount
  • one or more supporting services, such as Redis, Postgres, Kafka, or RabbitMQ

In a multi-cluster environment, related resources often need to run on the same workload cluster so they can communicate using cluster-local networking and DNS.

Nova supports this through grouping and co-location.

How Grouping Works​

Grouping is defined through the groupBy field in a SchedulePolicy.

groupBy tells Nova to treat resources that share the same label value as a single placement unit.

Nova automatically manages the resulting groups. Users generally define grouping behavior through SchedulePolicy; they do not usually modify the generated group objects directly.

Example SchedulePolicy​

This policy groups resources by the kubernetes.io/part-of label:

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: web-app-policy
spec:
  resourceSelectors:
    labelSelectors:
      - matchExpressions:
          - key: kubernetes.io/part-of
            operator: Exists
            values: []
  groupBy:
    labelKey: kubernetes.io/part-of

Example Workload Resources​

The following resources share labels that identify which application group they belong to:

apiVersion: v1
kind: ConfigMap
metadata:
  name: backend-app-config
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-app
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-db
  labels:
    kubernetes.io/part-of: web-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  labels:
    kubernetes.io/part-of: observability-stack

Resulting Behavior​

Nova creates two placement groups from these resources:

  1. web-app, containing:

    • backend-app-config
    • backend-app
    • redis
    • web-db

  2. observability-stack, containing:

    • prometheus

Each group is treated as one placement unit. Nova places the resources in a group on the same workload cluster when sufficient resources are available.

Capacity-aware Group Placement​

When grouped workloads are updated, Nova evaluates whether the current workload cluster still has sufficient CPU, memory, and, optionally, GPU resources for the entire group, and can still meet the group's constraints.

If the current workload cluster no longer has sufficient capacity or can no longer meet the group's constraints, Nova will reschedule the group to another eligible workload cluster, unless capacity-based relocation is disabled via the nova scheduler --skip-capacity-relocate option.

The Nova scheduler has a snapshot view of its workload clusters, and hence its placement code can under- or over-estimate the remaining capacity on the clusters. Groups placed in quick succession may all see the same available capacity, before the pods of the earlier groups are running and counted, and may over-commit a workload cluster. To avoid this over-commitment, set the nova scheduler --capacity-reservation-ttl option (Helm value scheduler.capacityReservationTTL) to a duration with a unit, e.g. 90s. Nova then reserves the resources of each newly placed group on its target workload cluster(s) for that duration, and excludes them when placing other groups. If a group gains members while its existing members are still reserved, and the new members fit on its current workload cluster only if those existing members are already counted in the cluster's reported capacity, Nova waits for the group's reservations to expire before placing the new members, rather than relocating the group. Note that capacity reservations are disabled by default.

Multi-cluster Group Placement​

By default, Nova places a group on a single workload cluster.

If Nova cannot find one workload cluster with enough available resources and multi-cluster capacity placement is enabled via the nova scheduler --multi-cluster-capacity option, Nova may consider placing replica-count-based members of the group across multiple workload clusters.

This mode should only be used when workload clusters share a network layer that allows pods and services to span clusters, such as with Cisco Cilium Cluster Mesh.

When using Cilium Cluster Mesh, services fronting pods across clusters are expected to include:

metadata:
  annotations:
    service.cilium.io/global: "true"

For headless services, also include:

metadata:
  annotations:
    service.cilium.io/global-sync-endpoint-slices: "true"

Cilium Cluster Mesh must also be configured with endpoint slice synchronization enabled.