Scalability

To demonstrate Nova Control Plane's scalability, we perform stress tests measuring the Control Plane's (API Server, etcd, Scheduler, Controller Manager) resource usage across varying fleet sizes, while actively orchestrating an application across every cluster in the fleet.

Test Environment

• Infrastructure: Nova control plane deployed on a dedicated AKS hosting cluster (eastus, 2 nodes, 4 vCPU / 16 GB each). The hosting cluster provides the API server, etcd, scheduler, and controller manager components that are monitored during the tests.

• Workload Clusters: Simulated using vclusters on a separate AKS cluster. Nodes are 8 vCPU / 32-64 GB, managed by the cluster autoscaler, which grows the fleet as vClusters are added — roughly 5-6 vClusters per node at this workload profile. This approach allows us to scale the number of virtual clusters without requiring physical cluster infrastructure for each one.

• Workload: Retail Store Application (Microservices) distributed via Nova Spread Duplicate Policy, deploying one instance to every cluster. Each instance consists of:

  • ServiceAccount: 5
  • Secret: 4
  • ConfigMap: 5
  • Service: 10
  • Deployment: 7 (1 pod each, all with replicas: 1)
  • StatefulSet: 3 (1 pod each, all with replicas: 1)

  That is 10 workload objects managed by Nova per cluster (7 Deployments + 3 StatefulSets), i.e. 2,000 workloads at 200 clusters.

• Measurement: Resource consumption is measured with kubectl top pod -n elotl in the hosting cluster once the fleet has settled at each cluster count. Each data point is the average of two settled snapshots; transient snapshots taken immediately after a scale-up are excluded, because agents are still registering and their metrics have not yet landed. Per-agent figures aggregate every Nova agent in the fleet at that point.

• Nova version: nova-v1.5.5

note

Rows marked with * (150 clusters) are estimates on the surrounding trend, not measurements. At that step the control plane was starved — the API server was pinned at its default 500m CPU limit and the Nova scheduler had exhausted its 64Mi ephemeral-storage limit — so kube-controller-manager crash-looped and agents disconnected. Raising both limits (see Tuning the control plane for scale) restored normal behaviour, which the 175 and 200 measurements confirm. In the charts these points are drawn as hollow markers.

Control plane totals

Active Clusters	Workloads	CPU Usage (cores)	Memory Usage (bytes)
25	250	90m	549Mi
50	500	194m	664Mi
75	750	350m	874Mi
100	1,000	350m	1034Mi
125	1,250	594m	1250Mi
150	1,500	733m*	1578Mi*
175	1,750	892m	1848Mi
200	2,000	1115m	1853Mi

Control plane, by component

Active Clusters	API Server	etcd	Controller Manager	Nova Scheduler
25	58m / 365Mi	20m / 74Mi	5m / 43Mi	8m / 67Mi
50	130m / 414Mi	30m / 100Mi	6m / 54Mi	28m / 96Mi
75	238m / 543Mi	38m / 140Mi	8m / 68Mi	65m / 124Mi
100	268m / 649Mi	32m / 147Mi	9m / 83Mi	42m / 155Mi
125	444m / 818Mi	48m / 150Mi	11m / 91Mi	91m / 191Mi
150*	512m / 1064Mi	63m / 187Mi	15m / 106Mi	143m / 221Mi
175	632m / 1179Mi	69m / 252Mi	17m / 134Mi	173m / 283Mi
200	777m / 1102Mi	90m / 268Mi	17m / 144Mi	231m / 339Mi

Nova agent footprint

Measured across every Nova agent in the fleet. Figures are the median across all agents at that fleet size.

Active Clusters	CPU median (min–max)	Memory median (min–max)
25	5m (3–12)	41Mi (39–43)
50	6.5m (5–15)	54Mi (51–56)
75	8m (6–40)	68Mi (53–70)
100	10m (7–46)	81Mi (74–83)
125	12m (8–24)	95Mi (83–97)
150*	14m (9–38)	121Mi (104–148)
175	15m (11–35)	137Mi (112–175)
200	17m (13–50)	142Mi (133–184)

Tuning the control plane for scale

Component	Field	Default	Recommended
API Server	resources.limits.cpu	500m	3 (request 1)
Nova Scheduler	resources.limits.ephemeral-storage	64Mi	512Mi