Nova: a Federated Orchestrator
Nova is a multi-cluster Kubernetes control plane that orchestrates workloads across multiple Kubernetes clusters. With Nova, workload clusters become opaque compute commodities to consumers of Kubernetes infrastructure.
Architecture
A typical deployment of Nova consists of one Nova control plane and multiple workload clusters. Nova agents are deployed to workload clusters to communicate with the Nova control plane. The core Nova control plane has a Kubernetes API server, etcd, and a few other Nova components (scheduler/rescheduler, controllers, etc). The overall architecture is shown in the following graphic.
The diagram below illustrates Nova in action.
Scalability & Performance
To demonstrate Nova's efficiency, we performed stress tests measuring the Nova Control Plane (API Server, etcd, Scheduler) across varying fleet sizes.
Test Environment
- Infrastructure: 2-node management cluster.
- Workload Clusters: Simulated using
vclusterson a dedicated worker node. - Workload: Retail Store Application (Microservices) distributed via Nova Spread Policy.
- Measurement: resource consumption was measured using
kubectl top pod -n elotl --sum=truecommand in the hosting cluster.
Scenario 1: Idle Fleet Connectivity
This test measures the resource footprint of the Nova Control Plane while maintaining connections to idle workload clusters.
| Active Clusters | CPU Usage | Memory Usage |
|---|---|---|
| 10 | 81m | 358Mi |
| 100 | 120m | 386Mi |
| 500 | 118m | 432Mi |
Scenario 2: Active Workload Orchestration
This test evaluates Nova's performance while actively managing an application across the fleet. We used a Spread Policy to deploy the Retail Store Application to every cluster.
| Active Clusters | CPU Usage | Memory Usage |
|---|---|---|
| 10 | 71m | 486Mi |
| 20 | 134m | 588Mi |
| 100 | 557m | 1100Mi |
| 200 | 1516m | 1689Mi |
Supported api-resources
Nova supports the following standard kubernetes objects as well as CRDs:
- configmaps
- namespaces
- persistentvolumeclaims
- pods
- secrets
- serviceaccounts
- services
- daemonsets
- deployments
- replicasets
- statefulsets
- horizontalpodautoscalers
- cronjobs
- jobs
- ingressclasses
- ingresses
- networkpolicies
- clusterrolebindings
- clusterroles
- rolebindings
- roles
- mutatingwebhookconfigurations
- validatingwebhookconfigurations
- priorityclasses
Agent and control plane communication
Nova agents running in workload clusters communicate with the Nova control plane only through the K8s API server. Nova agents don't interact with any other components in the Nova control plane, including the Nova scheduler or any other controllers.
How does authentication work between Nova agents and K8s API server?
Nova agents use the standard RBAC authorization to communicate with the K8s API server.
Each agent has a certificate/key pair signed by the Kubernetes root CA.
Currently, the agents have a cluster-admin ClusterRole with system:masters ClusterRoleBinding.