Skip to main content
Version: v1.1

Nova: a Federated Orchestrator

Nova is a multi-cluster Kubernetes control plane that orchestrates workloads across multiple Kubernetes clusters. With Nova, workload clusters become opaque compute commodities to consumers of Kubernetes infrastructure.

Architecture​

A typical deployment of Nova consists of one Nova control plane and multiple workload clusters. Nova agents are deployed in to workload clusters to communicate with the Nova control plane. The core Nova control plane has a Kubernetes API server, etcd, and a few other Nova components (scheduler/rescheduler, controllers, etc). The overall architecture looks like the following diagram.

Nova Architecture

The below diagram illustrates Nova in action.

Luna In Action

Supported api-resources​

Nova supports the following standard kubernetes objects as well as CRDs:

  • configmaps
  • namespaces
  • persistentvolumeclaims
  • pods
  • secrets
  • serviceaccounts
  • services
  • daemonsets
  • deployments
  • replicasets
  • statefulsets
  • horizontalpodautoscalers
  • cronjobs
  • jobs
  • ingressclasses
  • ingresses
  • networkpolicies
  • clusterrolebindings
  • clusterroles
  • rolebindings
  • roles
  • mutatingwebhookconfigurations
  • validatingwebhookconfigurations

System requirements​

Nova control plane hosting cluster​

Nova control plane itself is deployed into a Kubernetes cluster called the hosting cluster. In order for the Nova agent in the workload cluster to be able to communicate with Nova control plane, the hosting cluster needs to be able to provision an external LoadBalancer for a Kubernetes Service of Type LoadBalancer. This is implemented in most Kubernetes distributions(e.g., EKS, GKE).

Nova control plane uses ETCD as a backing store, so the hosting cluster should have a storage provisioner (e.g. AWS EBS CSI driver in EKS, link).

Nova control plane cannot be deployed into a GKE autopilot cluster.

Workload clusters​

Workload clusters needs to allow outbound traffic on port 443.

We recommend using a consistent Kubernetes version across workload clusters. We've tested EKS/GKE as the hosting cluster, and EKS/GKE/Minikube/Kind as the workload cluster.

Agent and control plane communication​

Nova agents running in workload clusters communicate with the Nova control plane only through the K8s API server. Nova agents don't interact with any other components in the Nova control plane, including Nova scheduler or any other controllers.

How does authentication work between Nova agents and K8s API server?​

Nova agents use the standard RBAC authorization to communicate with the K8s API server. Each agent has a certificate/key pair signed by the Kubernetes root CA. Currently, the agents have a cluster-admin ClusterRole with system:masters ClusterRoleBinding.