Install Nova with CLI
Overview
Purpose
This guide provides step-by-step instructions for installing Nova, a control plane and agent system designed to manage multiple Kubernetes clusters. By following this guide, you will set up the Nova Control Plane on a hosting Kubernetes cluster and deploy Nova Agents to workload clusters.
Scope
This guide covers:
- Prerequisites: Requirements before installing Nova.
- Installing novactl: How to download and set up the Nova CLI.
- Deploying Nova: Instructions for deploying the Nova Control Plane and Agents.
- Post-Installation Checks: Verifying the installation.
- Uninstalling Nova: Steps to remove Nova if needed.
Key Concepts
- Nova Control Plane: The central management unit running on a hosting Kubernetes cluster.
- Nova Agent: The component deployed to each workload cluster for management.
- novactl: The command-line interface (CLI) for installing, uninstalling and checking the status of a Nova deployment.
- Workload Cluster: A Kubernetes cluster managed by the Nova Control Plane.
- Hosting Cluster: A Kubernetes cluster where the Nova Control Plane runs.
Prerequisites
- At least 2 Kubernetes clusters up and running. One cluster will be the hosting cluster where nova control plane runs. Other clusters will be workload clusters that are managed by the nova control plane.
- Installed and configured
kubectl - Nova cannot be deployed to an Autopilot GKE cluster. Please validate that you are deploying to a non-Autopilot cluster.
- Cluster hosting Nova Control Plane MUST have storage provisioner and default
StorageClassconfigured. Nova Control Plane uses etcd as a backing store, which runs asStatefulSetand requiresPersistentVolumeto work.
Kubernetes compatibility
| Nova Version | Kubernetes Versions Supported |
|---|---|
| v0.10 | v1.29, v1.30, v1.31 |
| v0.9 | v1.26, v1.27, v1.28, v1.29 |
| v0.8 | v1.26, v1.27, v1.28, v1.29 |
| v0.7 | v1.25, v1.26, v1.27, v1.28 |
| v0.6 | v1.24, v1.25 |
Download novactl
novactl is our CLI that allows you to easily create new Nova Control Planes, register new Nova Workload Clusters, check the health of your Nova cluster, and more!
If you don't have the release tarball then in order to download the latest novactl version for your OS, run:
curl -s https://api.github.com/repos/elotl/novactl/releases/latest | \
jq -r '.assets[].browser_download_url' | \
grep "$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/;s/i386/386/;s/aarch64/arm64/')" | \
xargs -I {} curl -L {} -o novactl
Install novactl
Make the binary executable
Once you have the binary, run:
chmod +x novactl*
Place the binary in your PATH
The following is an example to install the plugin in /usr/local/bin for Unix-like operating systems:
sudo mv novactl* /usr/local/bin/novactl
If you accidentally downloadedmore than one novactl binary, please move only the binary that corresponds to the OS and ARCH of your machine to the /usr/local/bin location.
Install it as kubectl plugin
novactl is ready to work as kubectl plugin. Our docs assume you're using novactl as kubectl plugin. To make this work, simply run:
sudo novactl kubectl-install
And test if it works:
kubectl nova --help
Usage:
kubectl nova [command]
Available Commands:
get get resources (clusters, schedule groups or deployments) with additional context
help Help about any command
install Install new Nova Control Plane or connect a new workload cluster to Nova Control Plane
kubectl-install kubectl-install installs this binary as kubectl plugin
status Check status of Nova Control Plane installation
uninstall Uninstall Nova Control Plane or disconnect a workload cluster from Nova Control Plane
Flags:
-h, --help help for kubectl-nova
-v, --version version for kubectl-nova
Use "kubectl-nova [command] --help" for more information about a command.
Upgrading novactl
If you want to upgrade novactl to latest version, it's enough to run all previous steps starting from Download novactl up to this point again. This will automatically download latest version and replace your local binary with it.
Deploy
To deploy Nova Control Plane to one cluster and use another as a workload cluster, make sure you have at least two contexts in your kubeconfig.
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
k8s-cluster-1 wlc-1 wlc-1
k8s-cluster-2 wlc-2 wlc-2
k8s-cluster-hosting-cp hosting-cluster hosting-cluster
Install Nova Control Plane
For installing Nova Control Plane use kubectl nova install control-plane command
kubectl nova install control-plane --help
Install new Nova Control Plane.
Installs Nova Control Plane components in current cluster, outputs configuration to Nova home directory at ~/.nova
Usage:
kubectl nova install control-plane --context=[hosting cluster context] [CLUSTER NAME] [flags]
Aliases:
control-plane, cp
Examples:
Start Nova Control Plane with default configuration:
kubectl nova install control-plane nova-cp
Flags:
--dry-run If passed, objects are printed out instead of being installed
--gcp-access-key string GCP access key JSON file path. Set for Nova Control Plane running in GKE.
--gcp-project-id string GCP Project ID. Set for Nova Control Plane running in GKE.
-h, --help help for control-plane
--image-repository string (default "elotl/nova-scheduler")
--image-tag string (default "v0.6.1")
--version string (default "v0.6.1")
Global Flags:
--context string
--namespace string (default "elotl")
--rollback Rollback changes made during installation in case it fails.
The Nova Control Plane will be deployed by default to the elotl namespace, which will be created if it does not already exist.
To deploy Nova Control Plane run:
export NOVA_NAMESPACE=elotl
export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_CONTROLPLANE_CONTEXT=nova
kubectl nova install control-plane --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} ${NOVA_CONTROLPLANE_CONTEXT}
Here, nova is the name of your Nova Control Plane as well as the context name to interact with the Nova Control Plane. If you choose a different name, remember to replace all uses of --context=${NOVA_CONTROLPLANE_CONTEXT} in other commands in the documentation to the custom name chosen.
Installing Nova Control Plane... 🪄
Checking status of Nova Control Plane Components
* API Server status... Running √
* Kube-controller-manager status... Running √
* ETCD status... Running √
* Nova scheduler status... Running √
Nova Control Plane is healthy √
Checking presence of Nova Custom Resource Definitions
* Cluster CRD presence... installed √
0 workload clusters connected ‼
please connect at least one Cluster, otherwise Nova does not have a target cluster to run your workloads. Connecting clusters can be done by running novactl install agent <cluster-name> in correct Kube context.
* SchedulePolicy CRD presence... installed √
* 0 SchedulePolicies defined ‼
please create at least one SchedulePolicy, otherwise Nova does not know where to run your workloads. SchedulePolicy spec: https://docs.elotl.co/nova/intro
* ScheduleGroup CRD presence... installed √
All Nova Custom Resource Definitions installed √
Nova Control Plane components installed! 🚀
Nova kubeconfig is stored at /Users/example-user/.nova/nova/nova-kubeconfig
To interact with Nova, run:
export KUBECONFIG=~/.kube/config:/Users/example-user/.nova/nova/nova-kubeconfig
kubectl get cluster --context nova
Thank you for deploying and using Nova! Your participation is invaluable to us.
If you have any feedback, questions, or simply want to engage further:
Please join our Slack channel: http://tinyurl.com/slack-elotl
Or email us at info@elotl.co
Your input helps us improve our product for you. Thank you!
Troubleshooting Tips
You can find troubleshooting tips in the Troubleshooting section.
Install nova agent into workload cluster
Each workload cluster needs a Nova agent. The Nova agent will be deployed by default to the elotl namespace. Before deploying Nova agent, you need to ensure that the Nova's init-kubeconfig is present in the elotl namespace. Nova's init-kubeconfig provides a kube config to the Nova Control Plane. This kube config is used by Nova agent in the workload cluster to connect and register itself as a workload cluster in the Nova Control Plane.
export NOVA_NAMESPACE=elotl
export NOVA_CONTROLPLANE_CONTEXT=nova
export K8S_CLUSTER_CONTEXT_1=k8s-cluster-1
export K8S_CLUSTER_CONTEXT_2=k8s-cluster-2
export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2
Let's create the namespace first:
kubectl --context=${K8S_CLUSTER_CONTEXT_1} create namespace ${NOVA_NAMESPACE}
and copy init-kubeconfig from Nova Control Plane to workload cluster:
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get secret -n ${NOVA_NAMESPACE} nova-cluster-init-kubeconfig -o yaml | kubectl --context=${K8S_CLUSTER_CONTEXT_1} apply -f -
To connect a workload cluster to Nova, we will use kubectl nova install agent command.
kubectl nova install agent --help
Install new Nova Agent.
Installs Nova Agent components in current cluster making it part of Nova Workload Fleet in current Nova Control Plane.
Usage:
novactl install agent [CLUSTER NAME] [flags]
Examples:
Start Nova Agent in current cluster with default configuration:
kubectl nova install agent nova-agent-1
Start Nova Agent in some other context using specific version:
kubectl nova --context some-other-context install agent --image-tag v0.5.0 nova-agent-2
Flags:
--dry-run If passed, objects are printed out instead of being installed
-h, --help help for agent
--image-repository string --image-repository elotl/nova-agent (default "elotl/nova-agent")
--image-tag string --image-tag v0.0.0 (default "v0.6.1")
Global Flags:
--context string
--namespace string (default "elotl")
--rollback Rollback changes made during installation in case it fails.
Deploy the Nova agent to the workload cluster:
kubectl nova install agent --context=${K8S_CLUSTER_CONTEXT_1} --namespace=${NOVA_NAMESPACE} ${NOVA_WORKLOAD_CLUSTER_1}
Installing Nova Agent... 🪄
Nova Agent components installed! 🚀
Now lets check if that worked! Simply run:
kubectl get --context=${NOVA_CONTROLPLANE_CONTEXT} clusters
Remember to update path to your Nova Control Plane kubeconfig
NAME K8S-VERSION K8S-CLUSTER REGION ZONE READY IDLE STANDBY
wlc-1 1.28 wlc-1 True True False
What if I don't see my workload cluster listed?
Visit Troubleshooting section.
Install other workload clusters
If you have a second cluster, run the same commands with a different cluster and cluster name, e.g.,
kubectl --context=${K8S_CLUSTER_CONTEXT_2} create namespace ${NOVA_NAMESPACE}
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get secret --namespace=${NOVA_NAMESPACE} nova-cluster-init-kubeconfig -o yaml | kubectl --context=${K8S_CLUSTER_CONTEXT_2} apply -f -
kubectl nova install agent --context=${K8S_CLUSTER_CONTEXT_2} --namespace=${NOVA_NAMESPACE} ${NOVA_WORKLOAD_CLUSTER_2}
Installing Nova Agent... 🪄
Nova Agent components installed! 🚀
Post installation check
We should now see the newly added workload cluster registered in Nova Control Plane:
kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters
NAME K8S-VERSION K8S-CLUSTER REGION ZONE READY IDLE STANDBY
wlc-1 1.28 wlc-1 True True False
wlc-2 1.28 wlc-2 True True False
To get more insight into the clusters available resources:
kubectl nova --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters
| CLUSTER NAME | K8S VERSION | CLOUD PROVIDER | REGION | STATUS |
|----------------------------------------------------------------------------------------------|
| wlc-1 | 1.28 | gce | us-central1 | ClusterReady |
|----------------------------------------------------------------------------------------------|
| NODES |
|----------------------------------------------------------------------------------------------|
| NAME | AVAILABLE | AVAILABLE | AVAILABLE |
| | CPU | MEMORY | GPU |
| |
| gke-nova-example-agent-1-default-pool-25df6493-263w | 399m | 2332068Ki | 0 |
| gke-nova-example-agent-1-default-pool-25df6493-f9f8 | 427m | 2498615680 | 0 |
| |
| NODES' TAINTS |
| |
|----------------------------------------------------------------------------------------------|
| CLUSTER NAME | K8S VERSION | CLOUD PROVIDER | REGION | STATUS |
|----------------------------------------------------------------------------------------------|
| wlc-2 | 1.28 | gce | us-central1 | ClusterReady |
|----------------------------------------------------------------------------------------------|
| NODES |
|----------------------------------------------------------------------------------------------|
| NAME | AVAILABLE | AVAILABLE | AVAILABLE |
| | CPU | MEMORY | GPU |
| |
| gke-nova-example-agent-2-default-pool-55fcf389-74zh | 457m | 2460060Ki | 0 |
| gke-nova-example-agent-2-default-pool-55fcf389-n77s | 359m | 2336086400 | 0 |
| gke-nova-example-agent-2-gpu-pool-950c3823-mlqq | 677m | 2354840Ki | 0 |
| |
| NODES' TAINTS |
| |
| gke-nova-example-agent-2-gpu-pool-950c3823-mlqq |
| - nvidia.com/gpu:present:NoSchedule |
|----------------------------------------------------------------------------------------------|
Uninstall Nova
Uninstall Nova Agent from Workload cluster
kubectl nova uninstall agent --context=${K8S_CLUSTER_CONTEXT_1} ${NOVA_WORKLOAD_CLUSTER_1}
Uninstall Nova Control Plane
To uninstall Nova Control Plane components, run:
kubectl nova uninstall control-plane --context=${K8S_HOSTING_CLUSTER_CONTEXT} ${NOVA_CONTROLPLANE_CONTEXT}
Here, nova specified as the last argument is the name of your Nova control plane (chosen at the install step).
Default namespace is elotl.