Version: v1.3

Install Nova with CLI

Overview

Purpose

This guide provides step-by-step instructions for installing Nova, a control plane and agent system designed to manage multiple Kubernetes clusters. By following this guide, you will set up the Nova Control Plane on a hosting Kubernetes cluster and deploy Nova Agents to workload clusters.

Scope

This guide covers:

Prerequisites: Requirements before installing Nova.
Installing novactl: How to download and set up the Nova CLI.
Deploying Nova: Instructions for deploying the Nova Control Plane and Agents.
Post-Installation Checks: Verifying the installation.
Uninstalling Nova: Steps to remove Nova if needed.

Key Concepts

Nova Control Plane: The central management unit running on a hosting Kubernetes cluster.
Nova Agent: The component deployed to each workload cluster for management.
novactl: The command-line interface (CLI) for installing, uninstalling and checking the status of a Nova deployment.
Workload Cluster: A Kubernetes cluster managed by the Nova Control Plane.
Hosting Cluster: A Kubernetes cluster where the Nova Control Plane runs.

Prerequisites

At least 2 Kubernetes clusters up and running. One cluster will be the hosting cluster where the Nova control plane runs. Other clusters will be workload clusters that are managed by the Nova control plane.
Installed and configured kubectl
Nova cannot be deployed to an Autopilot GKE cluster. Please validate that you are deploying to a non-Autopilot cluster.
Cluster hosting Nova Control Plane MUST have storage provisioner and default StorageClass configured. Nova Control Plane uses etcd as a backing store, which runs as a StatefulSet and requires a PersistentVolume to work.

Kubernetes compatibility

Nova Version	Kubernetes Versions Supported
v1.3	v1.31, v1.32, v1.33
v1.2	v1.30, v1.31, v1.32
v1.1	v1.29, v1.30, v1.31
v1.0	v1.29, v1.30, v1.31

Download `novactl`

novactl is our CLI that allows you to easily create new Nova Control Planes, register new Nova Workload Clusters, check the health of your Nova cluster, and more!

If you don't have the release tarball then in order to download the latest novactl version for your OS, run:

curl -s https://api.github.com/repos/elotl/novactl/releases/latest | \
jq -r '.assets[].browser_download_url' | \
grep "$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/;s/i386/386/;s/aarch64/arm64/')" | \
xargs -I {} curl -L {} -o novactl

Install `novactl`

Make the binary executable

Once you have the binary, run:

chmod +x novactl*

Place the binary in your PATH

The following is an example to install the plugin in /usr/local/bin for Unix-like operating systems:

sudo mv novactl* /usr/local/bin/novactl

If you accidentally downloaded more than one novactl binary, please move only the binary that corresponds to the OS and ARCH of your machine to the /usr/local/bin location.

Install it as kubectl plugin

novactl is ready to work as a kubectl plugin. Our docs assume you're using novactl as a kubectl plugin. To make this work, simply run:

sudo novactl kubectl-install

And test if it works:

kubectl nova --help

Usage:
  kubectl nova [command]

Available Commands:
  get             get resources (clusters, schedule groups or deployments) with additional context
  help            Help about any command
  install         Install new Nova Control Plane or connect a new workload cluster to Nova Control Plane
  kubectl-install kubectl-install installs this binary as kubectl plugin
  status          Check status of Nova Control Plane installation
  uninstall       Uninstall Nova Control Plane or disconnect a workload cluster from Nova Control Plane

Flags:
  -h, --help      help for kubectl-nova
  -v, --version   version for kubectl-nova

Use "kubectl-nova [command] --help" for more information about a command.

Upgrading `novactl`

If you want to upgrade novactl to the latest version, please run all the previous steps starting from Download novactl up to this point again. This will automatically download the latest version and replace your local binary with it.

Deploy Nova

To deploy the Nova Control Plane to one cluster and use another as a workload cluster, make sure you have at least two contexts in your kubeconfig.

kubectl config get-contexts

CURRENT   NAME                     CLUSTER                AUTHINFO               NAMESPACE
          k8s-cluster-1            wlc-1                  wlc-1
          k8s-cluster-2            wlc-2                  wlc-2
          k8s-cluster-hosting-cp   hosting-cluster        hosting-cluster

Install the Nova Control Plane

For installing Nova Control Plane use kubectl nova install control-plane command

kubectl nova install control-plane --help

Install new Nova Control Plane.

Installs Nova Control Plane components in current cluster, outputs configuration to Nova home directory at ~/.nova

Usage:
  kubectl nova install control-plane --context=[hosting cluster context]  [CLUSTER NAME] [flags]

Aliases:
  control-plane, cp

Examples:

  Start Nova Control Plane with default configuration:

      kubectl nova install control-plane nova-cp


Flags:
      --dry-run                   If passed, objects are printed out instead of being installed
      --gcp-access-key string     GCP access key JSON file path. Set for Nova Control Plane running in GKE.
      --gcp-project-id string     GCP Project ID. Set for Nova Control Plane running in GKE.
  -h, --help                      help for control-plane
      --image-repository string    (default "elotl/nova-scheduler")
      --image-tag string           (default "v0.6.1")
      --version string             (default "v0.6.1")

Global Flags:
      --context string
      --namespace string    (default "elotl")
      --rollback           Rollback changes made during installation in case it fails.

The Nova Control Plane will be deployed by default to the elotl namespace, which will be created if it does not already exist.

To deploy the Nova Control Plane run:

export NOVA_NAMESPACE=elotl
export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_CONTROLPLANE_CONTEXT=nova

kubectl nova install control-plane --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} ${NOVA_CONTROLPLANE_CONTEXT}

Here, nova is the name of your Nova Control Plane as well as the context name to interact with the Nova Control Plane. If you choose a different name, remember to replace all uses of --context=${NOVA_CONTROLPLANE_CONTEXT} in other commands in the documentation to the custom name chosen.

Installing Nova Control Plane... 🪄
Checking status of Nova Control Plane Components

* API Server status... Running √
* Kube-controller-manager status... Running √
* ETCD status... Running √
* Nova scheduler status... Running √
Nova Control Plane is healthy √

Checking presence of Nova Custom Resource Definitions

* Cluster CRD presence... installed √
        0 workload clusters connected ‼
                please connect at least one Cluster, otherwise Nova does not have a target cluster to run your workloads. Connecting clusters can be done by running novactl install agent <cluster-name> in correct Kube context.
* SchedulePolicy CRD presence... installed √
        * 0 SchedulePolicies defined ‼
                please create at least one SchedulePolicy, otherwise Nova does not know where to run your workloads. SchedulePolicy spec: https://docs.elotl.co/nova/intro
* ScheduleGroup CRD presence... installed √
All Nova Custom Resource Definitions installed √

Nova Control Plane components installed! 🚀

Nova kubeconfig is stored at /Users/example-user/.nova/nova/nova-kubeconfig

To interact with Nova, run:
    export KUBECONFIG=~/.kube/config:/Users/example-user/.nova/nova/nova-kubeconfig
    kubectl get cluster --context nova

Thank you for deploying and using Nova! Your participation is invaluable to us.
If you have any feedback, questions, or simply want to engage further:
Please join our Slack channel: http://tinyurl.com/slack-elotl
Or email us at info@elotl.co
Your input helps us improve our product for you. Thank you!

Troubleshooting Tips

You can find troubleshooting tips in the Troubleshooting section.

Install the Nova agent

Each workload cluster needs a Nova agent. The Nova agent will be deployed by default to the elotl namespace. Before deploying the Nova agent, you need to ensure that Nova's init-kubeconfig is present in the elotl namespace. Nova's init-kubeconfig provides a kube config to the Nova Control Plane. This kube config is used by the Nova agent in the workload cluster to connect and register itself as a workload cluster to the Nova Control Plane.

export NOVA_NAMESPACE=elotl
export NOVA_CONTROLPLANE_CONTEXT=nova
export K8S_CLUSTER_CONTEXT_1=k8s-cluster-1
export K8S_CLUSTER_CONTEXT_2=k8s-cluster-2
export K8S_HOSTING_CLUSTER_CONTEXT=k8s-cluster-hosting-cp
export NOVA_WORKLOAD_CLUSTER_1=wlc-1
export NOVA_WORKLOAD_CLUSTER_2=wlc-2

Let's create the namespace first:

kubectl --context=${K8S_CLUSTER_CONTEXT_1} create namespace ${NOVA_NAMESPACE}

We then copy the init-kubeconfig secret from the Nova Control Plane to the workload cluster:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get secret -n ${NOVA_NAMESPACE} nova-cluster-init-kubeconfig -o yaml | kubectl --context=${K8S_CLUSTER_CONTEXT_1} apply -f -

To connect a workload cluster to Nova, we will use kubectl nova install agent command.

kubectl nova install agent --help

Install new Nova Agent.

Installs Nova Agent components in current cluster making it part of Nova Workload Fleet in current Nova Control Plane.

Usage:
  novactl install agent [CLUSTER NAME] [flags]

Examples:

  Start Nova Agent in current cluster with default configuration:

      kubectl nova install agent nova-agent-1

  Start Nova Agent in some other context using specific version:

      kubectl nova --context some-other-context install agent --image-tag v0.5.0 nova-agent-2


Flags:
      --dry-run                   If passed, objects are printed out instead of being installed
  -h, --help                      help for agent
      --image-repository string   --image-repository elotl/nova-agent (default "elotl/nova-agent")
      --image-tag string          --image-tag v0.0.0 (default "v0.6.1")

Global Flags:
      --context string
      --namespace string    (default "elotl")
      --rollback           Rollback changes made during installation in case it fails.

Deploy the Nova agent to the workload cluster:

kubectl nova install agent --context=${K8S_CLUSTER_CONTEXT_1} --namespace=${NOVA_NAMESPACE} ${NOVA_WORKLOAD_CLUSTER_1}

Installing Nova Agent... 🪄
Nova Agent components installed! 🚀

Now let's check if that worked. Simply run:

kubectl get --context=${NOVA_CONTROLPLANE_CONTEXT} clusters

Remember to update path to your Nova Control Plane kubeconfig

NAME    K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE   STANDBY
wlc-1   1.28          wlc-1                         True    True   False

What if I don't see my workload cluster listed?

Visit Troubleshooting section.

Install other workload clusters

If you have a second cluster, run the same commands with a different cluster and cluster name, e.g.,

kubectl --context=${K8S_CLUSTER_CONTEXT_2} create namespace ${NOVA_NAMESPACE}
  kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get secret --namespace=${NOVA_NAMESPACE} nova-cluster-init-kubeconfig -o yaml | kubectl --context=${K8S_CLUSTER_CONTEXT_2} apply -f -
  kubectl nova install agent --context=${K8S_CLUSTER_CONTEXT_2} --namespace=${NOVA_NAMESPACE} ${NOVA_WORKLOAD_CLUSTER_2}

Installing Nova Agent... 🪄
Nova Agent components installed! 🚀

Post installation check

We should now see the newly added workload cluster registered in Nova Control Plane:

kubectl --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters

NAME    K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE   STANDBY
wlc-1   1.28          wlc-1                         True    True   False
wlc-2   1.28          wlc-2                         True    True   False

To get more insight into the clusters available resources:

kubectl nova --context=${NOVA_CONTROLPLANE_CONTEXT} get clusters

  | CLUSTER NAME                  | K8S VERSION | CLOUD PROVIDER | REGION        | STATUS        |
  |----------------------------------------------------------------------------------------------|
  | wlc-1                         | 1.28        | gce            | us-central1   | ClusterReady  |
  |----------------------------------------------------------------------------------------------|
  |                                              NODES                                           |
  |----------------------------------------------------------------------------------------------|
  | NAME                                                 | AVAILABLE   | AVAILABLE   | AVAILABLE |
  |                                                      | CPU         | MEMORY      | GPU       |
  |                                                                                              |
  | gke-nova-example-agent-1-default-pool-25df6493-263w  | 399m        | 2332068Ki   | 0         |
  | gke-nova-example-agent-1-default-pool-25df6493-f9f8  | 427m        | 2498615680  | 0         |
  |                                                                                              |
  |                      NODES' TAINTS                                                           |
  |                                                                                              |
  |----------------------------------------------------------------------------------------------|



  | CLUSTER NAME                  | K8S VERSION | CLOUD PROVIDER | REGION        | STATUS        |
  |----------------------------------------------------------------------------------------------|
  | wlc-2                         | 1.28        | gce            | us-central1   | ClusterReady  |
  |----------------------------------------------------------------------------------------------|
  |                                              NODES                                           |
  |----------------------------------------------------------------------------------------------|
  | NAME                                                 | AVAILABLE   | AVAILABLE   | AVAILABLE |
  |                                                      | CPU         | MEMORY      | GPU       |
  |                                                                                              |
  | gke-nova-example-agent-2-default-pool-55fcf389-74zh  | 457m        | 2460060Ki   | 0         |
  | gke-nova-example-agent-2-default-pool-55fcf389-n77s  | 359m        | 2336086400  | 0         |
  | gke-nova-example-agent-2-gpu-pool-950c3823-mlqq      | 677m        | 2354840Ki   | 0         |
  |                                                                                              |
  |                      NODES' TAINTS                                                           |
  |                                                                                              |
  | gke-nova-example-agent-2-gpu-pool-950c3823-mlqq                                              |
  |     - nvidia.com/gpu:present:NoSchedule                                                      |
  |----------------------------------------------------------------------------------------------|

Uninstall Nova

Uninstall Nova Agent from Workload cluster

 kubectl nova uninstall agent --context=${K8S_CLUSTER_CONTEXT_1} ${NOVA_WORKLOAD_CLUSTER_1}

Uninstall Nova Control Plane

To uninstall Nova Control Plane components, run:

kubectl nova uninstall control-plane --context=${K8S_HOSTING_CLUSTER_CONTEXT} ${NOVA_CONTROLPLANE_CONTEXT}

Here, nova specified as the last argument is the name of your Nova control plane (chosen at the install step). Default namespace is elotl.

Install Nova with CLI

Overview​

Purpose​

Scope​

Key Concepts​

Prerequisites​

Kubernetes compatibility​

Download novactl​

Install novactl​

Make the binary executable​

Place the binary in your PATH​

Install it as kubectl plugin​

Upgrading novactl​

Deploy Nova​

Install the Nova Control Plane​

Troubleshooting Tips​

Install the Nova agent​

What if I don't see my workload cluster listed?​

Install other workload clusters​

Post installation check​