Skip to main content
Version: v1.5

Troubleshooting

This guide describes how to diagnose and resolve common issues when installing, configuring, and operating Nova.

If you encounter an issue that is not listed here, please contact the Elotl team.

Before troubleshooting scheduling or placement behavior, it may be helpful to review:

Installation​

Control Plane installation times out waiting for the condition​

Symptoms​

While installing the Nova Control Plane, the installation times out while installing the Nova API Server.

For example:

kubectl nova install control-plane --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} ${NOVA_CONTROLPLANE_CONTEXT}
Installing Nova Control Plane... 🪄
Cluster name - ${NOVA_CONTROLPLANE_CONTEXT}
Creating namespace elotl in control plane
Creating certificates
Generating certificates
Certificates successfully generated.
Installing Kube API Server...
timed out waiting for the condition

This usually means that the Nova API Server or one of its dependencies did not start properly.

The most common cause is that etcd cannot start because there is no storage provisioner or default StorageClass available in the hosting cluster.

Verify​

Check the Nova Control Plane pods in the hosting cluster:

kubectl get pods --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE}

Example output:

NAME                                      READY   STATUS             RESTARTS        AGE
apiserver-6bf98bb5d5-vv7wc                0/1     CrashLoopBackOff   6 (110s ago)    9m42s
etcd-0                                    0/1     Pending            0               9m42s
kube-controller-manager-76d5d96df-ntl6g   0/1     CrashLoopBackOff   6 (3m42s ago)   9m42s

In this example, apiserver and kube-controller-manager are starting and failing, while etcd is still in Pending state.

Also verify the etcd PersistentVolumeClaim:

kubectl get pvc --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE}

If the etcd pod is Pending or the PVC is not bound, verify that the hosting cluster has a working storage provisioner and default StorageClass.

Resolution​

Follow your cloud provider or Kubernetes distribution documentation to configure a storage provisioner and default StorageClass on the hosting cluster.

After storage is configured, uninstall the failed Nova Control Plane installation:

kubectl nova uninstall ${NOVA_CONTROLPLANE_CONTEXT} --context=${K8S_HOSTING_CLUSTER_CONTEXT}

Then install the Nova Control Plane again.

Diagnose with the novactl status CLI​

The Nova CLI includes a diagnostic subcommand, novactl status, which checks whether the Nova Control Plane is running and whether Nova CRDs are installed.

Verify​

Run:

kubectl nova status --context=${NOVA_CONTROLPLANE_CONTEXT} --hosting-cluster-context=${K8S_HOSTING_CLUSTER_CONTEXT} --hosting-cluster-nova-namespace=${NOVA_NAMESPACE}

Example output:

Checking status of Nova Control Plane Components

* API Server status... Running √
* Kube-controller-manager status... Running √
* ETCD status... Running √
* Nova scheduler status... Running √
Nova Control Plane is healthy √

Checking presence of Nova Custom Resource Definitions

* Cluster CRD presence... installed √
        * Cluster wlc-1 connected and ready √
        * Cluster wlc-2 connected and ready √
* SchedulePolicy CRD presence... installed √
        * 0 SchedulePolicies defined ‼
                please create at least one SchedulePolicy, otherwise Nova does not know where to run your workloads. SchedulePolicy spec: https://docs.elotl.co
* ScheduleGroup CRD presence... installed √
All Nova Custom Resource Definitions installed √

If one of the Nova Control Plane components is not running, Nova cannot function properly.

All Nova Control Plane components run in the ${NOVA_NAMESPACE} namespace on the hosting cluster.

To debug further, collect logs from the hosting cluster:

kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/nova-scheduler
kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/apiserver
kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/kube-controller-manager
kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} statefulset/etcd

Workload cluster does not appear in the Nova Control Plane​

Symptoms​

The Nova agent was installed on a workload cluster, but the workload cluster does not appear in the Nova Control Plane.

Verify​

First, check that the Nova agent deployment exists and is running in the workload cluster:

kubectl get deployment nova-agent --context=${K8S_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE}

If the agent installation completed successfully and the agent pod is running, something may have gone wrong during the agent registration process.

Collect the Nova agent logs from the workload cluster:

kubectl get pods --context=${K8S_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} -o name -l "app.kubernetes.io/name"="nova-agent" | xargs -I {} kubectl logs --context=${K8S_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} {}

Start debugging from the Nova agent logs.

Resolution​

Common causes include:

  • The Nova agent cannot reach the Nova Control Plane endpoint.
  • The agent registration process failed.
  • Credentials or certificates were not created correctly.
  • Network policy, firewall, DNS, or load balancer configuration prevents communication.

Scheduling​

Workload was never scheduled​

Symptoms​

A Kubernetes resource exists in the Nova Control Plane, but Nova does not schedule it to a workload cluster.

Nova scheduling is a multi-step process. First, Nova tries to find a matching SchedulePolicy when a new resource is created.

Verify​

Check whether the resource matched a SchedulePolicy:

kubectl get events --context=${NOVA_CONTROLPLANE_CONTEXT} --namespace=<resource-namespace> --field-selector=involvedObject.name=<resource-name>

If a matching SchedulePolicy is found, Nova returns a Kubernetes Event indicating that the object matched a policy.

For example:

kubectl get events --context=${NOVA_CONTROLPLANE_CONTEXT} --namespace=<resource-namespace> --field-selector=involvedObject.name=<resource-name>
16s Normal SchedulePolicyMatched <resource-namespace>/<resource-name> schedule policy <policy-name> will be used to determine target cluster

If no SchedulePolicy matched​

If no SchedulePolicy matched the resource, verify the following:

  • Ensure the resource kind is supported by Nova. A full list of supported resource types is available in API Resources.
  • Check that the SchedulePolicy has the correct namespaceSelector and resourceSelector.
  • Verify that the resource is in one of the namespaces specified in the SchedulePolicy namespaceSelector.
  • For cluster-scoped objects, verify the label selector. Cluster-scoped objects are matched only by label selector.
  • Verify that the resource has labels that match the SchedulePolicy resourceSelectors.
  • Check whether the resource matches more than one SchedulePolicy. If multiple policies match, Nova sorts the SchedulePolicy resources alphabetically and uses the first one.
  • Verify that the resource is not in a restricted namespace. Namespaces starting with kube- or elotl are restricted, and Nova ignores them.

Workload matched a SchedulePolicy but was not placed​

Symptoms​

A resource matched a SchedulePolicy, but it did not transition into a running state on a workload cluster.

There are several common causes:

  • The clusterSelector does not match any workload clusters.
  • The matching workload clusters do not have enough free capacity.
  • The target namespace does not exist in the workload cluster.
  • The Nova agent in the workload cluster is not functioning correctly.

Verify cluster selector​

If the SchedulePolicy has a clusterSelector that does not match any workload clusters, Nova cannot place the workload.

To see workload clusters connected to Nova, run:

kubectl get clusters --show-labels --context=${NOVA_CONTROLPLANE_CONTEXT}

Compare the output with the SchedulePolicy .spec.clusterSelector.

Resolution​

Edit the clusterSelector so it matches one or more workload clusters.

For more details, see Policy-based Scheduling and Labels and Annotations.

Verify available capacity​

A SchedulePolicy may match one or more workload clusters, but those workload clusters may not have enough free capacity to run the workload.

If you are using group scheduling, check whether the resource was added to a ScheduleGroup:

kubectl get events --context=${NOVA_CONTROLPLANE_CONTEXT} --namespace=<resource-namespace> --field-selector=involvedObject.name=<resource-name>

Your resource should have an event similar to:

added to ScheduleGroup <schedule-group-name> which contains objects with groupBy.labelKey <foo>=<bar>

Then, inspect the ScheduleGroup:

kubectl describe schedulegroup <schedule-group-name> --context=${NOVA_CONTROLPLANE_CONTEXT}

In the Events section, you may see a line similar to:

Normal  ScheduleGroupSyncedToWorkloadCluster  8s    nova-scheduler  Multiple clusters matching policy <policy-name> (empty cluster selector): <cluster-names>; group policy <schedule-group-name> does not fit in any cluster;

Resolution​

If the workload or ScheduleGroup does not fit in any matching workload cluster:

  • Add capacity to an eligible workload cluster.
  • Update the SchedulePolicy so more workload clusters are eligible.
  • Review resource requests on the workloads in the group.
  • If applicable, use just-in-time workload clusters or a supported autoscaler to provide additional capacity.

Verify namespace exists in the workload cluster​

A SchedulePolicy may match one or more workload clusters, and there may be enough free capacity, but the workload cannot be created because the namespace does not exist in the target workload cluster.

Resolution​

Either create the namespace manually in the workload cluster:

kubectl create namespace <your-namespace> --context=${K8S_CLUSTER_CONTEXT}

or schedule the Namespace object using Nova.

Remember that a Namespace is treated like any other resource. It must have labels that match the desired SchedulePolicy resourceSelector.

Verify Nova agent health​

A SchedulePolicy may match one or more workload clusters, and there may be enough free capacity, but resources may still fail to appear if the Nova agent in the target workload cluster is having issues.

Verify​

Collect logs from the Nova agent in the workload cluster:

kubectl logs --context=${K8S_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/nova-agent

Resolution​

Review the agent logs for registration, connectivity, or resource application errors.

If the issue is not clear, contact the Elotl team at info@elotl.co.

Placement​

Workloads are automatically rescheduled​

Symptoms​

Nova supports automatic rescheduling in specific cases.

Automatic rescheduling can occur when:

  • Pods scheduled through Nova are Pending in the workload cluster because there is insufficient free capacity.
  • A Deployment scheduled through Nova does not have minimum availability.
  • A ScheduleGroup contains workloads that can no longer all fit on the current workload cluster.

Pending Pods​

Pods scheduled directly through Nova may be Pending in the workload cluster because there is insufficient free capacity.

note

This state does not occur if you scheduled a Deployment or another pod controller through Nova.

Deployment minimum availability​

A Deployment scheduled through Nova may have the following condition:

conditions:
  - lastTransitionTime: <does not matter>
    lastUpdateTime: <does not matter>
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available

ScheduleGroup rescheduling​

If you define a SchedulePolicy with groupBy settings, Nova schedules the entire ScheduleGroup as a unit.

If one of the deployments in the group has either of the preceding conditions, Nova reschedules the whole ScheduleGroup.

In this case, Nova sends a Kubernetes Event for the ScheduleGroup:

kubectl describe schedulegroup <my-group-name> --context=${NOVA_CONTROLPLANE_CONTEXT}

Example output:

Events:
Type     Reason                                Age   From            Message
----     ------                                ----  ----            -------
Warning  ReschedulingTriggered                 3s    nova-agent      deployment default/nginx-group-5 does not have minimum replicas available

Workloads are not moving when expected​

Symptoms​

A workload remains on its current workload cluster even though placement requirements appear to have changed.

Verify​

Check whether one of the following has changed:

  • The workload annotation target.
  • The matching SchedulePolicy.
  • The eligible workload cluster set.
  • The resource requests for the workload or group.
  • The available capacity in the current workload cluster.

If the workload is part of a group, inspect the ScheduleGroup:

kubectl describe schedulegroup <schedule-group-name> --context=${NOVA_CONTROLPLANE_CONTEXT}

Resolution​

Ensure that the workload still matches the intended SchedulePolicy, that the target workload cluster set is eligible, and that another eligible workload cluster has sufficient capacity.

Just-in-Time workload clusters are not created​

Symptoms​

Nova does not create or resume a workload cluster when additional capacity is expected.

Verify​

Check the following:

  • Just-in-time workload cluster behavior is enabled.
  • A suitable source workload cluster exists and is in a supported Kubernetes environment.
  • The source workload cluster is one of the currently supported cluster types for just-in-time cluster creation: EKS, GKE, AKS, or Kind.
  • The configured limit for created workload clusters has not been reached.
  • Existing accessible workload clusters do not have a Nova-recognized autoscaler that can provision the required capacity.
  • The workload's SchedulePolicy allows the created or resumed workload cluster to be eligible.

If Elotl Luna or Kubernetes Cluster Autoscaler is available, Nova prefers to use autoscaling on existing workload clusters before creating a new workload cluster.

Resolution​

Review the just-in-time workload cluster configuration and verify that an eligible source workload cluster can be cloned or resumed.

If no supported source workload cluster exists, add or register a supported cluster type before expecting Nova to create just-in-time workload clusters. Currently supported cluster types for just-in-time creation are EKS, GKE, AKS, and Kind.

Advanced Features​

GPU workloads are not scheduled​

Symptoms​

A GPU workload is created in the Nova Control Plane but is not placed on a workload cluster with GPU capacity.

Verify​

Check the following:

  • The workload cluster has GPU-enabled nodes.
  • The appropriate GPU drivers and operators are installed, such as the NVIDIA GPU Operator where applicable.
  • GPU resources are advertised by the workload cluster.
  • The workload uses the expected GPU resource requests.
  • If using NVIDIA MIG or Kubernetes Dynamic Resource Allocation (DRA), verify that the resource requests and device resources are exposed correctly to Kubernetes.
  • Any nodeSelector, node affinity, taints/tolerations, or other placement constraints are valid for the eligible workload clusters.

You can view available GPU resources through the Nova cluster inventory:

kubectl get clusters -o wide --context=${NOVA_CONTROLPLANE_CONTEXT}

Resolution​

Ensure the GPU resources are visible to Kubernetes and that the workload's resource requests match the resources exposed by the workload cluster.

Disaster recovery workflows are not running​

Symptoms​

A disaster recovery workflow does not run after an expected failure event or alert.

Verify​

Check the following:

  • Alerts are reaching Nova.
  • A ReceivedAlert is created.
  • A RecoveryPlan matches the incoming alert labels.
  • A RecoveryRun is created.
  • The RecoveryRun status reflects the expected execution state.

Resolution​

Review the ReceivedAlert, RecoveryPlan, and RecoveryRun resources to identify where the workflow stopped.

Collecting Diagnostics​

When contacting Elotl support, include as much diagnostic information as possible.

Nova status​

kubectl nova status --context=${NOVA_CONTROLPLANE_CONTEXT} --hosting-cluster-context=${K8S_HOSTING_CLUSTER_CONTEXT} --hosting-cluster-nova-namespace=${NOVA_NAMESPACE}

Nova Control Plane logs​

Collect logs from the hosting cluster:

kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/nova-scheduler
kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/apiserver
kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/kube-controller-manager
kubectl logs --context=${K8S_HOSTING_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} statefulset/etcd

Nova agent logs​

Collect logs from the affected workload cluster:

kubectl logs --context=${K8S_CLUSTER_CONTEXT} --namespace=${NOVA_NAMESPACE} deploy/nova-agent

Scheduling events​

Collect events for the affected resource:

kubectl get events --context=${NOVA_CONTROLPLANE_CONTEXT} --namespace=<resource-namespace> --field-selector=involvedObject.name=<resource-name>

ScheduleGroup details​

If group scheduling is involved, collect the ScheduleGroup details:

kubectl describe schedulegroup <schedule-group-name> --context=${NOVA_CONTROLPLANE_CONTEXT}