Skip to main content
Version: v1.5

Install Kubernetes Operators with Nova

Overview​

This tutorial shows how to install a Kubernetes Operator through the Nova Control Plane.

The Prometheus Operator is used as an example, but the same pattern can be used for other operators whose manifests and custom resources are supported by Nova.

Operator manifests are standard Kubernetes resources. When those resources are applied to the Nova Control Plane and match a SchedulePolicy, Nova schedules them to an eligible workload cluster.

note

The examples in this tutorial are for illustrative purposes. In a production environment, customize the SchedulePolicy, operator manifests, CRDs, RBAC, and custom resources for your own workload clusters and application requirements.

Before You Begin​

Before starting, make sure you have:

  • A running Nova Control Plane.
  • At least one workload cluster registered with Nova.
  • kubectl configured with access to the Nova Control Plane.
  • A workload cluster label you can use in a SchedulePolicy.
  • curl and jq available locally if you use the example commands.
  • The add_labels.sh helper script, or another way to add labels to downloaded manifests.

This tutorial uses the following contexts:

  • ${NOVA_CONTROLPLANE_CONTEXT} for the Nova Control Plane.
  • ${K8S_CLUSTER_CONTEXT_1} for the workload cluster where the operator is expected to run.

How It Works​

In this workflow:

  1. Create a SchedulePolicy in the Nova Control Plane.
  2. Label the operator manifests so they match the SchedulePolicy.
  3. Apply the operator manifests to the Nova Control Plane.
  4. Nova places the operator resources on an eligible workload cluster.
  5. Apply any required RBAC and custom resources through the Nova Control Plane.
  6. Verify the operator and custom resources in the selected workload cluster.

Many operators require CRDs to exist before their custom resources can be created. Make sure the operator CRDs are installed through the Nova Control Plane before applying custom resources that depend on them.

note

Operators often provide cluster-level functionality. If an operator needs to reconcile resources across multiple workload clusters, configure the SchedulePolicy to spread or duplicate the operator resources across those workload clusters.

This is especially important when the operator’s custom resources may be scheduled to different workload clusters. The operator should be running in each workload cluster where it is expected to manage those resources.

Step 1: Define a SchedulePolicy​

Create a SchedulePolicy that selects the operator resources and places them on an eligible workload cluster.

Apply the following SchedulePolicy to the Nova Control Plane:

kubectl apply --context=${NOVA_CONTROLPLANE_CONTEXT} -f - <<'EOF'
apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: prometheus-operator-policy
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  clusterSelector:
    matchLabels:
      kubernetes.io/metadata.name: wlc-1
  resourceSelectors:
    labelSelectors:
    - matchLabels:
        nova.elotl.co/policy: prometheus-operator-policy
EOF

Step 2: Prepare the operator manifests​

This tutorial uses the Prometheus Operator bundle from the official Prometheus Operator project.

The bundle must be labeled so the resources match the SchedulePolicy resourceSelector.

This tutorial uses a helper script to add labels to the upstream Prometheus Operator bundle. You can also add the labels manually or use another manifest processing tool, such as Kustomize.

Download the helper script and make it executable:

curl -sL https://raw.githubusercontent.com/elotl/try-nova/refs/heads/main/add_labels.sh -o add_labels.sh
chmod +x add_labels.sh

Step 3: Install the Prometheus Operator through the Nova Control Plane​

Download the latest Prometheus Operator bundle, add the Nova policy label, and apply it to the Nova Control Plane:

LATEST=$(curl -s https://api.github.com/repos/prometheus-operator/prometheus-operator/releases/latest | jq -cr .tag_name)

curl -sL https://github.com/prometheus-operator/prometheus-operator/releases/download/${LATEST}/bundle.yaml | \
  ./add_labels.sh -l nova.elotl.co/policy=prometheus-operator-policy | \
  kubectl apply --context=${NOVA_CONTROLPLANE_CONTEXT} -f -

Step 4: Verify the operator is running​

First, verify that the operator resources exist in the Nova Control Plane:

kubectl get deploy --context=${NOVA_CONTROLPLANE_CONTEXT}

Then verify that the operator resources were placed on the expected workload cluster:

kubectl get deploy --context=${K8S_CLUSTER_CONTEXT_1}
kubectl get pods --context=${K8S_CLUSTER_CONTEXT_1}

Wait for the Prometheus Operator pods to become ready.

Step 5: Create RBAC for Prometheus​

Next, create the RBAC resources required by the Prometheus custom resource.

Apply the RBAC resources to the Nova Control Plane:

kubectl apply --context=${NOVA_CONTROLPLANE_CONTEXT} -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default
EOF

Step 6: Deploy Prometheus​

After the operator and RBAC resources are available, create a Prometheus custom resource.

Apply the following Prometheus custom resource to the Nova Control Plane:

kubectl apply --context=${NOVA_CONTROLPLANE_CONTEXT} -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
EOF

Step 7: Verify Prometheus​

Verify that the Prometheus custom resource exists in the Nova Control Plane:

kubectl get prometheus --context=${NOVA_CONTROLPLANE_CONTEXT}

Then verify that the Prometheus resources are running in the selected workload cluster:

kubectl get pods --context=${K8S_CLUSTER_CONTEXT_1}

The Prometheus Operator and Prometheus custom resource should now be running in the selected workload cluster.

Cleanup​

To remove the example resources, delete them from the Nova Control Plane:

kubectl delete prometheus prometheus                     --context=${NOVA_CONTROLPLANE_CONTEXT}
kubectl delete clusterrolebinding prometheus             --context=${NOVA_CONTROLPLANE_CONTEXT}
kubectl delete clusterrole prometheus                    --context=${NOVA_CONTROLPLANE_CONTEXT}
kubectl delete serviceaccount prometheus                 --context=${NOVA_CONTROLPLANE_CONTEXT}
kubectl delete schedulepolicy prometheus-operator-policy --context=${NOVA_CONTROLPLANE_CONTEXT}

If you want to remove the Prometheus Operator resources, delete the same labeled bundle resources from the Nova Control Plane using the manifests or tooling you used to install them.