Installing Kubernetes Operators

Overview

Welcome to Nova Control Plane's guide on managing Kubernetes Operators! Nova provides a simplified way to manage Kubernetes Operators across different clusters. This guide will walk you through the steps to install and manage Kubernetes Operators using Nova. We'll use the Prometheus Operator as an example, but the principles can be applied to other operators as well. This guide is based on the official Prometheus Operator documentation.

Note: The examples provided are for illustrative purposes. In a real-world setup, make sure to customize the SchedulePolicy, deployment, CRDs, and values according to your specific needs.

Prerequisites

Before proceeding, you'll need to set labels on the resources you're applying. We've created a Python helper script for this purpose. Save it as add_labels.py and make it executable with chmod +x add_labels.py.

Click here to view the script

#!/usr/bin/env python3
"""
add_labels.py

This script adds labels to Kubernetes resource YAML files. It supports three modes:
- "cluster": Adds labels only to cluster-scoped resources.
- "namespace": Adds labels only to namespace-scoped resources.
- "all": Adds labels to all resources.

Usage:
    add_labels.py mode label_key label_value < input.yaml > output.yaml

Dependencies:
    - PyYAML: A YAML parser for Python.

Author: Elotl Inc.
Date: 28.08.2023
"""

import sys
import yaml
import subprocess
import re
import argparse
from typing import List, Set, Union, Dict

CMD_FETCH_CLUSTER_RESOURCES = "kubectl api-resources --namespaced=false | awk '{print $NF}' | tail -n +2"

def preprocess_yaml(yaml_str: str) -> str:
    """Escape special characters in a YAML string."""
    lines = yaml_str.split("\n")
    for i, line in enumerate(lines):
        if re.match(r"^\s*-\s*=\s*$", line):
            lines[i] = line.replace("=", "\"=\"")
        elif re.match(r"^\s*-\s*=~\s*$", line):
            lines[i] = line.replace("=~", "\"=~\"")
    return "\n".join(lines)

def get_cluster_resources() -> Set[str]:
    """Fetch the list of cluster-scoped resources from the Kubernetes cluster."""
    try:
        cluster_resources = subprocess.check_output(CMD_FETCH_CLUSTER_RESOURCES, shell=True, text=True)
        return set(cluster_resources.strip().split("\n"))
    except subprocess.CalledProcessError:
        sys.stderr.write("Error fetching cluster-scoped resources. Make sure kubectl is installed and configured.\n")
        sys.exit(1)

def should_add_label(doc_kind: str, mode: str, cluster_resources: Set[str]) -> bool:
    """Determine if a label should be added based on the mode and kind of the resource."""
    return (
        mode == "all" or
        (mode == "cluster" and doc_kind in cluster_resources) or
        (mode == "namespace" and doc_kind not in cluster_resources)
    )

def add_labels(documents: List[Union[None, Dict]], mode: str, label_key: str, label_value: str) -> List[Union[None, Dict]]:
    """Add labels to Kubernetes YAML documents."""
    cluster_resources = get_cluster_resources() if mode != "all" else set()

    for doc in documents:
        if doc is None or 'kind' not in doc:
            continue
        doc.setdefault('metadata', {}).setdefault('labels', {})

        if should_add_label(doc['kind'], mode, cluster_resources):
            doc['metadata']['labels'][label_key] = label_value

    return documents

def main() -> None:
    """Main function."""
    parser = argparse.ArgumentParser(description='Add labels to Kubernetes YAML files.')
    parser.add_argument('mode', choices=['cluster', 'namespace', 'all'], help='Mode to apply labels. Can be "cluster", "namespace", or "all".')
    parser.add_argument('label_key', help='The key of the label to add.')
    parser.add_argument('label_value', help='The value of the label to add.')

    args = parser.parse_args()

    try:
        yaml_str = preprocess_yaml(sys.stdin.read())
        documents = list(yaml.safe_load_all(yaml_str))
        documents = add_labels(documents, args.mode, args.label_key, args.label_value)

        for doc in documents:
            if doc is not None:
                yaml.safe_dump(doc, sys.stdout, default_flow_style=False)
                print('---')
    except yaml.YAMLError as e:
        sys.stderr.write(f"YAML Error: {e}\n")
        sys.exit(1)
    except Exception as e:
        sys.stderr.write(f"Unexpected Error: {e}\n")
        sys.exit(1)

if __name__ == "__main__":
    main()

Steps to Install Operators

1. Define Your SchedulePolicy

Start by defining a SchedulePolicy tailored to your specific clusters and namespaces. This sets the stage for deploying your Operator.

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: prometheus-operator-policy
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  clusterSelector:
    matchLabels:
      kubernetes.io/metadata.name: kind-workload-1
  resourceSelectors:
    labelSelectors:
    - matchLabels:
        nova.elotl.co/policy: prometheus-operator-policy

Apply the policy:

kubectl create -f example-operator-policy.yaml

2. Install Prometheus Operator and Resources

Use the following one-liner to download, label, and apply all the necessary Prometheus Operator resources. This command uses a helper script, add_labels.sh, to add the appropriate labels for Nova management:

LATEST=$(curl -s https://api.github.com/repos/prometheus-operator/prometheus-operator/releases/latest | jq -cr .tag_name)
curl -sL https://github.com/prometheus-operator/prometheus-operator/releases/download/${LATEST}/bundle.yaml | \
  ./add_labels.py all nova.elotl.co/policy prometheus-operator-policy | \
  kubectl create -f -

3. Verify the Operator is Running

After applying the one-liner, you can verify that the Prometheus Operator is running successfully:

kubectl get deploy

Wait for the Prometheus Operator's to show running pods.

Congratulations! You've successfully installed the Prometheus Operator and its resources in your kind-workload-1 cluster using Nova, and all of this was done with a convenient one-liner.

4. Deploying Prometheus

Now we need to setup RBAC for our Prometheus and deploy Prometheus itself. First, save this as prom-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default

And run:

kubectl create -f prom-rbac.yaml

This will setup RBAC needed by Prometheus deployment.

With that out of the way, we can finally deploy Prometheus itself!

Save this as prometheus.yaml:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    nova.elotl.co/policy: prometheus-operator-policy
  name: prometheus
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false

Run:

kubectl create -f prometheus.yaml

And thats it! You now have a running Prometheus stack inside your kind-workload-1 cluster!

Learn more about our SchedulePolicies and how you can deploy operators across multiple clusters in our Tutorials section.

Reminder: Customize According to Your Needs

As mentioned before, the examples provided here are solely for illustrative purposes. For your specific needs, make sure to adapt the SchedulePolicy, Operator deployment, CRDs, and custom resources.