Amazon EKS

This section details AWS specific configuration options.

Custom AMIs

NOTE: Non-bottlerocket (aws.isBottlerocketImage=false) custom AMIs must include the EKS nodes’ bootstrap script at /etc/eks/bootstrap.sh and bottlerocket (aws.isBottlerocketImage=true) custom AMIs must include aws.userData set to EKS cluster's TOML-formatted cluster information. Otherwise, nodes will not join the cluster.

You can tell Luna to use a specific AMI via the Helm values:

1. aws.amiIdGeneric for x86-64 nodes
2. aws.amiIdGenericArm for Arm64 nodes
3. aws.amiIdGpu for x86-64 nodes with GPU

Each of these configuration options accept an AMI ID. If the AMI doesn’t exist or is not accessible Luna will log an error and fall back to the latest generic EKS images.

Set these custom AMI IDs via helm values like this:

--set aws.amiIdGeneric=ami-1234567890
--set aws.amiIdGenericArm=ami-1234567890
--set aws.amiIdGpu=ami-1234567890

Custom AMIs with SSM

Amazon offers various EKS image families like Amazon Linux, Ubuntu, and BottleRocket. Luna can use AWS SSM to fetch the most up to date image from its store.

For Amazon Linux, you can get the latest EKS image for Kubernetes 1.27 on arm64 nodes at /aws/service/eks/optimized-ami/1.27/amazon-linux-2-arm64/recommended/image_id.

To configure a SSM query for each image types use imageSsmQueryGeneric, imageSsmQueryGenericArm, and imageSsmQueryGpu. All these parameters may include exactly one "%s" marker to replace with the Kubernetes version.

For example here’s how to use BottleRocket images:

--set aws.imageSsmQueryGeneric="/aws/service/bottlerocket/aws-k8s-%s/x86_64/latest/image_id"
--set aws.imageSsmQueryGenericArm="/aws/service/bottlerocket/aws-k8s-%s/arm64/latest/image_id"
--set aws.imageSsmQueryGpu="/aws/service/bottlerocket/aws-k8s-%s-nvidia/x86_64/latest/image_id"

To use Ubuntu:

--set aws.imageSsmQueryGeneric="/aws/service/canonical/ubuntu/eks/20.04/%s/stable/current/amd64/hvm/ebs-gp2/ami-id"
--set aws.imageSsmQueryGenericArm="/aws/service/canonical/ubuntu/eks/20.04/%s/stable/current/arm64/hvm/ebs-gp2/ami-id"

Block device mappings

To customize disk settings for your EKS nodes, use the aws.blockDeviceMappings option. Configure it with JSON with a format like this:

[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 42,
      "VolumeType": "gp2",
      "Encrypted": false
    }
  }
]

Use Helm’s --set-string, --set-json or --set-file options to set aws.blockDeviceMappings and avoid --set since it mangles its input.

For example:

$ cat block_device_mapping.json
[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 42,
      "VolumeType": "gp2",
      "Encrypted": false
    }
  }
]
$ helm ... --set-file aws.blockDeviceMappings=block_device_mapping.json

Bin Packing Zone Spread

When aws.binPackingZoneSpread is true (default false), Luna supports placement of bin packing pods that specify zone spread. To support bin packing zone spread, Luna keeps at least one bin packing node running in each zone associated with the EKS cluster as long as there are any Luna bin packing pods running.

User data

aws.userData allows you to define node setup information. This option is empty by default and always expects a string value, whether it's a script for non-Bottlerocket images or TOML-formatted configuration for Bottlerocket images. You can provide this string directly using --set-string or load it from a file using --set-file.

For non-bottlerocket images (aws.isBottlerocketImage=false), aws.userData specifies a script to be executed after nodes have been bootstrapped. For example specifying --set-string aws.userData="echo hello > /tmp/hello" will create a file named /tmp/hello with hello in it on the node once the EKS bootstrap script has completed. If you have a large script, we recommend you use the --set-file Helm option to load it:

$ cat myscript.sh
apt-get install my-package
$ ./deploy.sh ... --additional-helm-values "--set-file aws.userData=myscript.sh"

For bottlerocket images (aws.isBottlerocketImage=true), aws.userData specifies required EKS cluster information in TOML-format. This information can be generated into the file user-data.toml as described in https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART-EKS.md#kubernetes-cluster-info via:

eksctl get cluster --region your-cluster-region --name your-cluster-name -o json \
   | jq --raw-output '.[] | "settings.kubernetes.api-server = \"" + .Endpoint + "\"\nsettings.kubernetes.cluster-certificate =\"" + .CertificateAuthority.Data + "\"\n"' > user-data.toml

As with the non-bottlerocket case, you can use the --set-file Helm option to load it:

$ ./deploy.sh ... --additional-helm-values "--set-file aws.userData=user-data.toml"

Note that when you set aws.userDataType to Template, you take full control of node initialization through Go templates. This option is powerful but requires caution: you become responsible for properly configuring the node to join your cluster.

User data type

The aws.userDataType setting controls how node initialization scripts (user data) are generated for your Kubernetes nodes. This affects how nodes join your cluster and what configuration they receive.

Valid values are: empty, EKS, BottleRocket, or Template.

If unspecified, the system defaults to EKS for standard images or BottleRocket when isBottlerocketImage is set to true.

If aws.userDataType is set to Template, it lets you define the node’s user data via a Go template. The template receives these variables:

• Labels: the node’s labels. It is a map of strings. It is essential that this configuration option be correctly populated, otherwise nodes won’t accept the pods and won’t be scaled down.
• MaxPods: the maximum number of pods allowed on the node. It should be passed to the kubelet via the --max-pods option, or an equivalent configuration option.
• Taints: the node’s taints. It is a list of strings.
• ClusterName: a string with the name of the EKS cluster.

This option is useful for alternative node types, such as running RKE2 nodes in EC2.

Subnets

aws.subnets is a list of subnet IDs. If empty, Luna will use all the subnets attached to the cluster.

Security groups

Each EC2 node provisioned by Luna requires at least one security group to enable communication with the EKS cluster.

You can specify the security group IDs to attach to Luna nodes by setting the aws.securityGroups field:

aws:
  securityGroups: ["sg-0123456789"]

If aws.securityGroups is left empty, Luna will use security groups tagged with elotl.co/nodeless-cluster/name: <CLUSTER_NAME>.

If Luna is unable to determine a security group to use, the Luna manager will return an error.

IMDS Metadata

metaData defines the instance metadata for EKS nodes, it’s a JSON document conforming to this specification.

Example:

{
    "HttpEndpoint": "enabled",
    "HttpProtocolIpv6": "disabled",
    "HttpPutResponseHopLimit": 42,
    "HttpTokens": "required",
    "InstanceMetadataTagsState": "enabled"
}

Default: Empty.

Use --set-string or --set-file with Helm to set the instance metadata, --set will mangle in the input.