Amazon EKS

This section details AWS specific configuration options.

Custom AMIs

NOTE: Non-bottlerocket (aws.isBottlerocketImage=false) custom AMIs must include the EKS nodes’ bootstrap script at /etc/eks/bootstrap.sh and bottlerocket (aws.isBottlerocketImage=true) custom AMIs must include aws.userData set to EKS cluster's TOML-formatted cluster information. Otherwise, nodes will not join the cluster.

You can tell Luna to use a specific AMI via the Helm values:

1. aws.amiIdGeneric for x86-64 nodes
2. aws.amiIdGenericArm for ARM64 nodes
3. aws.amiIdGpu for x86-64 nodes with GPU
4. aws.amiIdGpuArm for ARM64 nodes with GPU

Each of these configuration options accept an AMI ID. If the AMI doesn’t exist or is not accessible Luna will log an error and fall back to the latest generic EKS images.

Set these custom AMI IDs via helm values like this:

--set aws.amiIdGeneric=ami-1234567890
--set aws.amiIdGenericArm=ami-1234567890
--set aws.amiIdGpu=ami-1234567890
--set aws.amiIdGpuArm=ami-1234567890

Custom AMIs with SSM

Amazon offers various EKS image families like Amazon Linux, Ubuntu, and BottleRocket. Luna can use AWS SSM to fetch the most up to date image from its store.

For Amazon Linux, you can get the latest EKS image for Kubernetes 1.27 on arm64 nodes at /aws/service/eks/optimized-ami/1.27/amazon-linux-2-arm64/recommended/image_id.

To configure a SSM query for each image types use imageSsmQueryGeneric and imageSsmQueryGenericArm for regular nodes; imageSsmQueryGpu and imageSsmQueryGpuArm for nodes with GPU. All these parameters may include exactly one "%s" marker to replace with the Kubernetes version.

For example here’s how to use BottleRocket images:

--set aws.imageSsmQueryGeneric="/aws/service/bottlerocket/aws-k8s-%s/x86_64/latest/image_id"
--set aws.imageSsmQueryGenericArm="/aws/service/bottlerocket/aws-k8s-%s/arm64/latest/image_id"
--set aws.imageSsmQueryGpu="/aws/service/bottlerocket/aws-k8s-%s-nvidia/x86_64/latest/image_id"
--set aws.imageSsmQueryGpuArm="/aws/service/bottlerocket/aws-k8s-%s-nvidia/arm64/latest/image_id"

To use Ubuntu:

--set aws.imageSsmQueryGeneric="/aws/service/canonical/ubuntu/eks/20.04/%s/stable/current/amd64/hvm/ebs-gp2/ami-id"
--set aws.imageSsmQueryGenericArm="/aws/service/canonical/ubuntu/eks/20.04/%s/stable/current/arm64/hvm/ebs-gp2/ami-id"

Block device mappings

To customize disk settings for your EKS nodes, use the aws.blockDeviceMappings option. Configure it with JSON with a format like this:

[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 42,
      "VolumeType": "gp2",
      "Encrypted": false
    }
  }
]

Use Helm’s --set-string, --set-json or --set-file options to set aws.blockDeviceMappings and avoid --set since it mangles its input.

For example:

$ cat block_device_mapping.json
[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 42,
      "VolumeType": "gp2",
      "Encrypted": false
    }
  }
]
$ helm ... --set-file aws.blockDeviceMappings=block_device_mapping.json

Bin Packing Zone Spread

When aws.binPackingZoneSpread is true (default false), Luna supports placement of bin packing pods that specify zone spread. To support bin packing zone spread, Luna keeps at least one bin packing node running in each zone associated with the EKS cluster as long as there are any Luna bin packing pods running.

User data

The aws.userData field allows you to define custom setup information for EC2 nodes. This field always expects a string value—either a shell script for non-Bottlerocket images or TOML-formatted configuration for Bottlerocket images. You can provide this string directly using --set-string or load it from a file using --set-file.

For EKS images, aws.userData is not supported because Amazon Linux 2023 has introduced a new boot process. User data is now a configuration file automatically generated by Luna, and it is no longer possible to provide a custom script for node initialization.

When using Bottlerocket images (aws.isBottlerocketImage=true), the aws.userData field must contain EKS cluster configuration information in TOML format. You can generate this configuration file (user-data.toml) as described in the Bottlerocket Quickstart for EKS using the following command:

eksctl get cluster --region your-cluster-region --name your-cluster-name -o json \
  | jq --raw-output '.[] | "settings.kubernetes.api-server = \"" + .Endpoint + "\"\nsettings.kubernetes.cluster-certificate =\"" + .CertificateAuthority.Data + "\"\n"' > user-data.toml

You can then load the generated configuration using the Helm --set-file option:

./deploy.sh ... --additional-helm-values "--set-file aws.userData=user-data.toml"

If you set aws.userDataType to Template, you assume full control of node initialization using Go templates. You are entirely responsible for ensuring the node is properly configured to join your cluster.

User data type

The aws.userDataType setting controls how node initialization scripts (user data) are generated for your Kubernetes nodes. This affects how nodes join your cluster and what configuration they receive.

Valid values are: empty, EKS, BottleRocket, or Template.

If unspecified, the system defaults to EKS for standard images or BottleRocket when isBottlerocketImage is set to true.

If aws.userDataType is set to Template, it lets you define the node’s user data via a Go template. The template receives these variables:

• Labels: the node’s labels. It is a map of strings. It is essential that this configuration option be correctly populated, otherwise nodes won’t accept the pods and won’t be scaled down.
• MaxPods: the maximum number of pods allowed on the node. It should be passed to the kubelet via the --max-pods option, or an equivalent configuration option.
• Taints: the node’s taints. It is a list of strings.
• ClusterName: a string with the name of the EKS cluster.

This option is useful for alternative node types, such as running RKE2 nodes in EC2.

Subnets

aws.subnets is a list of subnet IDs. If empty, Luna will use all the subnets attached to the cluster.

Security groups

Each EC2 node provisioned by Luna must be associated with at least one security group to enable communication with the EKS cluster.

You can explicitly specify which security groups to attach to Luna nodes by setting the aws.securityGroups field:

aws:
  securityGroups: ["sg-0123456789"]

If the aws.securityGroups field is left empty, Luna will automatically discover security groups by querying based on tags.

By default, Luna searches for security groups using the tag elotl.co/nodeless-cluster/name: <CLUSTER_NAME>.

To customize the tag used for this query, configure the aws.securityGroupSelector field. For example, to search by the tag customKey: customValue:

aws:
  securityGroupSelector:
    tagKey: customKey
    tagValue: customValue

If Luna cannot determine which security group to use, the Luna manager will return an error.

IMDS Metadata

metaData defines the instance metadata for EKS nodes, it’s a JSON document conforming to this specification.

Example:

{
    "HttpEndpoint": "enabled",
    "HttpProtocolIpv6": "disabled",
    "HttpPutResponseHopLimit": 42,
    "HttpTokens": "required",
    "InstanceMetadataTagsState": "enabled"
}

Default: Empty.

Use --set-string or --set-file with Helm to set the instance metadata, --set will mangle in the input.