Skip to main content
Version: v1.4

Amazon EKS

This section details AWS-specific configuration options.

Spot​

Use of Spot Instance Advisor on AWS EKS​

The AWS Spot Instance Advisor provides data on spot savings over on-demand prices and spot interruption frequency, averaged over the last 30 days per AWS region and instance type. The data is maintained by AWS in this S3 bucket.

If aws.useSpotAdvisor is set true, Luna fetches the AWS spot-advisor-data hourly and estimates spot price using the 30-day average savings, rather than the static ratio computed using the Luna option spotPriceRatioEstimate. When aws.useSpotAdvisor=true is specified, the options aws.maxSpotPriceRatio and aws.maxSpotInterruptBucket can be used to filter spot instance candidates.

The option aws.maxSpotPriceRatio indicates the highest spot price ratio that Luna should consider. For example, if aws.maxSpotPriceRatio is set to 0.6, Luna would consider a spot instance whose price is up to 60% of its on-demand price, but not one whose price is more than 60% of its on-demand price. The default value 1.0 means Luna considers spot even when it has no discount over the on-demand price, essentially disabling this filter.

The AWS spot instance advisor provides spot interruption frequency in 5 buckets 0-4 labeled as follows: 0:"<5%", 1:"5-10%", 2:"10-15%", 3:"15-20%", 4:">20%". The option aws.maxSpotInterruptBucket indicates the highest interrupt bucket that Luna should consider. For example, setting the value aws.maxSpotInterruptBucket to 2 means Luna would consider a spot instance whose interrupt frequency bucket is 0, 1, or 2, but not one whose interrupt frequency bucket is 3 or 4. The default value 4 means Luna considers a spot instance even in the highest interrupt frequency bucket, essentially disabling this filter.

Spot interruption message option on AWS EKS​

The user can set up an AWS SQS queue to receive Spot interruption messages, delivered two minutes before termination, and can provide that queue name to Luna via the AWS option spotSqsQueueName. When Luna receives a Spot termination message, it marks the node with node.elotl.co/spot-event: termination. Nodes with this annotation will be scaled down by Luna.

Custom AMIs​

NOTE: Non-bottlerocket (aws.isBottlerocketImage=false) custom AMIs must include the EKS nodes’ bootstrap script at /etc/eks/bootstrap.sh and bottlerocket (aws.isBottlerocketImage=true) custom AMIs must include aws.userData set to EKS cluster's TOML-formatted cluster information. Otherwise, nodes will not join the cluster.

You can tell Luna to use a specific AMI via the Helm values:

  1. aws.amiIdGeneric for x86-64 nodes
  2. aws.amiIdGenericArm for ARM64 nodes
  3. aws.amiIdGpu for x86-64 nodes with GPU
  4. aws.amiIdGpuArm for ARM64 nodes with GPU

Each of these configuration options accept an AMI ID. If the AMI doesn’t exist or is not accessible Luna will log an error and fall back to the latest generic EKS images.

Set these custom AMI IDs via helm values like this:

--set aws.amiIdGeneric=ami-1234567890
--set aws.amiIdGenericArm=ami-1234567890
--set aws.amiIdGpu=ami-1234567890
--set aws.amiIdGpuArm=ami-1234567890

Custom AMIs with SSM​

Amazon offers various EKS image families like Amazon Linux, Ubuntu, and BottleRocket. Luna can use AWS SSM to fetch the most up to date image from its store.

For Amazon Linux, you can get the latest EKS image for Kubernetes 1.27 on arm64 nodes at /aws/service/eks/optimized-ami/1.27/amazon-linux-2-arm64/recommended/image_id.

To configure a SSM query for each image types use imageSsmQueryGeneric and imageSsmQueryGenericArm for regular nodes; imageSsmQueryGpu and imageSsmQueryGpuArm for nodes with GPU. All these parameters may include exactly one "%s" marker to replace with the Kubernetes version.

For example here’s how to use BottleRocket images:

--set aws.imageSsmQueryGeneric="/aws/service/bottlerocket/aws-k8s-%s/x86_64/latest/image_id"
--set aws.imageSsmQueryGenericArm="/aws/service/bottlerocket/aws-k8s-%s/arm64/latest/image_id"
--set aws.imageSsmQueryGpu="/aws/service/bottlerocket/aws-k8s-%s-nvidia/x86_64/latest/image_id"
--set aws.imageSsmQueryGpuArm="/aws/service/bottlerocket/aws-k8s-%s-nvidia/arm64/latest/image_id"

To use Ubuntu:

--set aws.imageSsmQueryGeneric="/aws/service/canonical/ubuntu/eks/20.04/%s/stable/current/amd64/hvm/ebs-gp2/ami-id"
--set aws.imageSsmQueryGenericArm="/aws/service/canonical/ubuntu/eks/20.04/%s/stable/current/arm64/hvm/ebs-gp2/ami-id"

Block device mappings​

To customize disk settings for your EKS nodes, use the aws.blockDeviceMappings option. Configure it with JSON with a format like this:

[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 42,
      "VolumeType": "gp2",
      "Encrypted": false
    }
  }
]

Use Helm’s --set-string, --set-json or --set-file options to set aws.blockDeviceMappings and avoid --set since it mangles its input.

For example:

$ cat block_device_mapping.json
[
  {
    "DeviceName": "/dev/xvda",
    "Ebs": {
      "DeleteOnTermination": true,
      "VolumeSize": 42,
      "VolumeType": "gp2",
      "Encrypted": false
    }
  }
]
$ helm ... --set-file aws.blockDeviceMappings=block_device_mapping.json

Bin Packing Zone Spread​

When aws.binPackingZoneSpread is true (default false), Luna supports placement of bin packing pods that specify zone spread. To support bin packing zone spread, Luna keeps at least one bin packing node running in each zone associated with the EKS cluster as long as there are any Luna bin packing pods running.

User data​

The aws.userData field allows you to define custom setup information for EC2 nodes. This field always expects a string value—either a shell script for non-Bottlerocket images or TOML-formatted configuration for Bottlerocket images. You can provide this string directly using --set-string or load it from a file using --set-file.

For EKS images, aws.userData is not supported because Amazon Linux 2023 has introduced a new boot process. User data is now a configuration file automatically generated by Luna, and it is no longer possible to provide a custom script for node initialization.

When using Bottlerocket images (aws.isBottlerocketImage=true), the aws.userData field must contain EKS cluster configuration information in TOML format. You can generate this configuration file (user-data.toml) as described in the Bottlerocket Quickstart for EKS using the following command:

eksctl get cluster --region your-cluster-region --name your-cluster-name -o json \
  | jq --raw-output '.[] | "settings.kubernetes.api-server = \"" + .Endpoint + "\"\nsettings.kubernetes.cluster-certificate =\"" + .CertificateAuthority.Data + "\"\n"' > user-data.toml

You can then load the generated configuration using the Helm --set-file option:

./deploy.sh ... --additional-helm-values "--set-file aws.userData=user-data.toml"

If you set aws.userDataType to Template, you assume full control of node initialization using Go templates. You are entirely responsible for ensuring the node is properly configured to join your cluster.

User data type​

The aws.userDataType setting controls how node initialization scripts (user data) are generated for your Kubernetes nodes. This affects how nodes join your cluster and what configuration they receive.

Valid values are: empty, EKS, BottleRocket, or Template.

If unspecified, the system defaults to EKS for standard images or BottleRocket when isBottlerocketImage is set to true.

If aws.userDataType is set to Template, it lets you define the node’s user data via a Go template. The template receives these variables:

  • Labels: the node’s labels. It is a map of strings. It is essential that this configuration option be correctly populated, otherwise nodes won’t accept the pods and won’t be scaled down.
  • MaxPods: the maximum number of pods allowed on the node. It should be passed to the kubelet via the --max-pods option, or an equivalent configuration option.
  • Taints: the node’s taints. It is a list of strings.
  • ClusterName: a string with the name of the EKS cluster.

This option is useful for alternative node types, such as running RKE2 nodes in EC2.

Subnets​

aws.subnets is a list of subnet IDs. If empty, Luna will use all the subnets attached to the cluster.

Security groups​

Each EC2 node provisioned by Luna must be associated with at least one security group to enable communication with the EKS cluster.

You can explicitly specify which security groups to attach to Luna nodes by setting the aws.securityGroups field:

aws:
  securityGroups: ["sg-0123456789"]

If the aws.securityGroups field is left empty, Luna will automatically discover security groups by querying based on tags.

By default, Luna searches for security groups using the tag elotl.co/nodeless-cluster/name: <CLUSTER_NAME>.

To customize the tag used for this query, configure the aws.securityGroupSelector field. For example, to search by the tag customKey: customValue:

aws:
  securityGroupSelector:
    tagKey: customKey
    tagValue: customValue

If Luna cannot determine which security group to use, the Luna manager will return an error.

IMDS Metadata​

metaData defines the instance metadata for EKS nodes, it’s a JSON document conforming to this specification.

Example:

{
    "HttpEndpoint": "enabled",
    "HttpProtocolIpv6": "disabled",
    "HttpPutResponseHopLimit": 42,
    "HttpTokens": "required",
    "InstanceMetadataTagsState": "enabled"
}

Default: Empty.

Use --set-string or --set-file with Helm to set the instance metadata, --set will mangle in the input.