Microsoft AKS

This section details Azure-specific configuration options.

Spot

Special consideration for using Spot on Azure AKS

AKS nodes with spot pricing have a taint automatically applied to them. This means pods running on Spot nodes in AKS must have a toleration set in order to be scheduled and run on the nodes with Spot pricing. To use GPU spot nodes, the toleration also needs to be included on the Nvidia daemonsets.

In order to get the pods running on the spot nodes the operator must add a toleration corresponding to the kubernetes.azure.com/scalesetpriority=spot:NoSchedule taint.

spec:
  containers:
  - name: spot-example
  tolerations:
  - key: "kubernetes.azure.com/scalesetpriority"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

Pod Subnet for Dynamic Azure CNI Networking

You can indicate the pod subnet to be used by Dynamic Azure CNI networking for the bin packing node via azure.binPackingNodePodSubnet.

For example, if you would like your bin packing instances to use podsubnet1, do this:

--set azure.binPackingAKSPodSubnet=podsubnet1

Opportunistic Ephemeral OS Disks

Luna can utilize ephemeral OS disks on Azure when certain conditions are met. This feature can improve performance and reduce costs.

If the chosen node instance type supports ephemeral OS disk and has a cache size greater than opportunisticEphemeralOSDiskSizeGB:

1. It will use an ephemeral OS disk
2. The ephemeral OS disk type size will be set to opportunisticEphemeralOSDiskSizeGB.

For Azure supplied Linux images the size is around 30 GB; for Windows the size is around 128 GB.

If the node type doesn’t support ephemeral OS disk, or its cache isn’t large enough to accommodate the request, Luna will use the default OS disk type: managed.

To use this option:

--set azure.opportunisticEphemeralOSDiskSizeGB=<size in GB>

Ensure that the specified opportunisticEphemeralOSDiskSizeGB value is greater or equal to the size of the image running on the nodes.

Ultra SSD support

Azure ultra disks offer high throughput, high IOPS, and consistent low latency disk storage for your stateful applications. The option azure.enableUltraSSD allows you to enable ultra disks support for you AKS nodes. Note that there are several limitations with Ultra disks, for example they can’t be used for OS disks.

To enabled Ultra disks, do this:

--set azure.enableUltraSSD=true

Please note that Azure ultra disks require node pools deployed in availability zones and regions that support these disks, and are only supported by specific VM series. Refer to Ultra disk limitations documentation to learn more.

Kubelet disk type

azure.kubeletDiskType determines the placement of emptyDir volumes, container runtime data root, and Kubelet ephemeral storage. There are two possible values:

1. OS (default): emptyDir volumes, container runtime data root, and Kubelet ephemeral storage is hosted by the OS disk, no matter if this is managed or ephemeral.
2. Temporary: emptyDir volumes, container runtime data root, and Kubelet ephemeral storage is hosted by the temporary storage.

To use temporary storage for containers data, do the following:

--set azure.kubeletDiskType=Temporary

Refer to the Azure documentation to learn more about kubelet disk types.

OS disk type and size

azure.osDiskType and azure.osDiskSizeGB can be used to configure the OS disk for nodes.

There are two types of OS disk types:

1. Ephemeral (default) OS disks are stored only on the host machine, just like a temporary disk. This provides lower read/write latency, along with faster node scaling and cluster upgrades.
2. Managed Azure replicates the operating system disk for a virtual machine to Azure storage to avoid data loss should the VM need to be relocated to another host. Since containers aren't designed to have local state persisted, this behavior offers limited value while providing some drawbacks, including slower node provisioning and higher read/write latency.

OS Disk Size in GB to be used to specify the disk size for every machine in the master/agent pool. If you specify 0, it will apply the default disk size according to the node type selected.

--set azure.osDiskSizeGB=123
--set azure.osDiskType=Managed

Enable Node Public IP

You can indicate that Luna should enable AKS assignment of a public IP to the nodes it allocates via azure.enableNodePublicIP. The option is false by default.

To use this option, do this:

--set azure.enableNodePublicIP=true

Enable Artifact Streaming

Artifact Streaming on AKS is a preview feature that streams container images from ACR premium tier to AKS. The feature can greatly reduce image pull time, which is desirable when Luna is scaling up for workloads like Machine Learning that have large images.

To register the ArtifactStreamingPreview feature in your subscription and to enable Artifact Streaming on your ACR image(s), please follow the instructions here.

Once these prerequistes are done, you can indicate that Luna should enable Artifact Streaming on the nodes it allocates via azure.enableArtifactStreaming. The Luna option is false by default.

To set the option, do this:

--set azure.enableArtifactStreaming=true