Terraform EKS with Karpenter, Graviton, and Spot Instances

Overview

This repository contains Terraform code to deploy an EKS cluster on AWS, utilizing Karpenter for efficient node autoscaling. The cluster supports both x86 (amd64) and Graviton (arm64) architectures. Additionally, the configuration leverages Spot Instances for cost efficiency and includes a demonstration deployment for users to test the setup.

Prerequisites

Before using this Terraform repository, ensure you have the following installed:

• Terraform (latest version)
• AWS CLI (configured with credentials for your AWS account)
• kubectl (for interacting with the EKS cluster)

It needs to be made sure that service-linked role is created for creation of spot instances. It can be created with command.

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

Alternatively, it can be added to terraform as well.

resource "aws_iam_service_linked_role" "spot_instance_role" {
  aws_service_name = "spot.amazonaws.com"
}

Deployment Steps

1. Clone the Repository

git clone <repository-url>
cd <repository-folder>
terraform init
terraform plan
terraform apply --auto-approve

2. Connect to the cluster

aws eks --region eu-west-1 update-kubeconfig --name ex-karpenter
kubectl scale deployment inflate --replicas 5

3. Test creation of nodes

kubectl get nodes -L karpenter.sh/registered
kubectl get pods -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Selecting nodes based on criteria

To ensure your application pods are scheduled on nodes that meet specific requirements (e.g., ARM or x86 architecture, spot or on-demand instances), Kubernetes provides mechanisms such as node affinity, node selectors, and taints and tolerations. Here’s how you can achieve this:

1. Ensuring Scheduling Based on Node Architecture (ARM or x86)

Solution: Use Node Affinity.

You can specify a node affinity rule in the pod’s specification to ensure it only runs on nodes with the desired architecture.

Example Pod Spec:

apiVersion: v1
kind: Pod
metadata:
  name: example-arm-pod
spec:
  containers:
  - name: example
    image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - arm64 # Specify the desired architecture (e.g., arm64 or amd64)

If you want to target x86-based nodes, change the values field to amd64.

2. Ensuring Scheduling Based on Instance Type (Spot or On-Demand)

Solution: Use Node Affinity and Karpenter Capacity Type Tags.

Karpenter uses the label karpenter.sh/capacity-type to indicate the instance type (e.g., spot or on-demand). You can use this label in a pod’s affinity rule.

Example Pod Spec for Spot Instance:

apiVersion: v1
kind: Pod
metadata:
  name: example-spot-pod
spec:
  containers:
  - name: example
    image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: karpenter.sh/capacity-type
            operator: In
            values:
            - spot # Ensure the pod runs only on spot instances

For on-demand instances, change the values field to on-demand.

3. Combining Both Requirements (ARM/x86 and Spot/On-Demand)

You can combine both node affinity rules into a single pod spec to ensure the pod is scheduled on nodes meeting both criteria.

Example Pod Spec:

apiVersion: v1
kind: Pod
metadata:
  name: example-specific-pod
spec:
  containers:
  - name: example
    image: nginx
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values:
            - arm64 # Specify architecture
          - key: karpenter.sh/capacity-type
            operator: In
            values:
            - spot # Specify capacity type

4. Alternative: Using Taints and Tolerations

If you prefer stricter control, you can taint nodes to only accept specific pods. For instance, you can taint all spot instance nodes with spot=true:NoSchedule and then add a toleration in the pod spec to tolerate this taint.

Node Taint Example:

kubectl taint nodes <spot-node-name> spot=true:NoSchedule

Pod Spec with Toleration:

apiVersion: v1
kind: Pod
metadata:
  name: example-tolerant-pod
spec:
  containers:
  - name: example
    image: nginx
  tolerations:
  - key: "spot"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

This ensures only pods with the matching toleration can be scheduled on spot nodes.