Amazon EKS in Production Depth

Table of Contents#

1. Why This Part Exists
2. Managed Node Groups — Full Depth
3. Launch Templates and Custom AMIs
4. Karpenter — Modern Node Autoscaling
5. Karpenter Consolidation and Spot
6. EKS Auto Mode
7. EKS Pod Identity — The IRSA Successor
8. Pod Identity vs IRSA — Choosing and Migrating
9. EKS-Managed Add-ons
10. Add-on Update Strategy and Versioning
11. Fargate Profiles In Depth
12. Custom Networking and IP Prefix Delegation
13. Security Groups for Pods
14. API Endpoint Access Control
15. Multi-Tenancy Patterns on EKS
16. GitOps on EKS
17. EKS Observability
18. EKS Cost Optimization
19. EKS Upgrade Strategy In Depth
20. EKS Security Hardening Checklist
21. A Full Worked Example: Migrating from Cluster Autoscaler to Karpenter
22. Part 7 CLI Cheat Sheet
23. Common Mistakes
24. Worked Practice Problems
25. Summary and What's Next

---

Why This Part Exists#

Part 5 introduced EKS as one of three managed Kubernetes offerings — architecture, the EC2-vs-Fargate compute choice, IRSA, and the VPC CNI, each covered at a level that lets you compare EKS fairly against AKS and GKE. That comparison-level treatment is deliberately shallow on any single provider, because its job is breadth across three.

This Part goes the other direction: EKS only, and deep enough to actually run a production cluster on it. Everything here assumes you already know the material from Parts 1-6 — the Kubernetes API model, scheduling, networking, storage, and the basic EKS shape from Part 5 — and builds forward from there into the tools and patterns a real EKS platform team uses every day: Karpenter instead of the older Cluster Autoscaler, Pod Identity as the current recommended successor to IRSA, EKS Auto Mode as AWS's newest fully-managed compute option, and the operational muscle memory around upgrades, multi-tenancy, and cost that separates "we got a cluster running" from "we run this in production."

Diagram

---

Managed Node Groups — Full Depth#

Part 5 introduced EC2 Managed Node Groups at a surface level: "AWS automates the EC2 provisioning and lifecycle." Here's what that actually means operationally.

A Managed Node Group is AWS's abstraction over an underlying Auto Scaling Group (ASG, from the AWS Cloud Architecture series' compute tutorial) — EKS creates and owns that ASG for you, wires it into the cluster's bootstrap process, and exposes a simplified API surface (eksctl create nodegroup, or the equivalent Terraform/CloudFormation resource) instead of requiring you to hand-configure the ASG, launch template, and node bootstrap script yourself.

Diagram

Copy
# Create a managed node group with eksctl
eksctl create nodegroup \
  --cluster my-cluster \
  --name standard-workers \
  --node-type m6i.xlarge \
  --nodes 3 --nodes-min 3 --nodes-max 10 \
  --node-labels "workload-type=general" \
  --asg-access

# List node groups and their status
eksctl get nodegroup --cluster my-cluster

# Trigger a managed node group AMI/version update
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name standard-workers

What "managed" buys you concretely, worth naming explicitly:

Node taints and labels at the node-group level: a genuinely important, frequently-missed capability — you can apply taints and labels to an entire managed node group at creation time, so every node it ever creates (including future scale-out nodes) automatically carries them, without needing a DaemonSet or admission webhook to apply them after the fact.

Copy
eksctl create nodegroup \
  --cluster my-cluster \
  --name gpu-workers \
  --node-type g5.xlarge \
  --node-labels "workload-type=gpu" \
  --node-taints "nvidia.com/gpu=true:NoSchedule"

This directly reuses the taints/tolerations mechanism from Part 2 — the practical EKS-specific point is where you declare the taint (at the node group, not per-node or via a controller), so it survives node replacement automatically.

---

Launch Templates and Custom AMIs#

By default, a Managed Node Group uses an AWS-generated launch template built from the EKS-optimized AMI for your chosen Kubernetes version. Real production clusters frequently need more control than the default gives — custom disk sizing, additional bootstrap arguments, a hardened/custom AMI for compliance, or extra pre-installed agents (log shippers, security agents).

Diagram

Copy
# A custom launch template referencing a hardened custom AMI,
# enforcing IMDSv2 (a real, commonly-tested EKS security hardening step)
aws ec2 create-launch-template \
  --launch-template-name eks-hardened-workers \
  --launch-template-data '{
    "ImageId": "ami-0123456789abcdef0",
    "BlockDeviceMappings": [{"DeviceName": "/dev/xvda", "Ebs": {"VolumeSize": 100, "VolumeType": "gp3"}}],
    "MetadataOptions": {"HttpTokens": "required", "HttpPutResponseHopLimit": 2}
  }'

eksctl create nodegroup \
  --cluster my-cluster \
  --name hardened-workers \
  --launch-template-id lt-0123456789abcdef0

HttpTokens: required is worth calling out specifically — it forces IMDSv2 (session-token-based instance metadata access) instead of the older, credential-leak-prone IMDSv1, directly closing the exact class of SSRF-to-credential-theft vulnerability class covered in the DevSecOps series' secrets management tutorial (05-devsecops/04-secrets-management-and-iam.md). Real production EKS node groups should enforce this by default, not leave it at the account-level default.

Bottlerocket, worth knowing by name: AWS's own purpose-built, minimal, immutable Linux distribution specifically for running containers — no shell, no package manager, an immutable root filesystem, and an API-driven update model. Choosing Bottlerocket over Amazon Linux 2023 as your node AMI trades general-purpose flexibility for a meaningfully smaller attack surface — a genuinely strong, specific answer to "how would you harden your EKS worker nodes" in an interview.

---

Karpenter — Modern Node Autoscaling#

The single most important operational shift in EKS compute since Fargate, and a concept every EKS practitioner needs cold: Karpenter is an open-source (originally AWS-built, now a CNCF project) node autoscaler that replaces the traditional Cluster Autoscaler + fixed node group model with direct, just-in-time EC2 provisioning driven by actual unschedulable pod requirements.

Diagram

Why this is a genuinely different model from the traditional Cluster Autoscaler, worth stating precisely for an interview: the Cluster Autoscaler scales existing, pre-defined node groups up and down within their configured min/max — it has to guess in advance which instance types and sizes you'll need, provisioned into fixed-shape groups. Karpenter instead evaluates each unschedulable pod's actual resource requirements and provisions the right-sized instance for it directly, from a broad, flexible instance-type pool — no need to pre-define a matrix of node groups for every possible workload shape.

Copy
# A Karpenter NodePool - defines WHAT kinds of nodes Karpenter is allowed to launch
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: node.kubernetes.io/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
---
# The EC2NodeClass - defines HOW those nodes are configured (AMI, subnets, security groups)
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  subnetSelectorTerms:
    - tags: {karpenter.sh/discovery: my-cluster}
  securityGroupSelectorTerms:
    - tags: {karpenter.sh/discovery: my-cluster}
  role: KarpenterNodeRole-my-cluster

The NodePool / EC2NodeClass split is a deliberate, worth-naming separation of concerns: NodePool expresses scheduling constraints (what instance families, architectures, capacity types are acceptable — closely mirroring the pod affinity/requirement vocabulary from Part 2), while EC2NodeClass expresses infrastructure configuration (AMI, subnets, security groups, IAM role) — the same instance-selection policy can be reused across multiple infrastructure configurations, and vice versa.

---

Karpenter Consolidation and Spot#

Karpenter's second major capability, beyond just-in-time provisioning, is consolidation — continuously re-evaluating the fleet and actively replacing or removing nodes to reduce waste, not just scaling up on demand.

Diagram

Why this matters concretely, worth a specific number: the traditional Cluster Autoscaler only scales up reactively and scales down on a conservative timer (commonly 10 minutes of underutilization before removing a node) — Karpenter's consolidation loop runs continuously and can react in seconds, which in real fleets commonly translates into a 20-40% reduction in EC2 spend for spiky or right-sizing-sensitive workloads, without any change to the workloads themselves.

Spot integration is where Karpenter's design pays off most directly: because it selects from a broad instance pool rather than a single pre-defined type, it can pick from many Spot capacity pools simultaneously, dramatically reducing the odds of an interruption-heavy pool being the only option — directly extending the Spot discussion from the Capacity Planning & Performance series' autoscaling tutorial into an EKS-specific implementation.

Copy
# A NodePool weighted toward Spot, with graceful interruption handling
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-preferred
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
      nodeClassRef: {group: karpenter.k8s.aws, kind: EC2NodeClass, name: default}
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Karpenter automatically drains a node in response to an EC2 Spot two-minute interruption notice, giving in-flight pods a real chance at a graceful rescheduling — the exact same SIGTERM/readiness-probe discipline from Part 2 and Part 5's node-upgrade discussion applies here too: a pod that ignores SIGTERM will still be abruptly cut off regardless of how much warning Karpenter itself receives.

---

EKS Auto Mode#

EKS Auto Mode, one of the newest and most significant EKS features, takes the "managed compute" spectrum introduced in Part 5 one step further than Managed Node Groups or even Karpenter running yourself: AWS fully owns node provisioning, scaling, upgrading, and even the core system add-ons — you interact almost entirely at the pod level.

Diagram

How this differs from running Karpenter yourself, worth stating precisely: Auto Mode isn't "Karpenter as a product" — it's AWS operating the entire compute layer, including the underlying node OS lifecycle and patching, with Karpenter-style just-in-time provisioning as an implementation detail you don't directly configure. Self-run Karpenter gives you full control over NodePool/EC2NodeClass definitions, custom AMIs, and node-level customization; Auto Mode trades that control away entirely in exchange for essentially zero node-layer operational burden — directly extending the "how much do you want to own vs. hand off" spectrum from Part 5's Fargate/Autopilot discussion.

Copy
# Enabling EKS Auto Mode on cluster creation
eksctl create cluster --name my-cluster --enable-auto-mode

When Auto Mode is the right call, worth stating as a clear decision rule: teams that want Kubernetes's workload API without owning ANY node-layer operations, and are comfortable with AWS's default node configuration and update cadence. When it's not: teams needing custom AMIs (compliance-mandated hardened images), DaemonSet-heavy workloads with specific node-level requirements, or fine-grained control over instance selection beyond what Auto Mode's built-in logic exposes.

---

EKS Pod Identity — The IRSA Successor#

Part 5 covered IRSA (IAM Roles for Service Accounts) as EKS's workload-identity mechanism — OIDC federation between a Kubernetes ServiceAccount and an IAM role, eliminating long-lived AWS credentials in pods. EKS Pod Identity, released in late 2023, is AWS's newer, simpler successor mechanism, and deserves real treatment here since Part 5 only introduced IRSA at intro depth.

Diagram

The concrete mechanical difference from IRSA, worth naming precisely: IRSA requires the pod's ServiceAccount token to be exchanged through the cluster's own OIDC identity provider, which means every EKS cluster needs its own registered OIDC provider in IAM, and every IAM role's trust policy has to reference that specific cluster's OIDC provider ARN. Pod Identity removes the OIDC provider dependency entirely — a small Pod Identity Agent DaemonSet running on each node handles the credential exchange directly against AWS STS's AssumeRoleForPodIdentity API, using a much simpler Pod Identity Association (cluster + namespace + ServiceAccount → IAM role) instead of an OIDC trust policy.

Copy
# Create a Pod Identity association - dramatically simpler than IRSA's
# OIDC-provider-plus-trust-policy setup
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace default \
  --service-account checkout-service \
  --role-arn arn:aws:iam::123456789012:role/checkout-s3-access

Copy
# No special annotation needed on the ServiceAccount itself with Pod
# Identity - the association lives in AWS, not in the ServiceAccount manifest
apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout-service
  namespace: default

Why the IAM role trust policy is also simpler with Pod Identity, a genuinely concrete, citable difference:

Copy
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "pods.eks.amazonaws.com"},
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}

Compare this to IRSA's trust policy, which must reference the specific cluster's OIDC provider ARN and the exact namespace/ServiceAccount as a StringEquals condition on the OIDC provider's sub claim — Pod Identity's trust policy is cluster-agnostic, meaning the same IAM role's trust policy doesn't need editing when you associate it with a different cluster, a real operational win for organizations running many EKS clusters.

---

Pod Identity vs IRSA — Choosing and Migrating#

Diagram

A strong, senior-level answer on which to use: for any new EKS-only workload, Pod Identity is AWS's current recommendation — simpler setup, cluster-agnostic trust policies, and one less moving part (no OIDC provider to manage). IRSA remains fully supported and is still the right choice specifically when a workload identity pattern needs to work identically across EKS and a self-managed or non-AWS Kubernetes cluster using the same generic OIDC-federation mechanism, or during a migration window where both must coexist.

Migrating an existing workload from IRSA to Pod Identity is genuinely low-risk and can be done incrementally, since both can be associated with the same IAM role simultaneously during a transition:

1. Install the EKS Pod Identity Agent add-on on the cluster (see the Add-ons section below).
2. Add the pods.eks.amazonaws.com principal to the existing IAM role's trust policy, alongside the existing OIDC trust statement (both can coexist).
3. Create the Pod Identity Association for the same namespace/ServiceAccount.
4. Remove the eks.amazonaws.com/role-arn annotation from the ServiceAccount once confirmed working — Pod Identity doesn't need it, and leaving it doesn't break anything either, but a clean migration removes it.
5. Once every workload using that role has migrated, remove the OIDC-specific trust policy statement.

---

EKS-Managed Add-ons#

Core cluster components — the pieces that make basic pod networking, DNS, and storage actually function — can be run either as unmanaged, self-installed Kubernetes resources, or as EKS-managed add-ons, where AWS handles versioning, updates, and health monitoring as a first-class EKS API object rather than something you kubectl apply and forget about.

Diagram

Copy
# List available add-on types and versions for a cluster's Kubernetes version
aws eks describe-addon-versions --kubernetes-version 1.31 \
  --query 'addons[].addonName'

# Install the EBS CSI driver as a managed add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::123456789012:role/ebs-csi-role

# Check add-on health
aws eks describe-addon --cluster-name my-cluster --addon-name aws-ebs-csi-driver \
  --query 'addon.health'

Why choosing managed add-ons over self-installed versions is worth doing by default, a concrete operational argument: a self-installed CNI/CSI/CoreDNS still requires you to track upstream releases, test compatibility with your specific EKS Kubernetes version, and apply updates manually — a managed add-on surfaces this as a simple version-compatibility check against your cluster's control-plane version, and AWS actively tests each add-on version against each supported EKS Kubernetes version before publishing it as installable.

A real, worth-knowing exception: teams running a third-party CNI (Cilium or Calico, for instance, chosen for advanced network policy or eBPF-based dataplane features beyond what the VPC CNI offers) deliberately do NOT use the managed VPC CNI add-on — this is a legitimate, common architectural choice, not a mistake, when the workload genuinely needs capabilities the default CNI doesn't provide.

---

Add-on Update Strategy and Versioning#

Diagram

Copy
# Update an add-on, preserving any custom configuration you've applied
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.19.0-eksbuild.1 \
  --resolve-conflicts PRESERVE

Why PRESERVE is almost always the correct choice for the VPC CNI specifically, a real operational trap to know about: teams commonly customize VPC CNI configuration for custom networking or prefix delegation (see the next section) — updating with OVERWRITE silently reverts those customizations back to AWS defaults, which can break pod networking cluster-wide immediately after what looked like a routine update. This is a genuinely common, real production incident pattern worth naming explicitly in an interview: "always use PRESERVE when updating an add-on you've customized, and diff the add-on's actual applied configuration before and after any update."

---

Fargate Profiles In Depth#

Part 5 introduced EKS Fargate as "serverless containers, no node management" — a Fargate profile is the specific mechanism that determines which pods actually run on Fargate versus EC2 nodes within the same cluster.

Diagram

Copy
# Create a Fargate profile - any pod in the "serverless" namespace
# with the label workload=fargate runs on Fargate
eksctl create fargateprofile \
  --cluster my-cluster \
  --name serverless-workloads \
  --namespace serverless \
  --labels workload=fargate

A cluster can mix Fargate and EC2/Karpenter compute freely, and this is a genuinely common real architecture — stateless, bursty API workloads on Fargate profiles (no node capacity planning needed), while DaemonSet-requiring infrastructure (log shippers, node-level security agents) runs on a small EC2 or Karpenter-managed node group specifically because Fargate's per-pod-isolated model can't support DaemonSets at all (the exact constraint named in Part 5's Problem 1).

Fargate-specific limits genuinely worth knowing as concrete facts, not vague caveats: a Fargate pod is capped at a maximum of 4 vCPU / 16 GiB memory per pod (as of current published limits — always verify current values, since these have been raised over time), each Fargate pod gets its own dedicated ENI and security group assignment by default (a real cost and IP-consumption consideration at scale, directly connecting to the VPC CNI IP-exhaustion discussion in Part 5), and Fargate pods cannot use hostNetwork, hostPort, privileged containers, or DaemonSets — all genuinely common EKS interview trip-ups for candidates who assume Fargate is a drop-in replacement for every EC2-based workload.

---

Custom Networking and IP Prefix Delegation#

Part 5 flagged VPC IP exhaustion as a real EKS-specific capacity concern, since the default VPC CNI assigns pods real, routable IPs directly from your VPC's subnet space. Two real, commonly-used mitigations deserve depth here.

Diagram

Copy
# Enabling prefix delegation on the VPC CNI add-on config
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  ENABLE_PREFIX_DELEGATION: "true"
  WARM_PREFIX_TARGET: "1"

Why prefix delegation is the more commonly reached-for fix, worth a concrete number: without it, an m5.large instance's ENIs can hold roughly 29 pod IPs total; with prefix delegation enabled, the same instance type can support several hundred pods — a genuinely dramatic increase that directly extends node density (and therefore lowers node count and cost) for pod-dense workloads, without changing instance type at all. Custom networking (routing pod traffic through a secondary CIDR block associated with the VPC) is the right tool specifically when the primary VPC CIDR itself is running low on free address space across the account, not just per-node ENI capacity — the two techniques solve genuinely different constraints and are often used together.

---

Security Groups for Pods#

A distinctive, EKS-specific capability: assigning AWS security groups directly to individual pods, not just to nodes — genuinely useful when different pods on the same node need different network-level access controls at the AWS security-group layer, on top of (not instead of) Kubernetes NetworkPolicy.

Diagram

Copy
# SecurityGroupPolicy - assigns a specific AWS security group to
# matching pods, requires ENABLE_POD_ENI=true on the VPC CNI
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: db-access-pods
spec:
  podSelector:
    matchLabels:
      role: database-client
  securityGroups:
    groupIds:
      - sg-0123456789abcdef0

Why this genuinely matters, distinct from NetworkPolicy, worth stating precisely: Kubernetes NetworkPolicy (deepened further in Part 3) operates entirely inside the cluster's own networking model — it can't directly reference or be evaluated by AWS-side resources like an RDS instance's own security group rules. Security Groups for Pods lets a specific pod be treated, at the AWS networking layer, exactly like an EC2 instance with its own security group — meaning an RDS security group rule that only allows traffic from a specific pod-level security group (rather than the entire node's broad security group) becomes possible, a meaningfully tighter blast-radius than "any pod on this node can reach the database."

---

API Endpoint Access Control#

Every EKS cluster exposes its Kubernetes API server through an endpoint — by default, publicly reachable from the internet (though still protected by IAM/RBAC authentication, per Part 1's control-plane discussion). This is a real, commonly-tested EKS security configuration choice.

Diagram

Copy
# Restrict the public endpoint to specific CIDR ranges (e.g., your
# office IP range and CI/CD runner IPs), keeping private access on
aws eks update-cluster-config \
  --name my-cluster \
  --resources-vpc-config \
  endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="203.0.113.0/24"

A strong, senior-level answer on which to choose, worth stating as an explicit tradeoff: private-only endpoints minimize the cluster's internet-facing attack surface most aggressively, but add real operational friction — every kubectl invocation, including from CI/CD runners, now requires network path into the VPC. Public + Private with a restricted publicAccessCidrs allowlist is the most common real production configuration, balancing genuine attack-surface reduction against not requiring VPN/bastion infrastructure for every engineer's local kubectl access — directly reusing the "public vs private subnet" tradeoff framing from the AWS Cloud Architecture series' networking tutorial, applied here to the control plane specifically.

---

Multi-Tenancy Patterns on EKS#

Running multiple teams or applications ("tenants") on one shared EKS cluster, safely, combines several mechanisms already covered individually — worth assembling into one coherent picture here.

Diagram

Three real multi-tenancy models worth naming as distinct, named patterns, not just "use namespaces":

A strong closing interview line on this: "Namespace-based soft multi-tenancy is the right default for trusted internal teams on a shared platform — it's the least operationally expensive and Kubernetes's own RBAC/NetworkPolicy/ResourceQuota primitives are specifically designed for it. I'd only move to node-group or cluster-per-tenant isolation when there's a genuine untrusted-tenant or strict compliance-boundary requirement that soft multi-tenancy can't satisfy — the cost difference between the three models is real and shouldn't be paid without a concrete reason."

---

GitOps on EKS#

GitOps — declarative infrastructure and application state, continuously reconciled from a Git repository — is covered in full generality in the Automation, CI/CD & GitOps series (11-automation-cicd-gitops/03-gitops.md); here's the EKS-specific operational shape.

Diagram

Copy
# Installing ArgoCD onto an EKS cluster - it's just Kubernetes
# manifests/a Helm chart, deployed like any other workload
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd --namespace argocd --create-namespace

Why GitOps pairs especially well with EKS specifically, worth stating explicitly: because EKS's control plane is fully AWS-managed (Part 5), the only thing a platform team needs to keep continuously correct is workload and configuration state — GitOps's continuous-reconciliation model fits that responsibility boundary precisely, letting Git become the single source of truth for "what should be running" while AWS owns "is the control plane itself healthy." This also directly closes a real security gap: a GitOps controller with pull-only Git access and a scoped Pod Identity role means no human or CI system needs standing kubectl apply credentials to the cluster at all — changes flow entirely through a reviewed Git merge, extending the least-privilege discussion from the DevSecOps series' CI/CD pipeline security tutorial (05-devsecops/05-cicd-pipeline-security-and-supply-chain.md).

---

EKS Observability#

Diagram

Copy
# Enable Container Insights on an existing cluster (as an EKS add-on)
aws eks create-addon --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability

Choosing Container Insights vs. ADOT, a genuine, worth-stating tradeoff: Container Insights is the fastest path to CloudWatch-native dashboards with essentially zero configuration, well-suited to teams already standardized on CloudWatch as their observability backend. ADOT is the right choice when vendor portability matters — the same OpenTelemetry instrumentation and collector configuration can export to CloudWatch today and a different backend (Grafana, Datadog, an open-source stack) later without re-instrumenting application code, directly extending the OpenTelemetry discussion in the Observability series (04-observability).

Fluent Bit over the older Fluentd, worth a concrete reason: Fluent Bit is written in C rather than Ruby, uses meaningfully less memory per node (a real consideration when running as a DaemonSet across every node in a large cluster), and is AWS's own default recommendation for EKS log shipping as of current guidance — Fluentd remains supported but is the older, heavier option.

---

EKS Cost Optimization#

Diagram

A concrete, worked cost comparison, worth internalizing the shape of even without exact current prices: a steady-state, always-on workload is almost always cheaper on Managed Node Groups or Karpenter with On-Demand/Reserved capacity than on Fargate, because Fargate's per-pod premium compounds continuously; a spiky, bursty workload with long idle periods is frequently cheaper on Fargate, because you pay nothing for idle node capacity sitting unused between bursts — the right compute model is a function of the workload's utilization shape, not a fixed "Fargate is always more/less expensive" rule.

Compute Savings Plans specifically, worth knowing as distinct from EC2-only Reserved Instances: unlike EC2 Reserved Instances (locked to a specific instance family/region), a Compute Savings Plan is a commitment to a dollar-per-hour spend level, applied automatically across EC2 and Fargate and Lambda usage — a materially better fit for a Karpenter-managed fleet whose exact instance-type mix changes continuously, since the discount isn't tied to a specific instance type at all.

---

EKS Upgrade Strategy In Depth#

Part 5 covered the general cordon-and-drain node upgrade mechanics common to all managed Kubernetes providers. EKS-specific upgrade strategy deserves more depth given how operationally significant it is.

Diagram

Why Blue/Green node group upgrades are frequently preferred for genuinely critical production workloads over in-place, a real, concrete operational argument: in-place upgrades replace nodes within the existing node group, meaning a failed or problematic new-version node directly reduces the capacity of your ONLY node group for that workload type mid-upgrade. Blue/Green keeps the old, known-good node group fully intact and running until the new one is verified healthy, giving you a genuine, fast rollback path (just shift traffic/scheduling back to the old group) rather than needing to reverse an in-place update.

Extended Support, worth knowing as a real, named EKS-specific concept: AWS offers a paid Extended Support period for EKS Kubernetes versions past their standard support end-date, giving teams more runway to plan an upgrade rather than being forced onto an unsupported version — a genuinely practical fact for any team managing upgrade cadence against real business constraints (change freezes, major release cycles) rather than always being able to upgrade the moment a version reaches end-of-standard-support.

Copy
# A real upgrade sequence: control plane, then add-ons, then nodes
aws eks update-cluster-version --name my-cluster --kubernetes-version 1.31

# Wait for control plane upgrade to complete, THEN update add-ons
aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni \
  --resolve-conflicts PRESERVE

# THEN update node groups (in-place example)
aws eks update-nodegroup-version --cluster-name my-cluster --nodegroup-name standard-workers

The strict ordering matters, worth stating explicitly: control plane, then add-ons, then nodes — attempting to upgrade nodes to a Kubernetes version newer than the control plane is rejected outright (nodes can run up to a limited number of minor versions behind the control plane, but never ahead of it), and add-ons compatible with the new control-plane version should be confirmed before rolling nodes forward, since an add-on/node-kernel mismatch is a real, if less common, source of upgrade incidents.

---

EKS Security Hardening Checklist#

A consolidated, EKS-specific checklist, pulling together points made throughout this Part plus direct references back to the DevSecOps series for the generic Kubernetes security material already covered there (05-devsecops/03-container-and-kubernetes-security.md):

Copy
# Enable EKS control plane audit logging - genuinely commonly missed,
# since it's opt-in per log type, not on by default
aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator"],"enabled":true}]}'

---

A Full Worked Example: Migrating from Cluster Autoscaler to Karpenter#

A concrete, step-by-step migration — the single most common real EKS modernization project a platform team runs, and a genuinely strong thing to be able to walk through end-to-end in an interview.

Diagram

Step 1 — install Karpenter without touching existing capacity:

Step 1: Install Karpenter via Helm

Copy
helm registry login public.ecr.aws
helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version v1.0.0 \
  --namespace kube-system \
  --set settings.clusterName=my-cluster \
  --set settings.interruptionQueue=my-cluster-karpenter

Step 2: Apply a NodePool matching the existing node group's instance policy

Copy
kubectl apply -f nodepool-general-purpose.yaml
kubectl apply -f ec2nodeclass-default.yaml

Step 3: Confirm Karpenter is actually provisioning nodes for new pending pods

Copy
kubectl get nodeclaims
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

Step 4: Reduce the old Managed Node Group's desired/max size gradually

Copy
aws eks update-nodegroup-config --cluster-name my-cluster \
  --nodegroup-name standard-workers \
  --scaling-config minSize=0,maxSize=0,desiredSize=0

Step 5: Once the old node group is empty, remove it and the Cluster Autoscaler deployment

Copy
eksctl delete nodegroup --cluster my-cluster --name standard-workers
kubectl delete deployment cluster-autoscaler -n kube-system

Why the gradual, parallel-running approach matters, worth stating explicitly: running Karpenter alongside the still-active old node group during the transition means every step is independently verifiable and reversible — if Karpenter-provisioned nodes show a problem, the old node group is still there to absorb load, and the migration can pause or roll back at any step rather than being an all-or-nothing cutover. This mirrors the same blue/green risk-reduction principle from the upgrade-strategy section above, applied to a compute-model migration instead of a version upgrade.

A real, worth-naming pitfall during this migration: forgetting to also remove the Cluster Autoscaler's IAM permissions and its own node-group autoscaling:* tags once fully migrated — a leftover, un-deleted Cluster Autoscaler deployment with stale node-group references will generate continuous, noisy error logs (harmless, but a genuine operational annoyance and a sign the migration wasn't fully cleaned up) even after all real capacity has moved to Karpenter.

---

Part 7 CLI Cheat Sheet#

Copy
# Managed Node Groups
eksctl create nodegroup --cluster my-cluster --name workers --node-type m6i.xlarge
eksctl get nodegroup --cluster my-cluster
aws eks update-nodegroup-version --cluster-name my-cluster --nodegroup-name workers
aws eks update-nodegroup-config --cluster-name my-cluster --nodegroup-name workers \
  --scaling-config minSize=2,maxSize=10,desiredSize=3

# Karpenter
kubectl get nodepools
kubectl get nodeclaims
kubectl describe nodepool general-purpose
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

# Pod Identity
aws eks create-pod-identity-association --cluster-name my-cluster \
  --namespace default --service-account my-sa --role-arn arn:aws:iam::123456789012:role/my-role
aws eks list-pod-identity-associations --cluster-name my-cluster
aws eks delete-pod-identity-association --cluster-name my-cluster --association-id a-abc123

# Add-ons
aws eks list-addons --cluster-name my-cluster
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni
aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni \
  --addon-version v1.19.0-eksbuild.1 --resolve-conflicts PRESERVE

# Fargate profiles
eksctl create fargateprofile --cluster my-cluster --name serverless --namespace serverless
eksctl get fargateprofile --cluster my-cluster

# Cluster and endpoint configuration
aws eks update-cluster-version --name my-cluster --kubernetes-version 1.31
aws eks update-cluster-config --name my-cluster \
  --resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="203.0.113.0/24"
aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit"],"enabled":true}]}'

# Diagnostics
kubectl get nodes -o wide --label-columns=karpenter.sh/capacity-type,node.kubernetes.io/instance-type
aws eks describe-cluster --name my-cluster --query 'cluster.status'
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name workers --query 'nodegroup.health'

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A platform team runs EKS with a fixed set of Managed Node Groups sized for peak traffic, and observes significant EC2 spend during off-peak hours when actual pod resource usage is a fraction of provisioned node capacity. What would you recommend, and why?

Answer: Introduce Karpenter (or migrate fully to it) instead of relying solely on fixed-shape Managed Node Groups. Karpenter's continuous consolidation loop actively removes or downsizes underutilized nodes in near-real-time — rather than a fixed node group sized for peak and left running at low utilization off-peak, Karpenter provisions capacity that tracks actual unschedulable-pod demand and aggressively right-sizes as that demand drops. Combined with a Spot-weighted NodePool for interruption-tolerant workloads and a Compute Savings Plan for the remaining On-Demand baseline, this directly targets the described waste — the traditional Cluster Autoscaler's conservative, timer-based scale-down wouldn't capture nearly as much of this savings, since it only removes empty nodes on a fixed delay rather than continuously right-sizing underutilized ones.

Problem 2: A security review flags that an EKS cluster's IAM role trust policies must be individually edited every time a workload identity needs to be replicated to a second, disaster-recovery EKS cluster in another region. What's the concrete fix, and why does it work?

Answer: Migrate the affected workloads from IRSA to EKS Pod Identity. IRSA's trust policies are cluster-specific — each references the exact OIDC provider ARN of one specific cluster, so replicating a workload's identity to a second cluster genuinely requires editing (or duplicating) the IAM role's trust policy to add the new cluster's OIDC provider. Pod Identity's trust policy instead just trusts the generic pods.eks.amazonaws.com service principal — cluster-agnostic — with the actual cluster+namespace+ServiceAccount binding held in a lightweight Pod Identity Association object per cluster instead of baked into the IAM trust policy itself. Migrating removes the need to touch the trust policy at all when adding a DR cluster; only a new Association needs to be created there.

Problem 3: An engineer requests full cluster-per-tenant isolation for every team on a new internal developer platform built on EKS, citing "maximum security." The platform team pushes back. What's the reasoning, and what would you propose instead?

Answer: Cluster-per-tenant is the most operationally expensive multi-tenancy model — every additional cluster is a separate control plane to upgrade, patch, monitor, and secure, multiplying the platform team's ongoing operational burden roughly linearly with tenant count, without a correspondingly large increase in actual isolation guarantee for trusted internal teams (as opposed to genuinely untrusted third parties). The right question isn't "what's maximally isolated" but "what isolation does this specific trust boundary actually require." For trusted internal teams, namespace-based soft multi-tenancy — RBAC scoped per namespace, default-deny NetworkPolicy, ResourceQuota/LimitRange, and per-namespace Pod Identity associations — gives strong, real isolation at a fraction of the operational cost, and is Kubernetes's own primitives working as designed. Escalating to node-group-per-tenant or cluster-per-tenant should be reserved for a genuine, named requirement (a specific compliance boundary, or a tenant that's actually untrusted), not applied uniformly "to be safe."

---

Summary and What's Next#

• Managed Node Groups wrap an AWS-owned Auto Scaling Group with automated bootstrap, graceful termination, and AMI version tracking — launch templates and custom AMIs (including Bottlerocket) give you control beyond the defaults when needed.
• Karpenter replaces fixed-shape node groups with just-in-time, right-sized provisioning driven by actual unschedulable pod requirements, plus continuous consolidation that a timer-based Cluster Autoscaler can't match — a genuinely major EC2 cost lever.
• EKS Auto Mode takes managed compute one step further than Karpenter-you-run-yourself: AWS owns the entire node layer, including OS patching, in exchange for reduced customization.
• EKS Pod Identity is AWS's current recommended successor to IRSA — simpler setup, cluster-agnostic trust policies — while IRSA remains the right choice for cross-platform OIDC-federation portability.
• EKS-managed add-ons (VPC CNI, CoreDNS, kube-proxy, EBS/EFS CSI, Pod Identity Agent) centralize version tracking and health monitoring — always update customized add-ons with PRESERVE, never OVERWRITE.
• Fargate profiles, custom networking/prefix delegation, and Security Groups for Pods round out EKS's distinctive compute and networking mechanisms beyond what Part 5 introduced.
• Multi-tenancy, GitOps, observability, cost optimization, upgrade strategy, and security hardening are the operational disciplines that separate "a cluster is running" from a production-grade EKS platform — each reuses primitives from Parts 1-6 and the DevSecOps/Automation series, applied specifically to EKS's own tooling.

This is the final part of the Kubernetes Deep Dive series. For the AWS-native compute services that sit alongside EKS (ECS, Fargate for ECS, Lambda), continue to the AWS Cloud Architecture series' dedicated service deep-dive chapters.