# Kubernetes Deep Dive — Part 7: Amazon EKS in Production Depth

> **Series:** Kubernetes Deep Dive (7 of 7)
> **Part 1:** `01-architecture-and-control-plane.md` — Architecture & Control Plane
> **Part 2:** `02-scheduling-and-workloads.md` — Scheduling & Workload Objects
> **Part 3:** `03-networking-and-storage.md` — Networking (CNI) & Storage (CSI)
> **Part 4:** `04-service-mesh-and-advanced-topics.md` — Service Mesh, etcd & Operators
> **Part 5:** `05-managed-kubernetes-eks-aks-gke.md` — Managed Kubernetes: EKS, AKS, GKE
> **Part 6:** `06-onprem-and-cluster-provisioning.md` — On-Prem & Self-Managed Kubernetes
> **Part 7:** This file — Amazon EKS in Production Depth
> **Questions:** `questions.md`

## Table of Contents

1. [Why This Part Exists](#why-this-part-exists)
2. [Managed Node Groups — Full Depth](#managed-node-groups--full-depth)
3. [Launch Templates and Custom AMIs](#launch-templates-and-custom-amis)
4. [Karpenter — Modern Node Autoscaling](#karpenter--modern-node-autoscaling)
5. [Karpenter Consolidation and Spot](#karpenter-consolidation-and-spot)
6. [EKS Auto Mode](#eks-auto-mode)
7. [EKS Pod Identity — The IRSA Successor](#eks-pod-identity--the-irsa-successor)
8. [Pod Identity vs IRSA — Choosing and Migrating](#pod-identity-vs-irsa--choosing-and-migrating)
9. [EKS-Managed Add-ons](#eks-managed-add-ons)
10. [Add-on Update Strategy and Versioning](#add-on-update-strategy-and-versioning)
11. [Fargate Profiles In Depth](#fargate-profiles-in-depth)
12. [Custom Networking and IP Prefix Delegation](#custom-networking-and-ip-prefix-delegation)
13. [Security Groups for Pods](#security-groups-for-pods)
14. [API Endpoint Access Control](#api-endpoint-access-control)
15. [Multi-Tenancy Patterns on EKS](#multi-tenancy-patterns-on-eks)
16. [GitOps on EKS](#gitops-on-eks)
17. [EKS Observability](#eks-observability)
18. [EKS Cost Optimization](#eks-cost-optimization)
19. [EKS Upgrade Strategy In Depth](#eks-upgrade-strategy-in-depth)
20. [EKS Security Hardening Checklist](#eks-security-hardening-checklist)
21. [A Full Worked Example: Migrating from Cluster Autoscaler to Karpenter](#a-full-worked-example-migrating-from-cluster-autoscaler-to-karpenter)
22. [Part 7 CLI Cheat Sheet](#part-7-cli-cheat-sheet)
23. [Common Mistakes](#common-mistakes)
24. [Worked Practice Problems](#worked-practice-problems)
25. [Summary and What's Next](#summary-and-whats-next)

---

## Why This Part Exists

Part 5 introduced EKS as one of three managed Kubernetes offerings — architecture, the EC2-vs-Fargate compute choice, IRSA, and the VPC CNI, each covered at a level that lets you compare EKS fairly against AKS and GKE. That comparison-level treatment is deliberately shallow on any single provider, because its job is breadth across three.

**This Part goes the other direction: EKS only, and deep enough to actually run a production cluster on it.** Everything here assumes you already know the material from Parts 1-6 — the Kubernetes API model, scheduling, networking, storage, and the basic EKS shape from Part 5 — and builds forward from there into the tools and patterns a real EKS platform team uses every day: Karpenter instead of the older Cluster Autoscaler, Pod Identity as the current recommended successor to IRSA, EKS Auto Mode as AWS's newest fully-managed compute option, and the operational muscle memory around upgrades, multi-tenancy, and cost that separates "we got a cluster running" from "we run this in production."

```mermaid
graph TD
    Part5["Part 5: EKS vs AKS vs GKE<br/>(architecture, EC2 vs Fargate,<br/>IRSA basics, VPC CNI basics)"] --> Part7["Part 7 (this file): EKS ONLY,<br/>production depth"]
    Part7 --> Compute["Node compute: Managed<br/>Node Groups, Karpenter,<br/>Auto Mode, Fargate profiles"]
    Part7 --> Identity["Identity: Pod Identity,<br/>IRSA migration"]
    Part7 --> Network["Networking: custom<br/>networking, security groups<br/>for pods, endpoint access"]
    Part7 --> Ops["Operations: upgrades,<br/>multi-tenancy, GitOps,<br/>observability, cost"]
```

---

## Managed Node Groups — Full Depth

Part 5 introduced EC2 Managed Node Groups at a surface level: "AWS automates the EC2 provisioning and lifecycle." Here's what that actually means operationally.

A **Managed Node Group** is AWS's abstraction over an underlying Auto Scaling Group (ASG, from the AWS Cloud Architecture series' compute tutorial) — EKS creates and owns that ASG for you, wires it into the cluster's bootstrap process, and exposes a simplified API surface (`eksctl create nodegroup`, or the equivalent Terraform/CloudFormation resource) instead of requiring you to hand-configure the ASG, launch template, and node bootstrap script yourself.

```mermaid
graph TD
    MNG["EKS Managed Node Group"] --> ASG["Underlying Auto Scaling<br/>Group (AWS-managed, but<br/>visible in your account)"]
    ASG --> LT["Launch Template (either<br/>AWS-generated default,<br/>or your own custom one)"]
    MNG --> Lifecycle["AWS handles: node<br/>registration, graceful<br/>termination (cordon+drain<br/>before scale-in), AMI<br/>updates via managed<br/>node group update API"]
```

```bash
# Create a managed node group with eksctl
eksctl create nodegroup \
  --cluster my-cluster \
  --name standard-workers \
  --node-type m6i.xlarge \
  --nodes 3 --nodes-min 3 --nodes-max 10 \
  --node-labels "workload-type=general" \
  --asg-access

# List node groups and their status
eksctl get nodegroup --cluster my-cluster

# Trigger a managed node group AMI/version update
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name standard-workers
```

**What "managed" buys you concretely, worth naming explicitly:**

| Capability | Self-managed ASG (pre-EKS-managed-node-groups pattern) | EKS Managed Node Group |
|---|---|---|
| Node bootstrap script | You write and maintain it | AWS provides and maintains it (via the EKS-optimized AMI's `bootstrap.sh`) |
| Graceful node termination on scale-in | You build cordon/drain logic yourself | Built in — AWS cordons and drains before terminating |
| AMI version tracking | Manual | `update-nodegroup-version` API tracks and applies the latest EKS-optimized AMI for your cluster's Kubernetes version |
| Node health checks | Manual CloudWatch alarm wiring | Built-in node health monitoring, unhealthy nodes automatically replaced |
| Visibility in EKS console | None — just an ASG | First-class EKS console object, correlated with the cluster |

**Node taints and labels at the node-group level:** a genuinely important, frequently-missed capability — you can apply taints and labels to an entire managed node group at creation time, so every node it ever creates (including future scale-out nodes) automatically carries them, without needing a DaemonSet or admission webhook to apply them after the fact.

```bash
eksctl create nodegroup \
  --cluster my-cluster \
  --name gpu-workers \
  --node-type g5.xlarge \
  --node-labels "workload-type=gpu" \
  --node-taints "nvidia.com/gpu=true:NoSchedule"
```

This directly reuses the taints/tolerations mechanism from Part 2 — the practical EKS-specific point is *where* you declare the taint (at the node group, not per-node or via a controller), so it survives node replacement automatically.

---

## Launch Templates and Custom AMIs

By default, a Managed Node Group uses an AWS-generated launch template built from the **EKS-optimized AMI** for your chosen Kubernetes version. Real production clusters frequently need more control than the default gives — custom disk sizing, additional bootstrap arguments, a hardened/custom AMI for compliance, or extra pre-installed agents (log shippers, security agents).

```mermaid
graph TD
    Default["Default launch template<br/>(AWS-managed): EKS-<br/>optimized AMI, default<br/>disk size, standard<br/>bootstrap args"] --> DefaultUse["Fine for most<br/>general-purpose workloads"]
    Custom["Custom launch template<br/>(you own it, node group<br/>references it)"] --> CustomUse["Custom AMI (hardened/<br/>compliance-required base),<br/>larger root volume,<br/>extra userdata, custom<br/>instance metadata options<br/>(IMDSv2 enforcement)"]
```

```bash
# A custom launch template referencing a hardened custom AMI,
# enforcing IMDSv2 (a real, commonly-tested EKS security hardening step)
aws ec2 create-launch-template \
  --launch-template-name eks-hardened-workers \
  --launch-template-data '{
    "ImageId": "ami-0123456789abcdef0",
    "BlockDeviceMappings": [{"DeviceName": "/dev/xvda", "Ebs": {"VolumeSize": 100, "VolumeType": "gp3"}}],
    "MetadataOptions": {"HttpTokens": "required", "HttpPutResponseHopLimit": 2}
  }'

eksctl create nodegroup \
  --cluster my-cluster \
  --name hardened-workers \
  --launch-template-id lt-0123456789abcdef0
```

**`HttpTokens: required` is worth calling out specifically** — it forces IMDSv2 (session-token-based instance metadata access) instead of the older, credential-leak-prone IMDSv1, directly closing the exact class of SSRF-to-credential-theft vulnerability class covered in the DevSecOps series' secrets management tutorial (`05-devsecops/04-secrets-management-and-iam.md`). Real production EKS node groups should enforce this by default, not leave it at the account-level default.

**Bottlerocket, worth knowing by name:** AWS's own purpose-built, minimal, immutable Linux distribution specifically for running containers — no shell, no package manager, an immutable root filesystem, and an API-driven update model. Choosing Bottlerocket over Amazon Linux 2023 as your node AMI trades general-purpose flexibility for a meaningfully smaller attack surface — a genuinely strong, specific answer to "how would you harden your EKS worker nodes" in an interview.

---

## Karpenter — Modern Node Autoscaling

**The single most important operational shift in EKS compute since Fargate**, and a concept every EKS practitioner needs cold: **Karpenter** is an open-source (originally AWS-built, now a CNCF project) node autoscaler that replaces the traditional Cluster Autoscaler + fixed node group model with direct, just-in-time EC2 provisioning driven by actual unschedulable pod requirements.

```mermaid
sequenceDiagram
    participant Pod as Unschedulable Pod
    participant Sched as kube-scheduler
    participant Karpenter
    participant EC2 as EC2 Fleet API

    Pod->>Sched: Pending - no node fits
    Sched->>Karpenter: Watches for unschedulable pods
    Karpenter->>Karpenter: Evaluates pod's actual<br/>requirements - CPU, memory,<br/>GPU, arch, zone, taints
    Karpenter->>EC2: Directly requests the BEST-FIT<br/>instance type and capacity type<br/>- no pre-defined node group needed
    EC2-->>Karpenter: Instance launched
    Karpenter->>Pod: Node registers, pod schedules
```

**Why this is a genuinely different model from the traditional Cluster Autoscaler**, worth stating precisely for an interview: the Cluster Autoscaler scales *existing, pre-defined node groups* up and down within their configured min/max — it has to guess in advance which instance types and sizes you'll need, provisioned into fixed-shape groups. **Karpenter instead evaluates each unschedulable pod's actual resource requirements and provisions the right-sized instance for it directly, from a broad, flexible instance-type pool** — no need to pre-define a matrix of node groups for every possible workload shape.

```yaml
# A Karpenter NodePool - defines WHAT kinds of nodes Karpenter is allowed to launch
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: node.kubernetes.io/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
---
# The EC2NodeClass - defines HOW those nodes are configured (AMI, subnets, security groups)
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  subnetSelectorTerms:
    - tags: {karpenter.sh/discovery: my-cluster}
  securityGroupSelectorTerms:
    - tags: {karpenter.sh/discovery: my-cluster}
  role: KarpenterNodeRole-my-cluster
```

**The `NodePool` / `EC2NodeClass` split is a deliberate, worth-naming separation of concerns**: `NodePool` expresses *scheduling constraints* (what instance families, architectures, capacity types are acceptable — closely mirroring the pod affinity/requirement vocabulary from Part 2), while `EC2NodeClass` expresses *infrastructure configuration* (AMI, subnets, security groups, IAM role) — the same instance-selection policy can be reused across multiple infrastructure configurations, and vice versa.

---

## Karpenter Consolidation and Spot

Karpenter's second major capability, beyond just-in-time provisioning, is **consolidation** — continuously re-evaluating the fleet and actively replacing or removing nodes to reduce waste, not just scaling up on demand.

```mermaid
graph TD
    Underutilized["Karpenter detects a node<br/>running FEWER pods than<br/>a smaller/cheaper instance<br/>type could hold"] --> Consolidate["Provisions the smaller/<br/>cheaper replacement,<br/>reschedules pods onto it,<br/>terminates the old<br/>oversized node"]
    Empty["Karpenter detects a node<br/>with ZERO pods (all<br/>evicted/completed)"] --> Remove["Terminates it immediately<br/>- no fixed 'scale-down<br/>delay' window like the<br/>traditional Cluster<br/>Autoscaler"]
```

**Why this matters concretely, worth a specific number:** the traditional Cluster Autoscaler only scales *up* reactively and scales *down* on a conservative timer (commonly 10 minutes of underutilization before removing a node) — Karpenter's consolidation loop runs continuously and can react in seconds, which in real fleets commonly translates into a **20-40% reduction in EC2 spend** for spiky or right-sizing-sensitive workloads, without any change to the workloads themselves.

**Spot integration** is where Karpenter's design pays off most directly: because it selects from a broad instance pool rather than a single pre-defined type, it can pick from *many* Spot capacity pools simultaneously, dramatically reducing the odds of an interruption-heavy pool being the only option — directly extending the Spot discussion from the Capacity Planning & Performance series' autoscaling tutorial into an EKS-specific implementation.

```yaml
# A NodePool weighted toward Spot, with graceful interruption handling
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-preferred
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
      nodeClassRef: {group: karpenter.k8s.aws, kind: EC2NodeClass, name: default}
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
```

Karpenter automatically drains a node in response to an EC2 Spot two-minute interruption notice, giving in-flight pods a real chance at a graceful rescheduling — the exact same SIGTERM/readiness-probe discipline from Part 2 and Part 5's node-upgrade discussion applies here too: a pod that ignores SIGTERM will still be abruptly cut off regardless of how much warning Karpenter itself receives.

---

## EKS Auto Mode

**EKS Auto Mode**, one of the newest and most significant EKS features, takes the "managed compute" spectrum introduced in Part 5 one step further than Managed Node Groups or even Karpenter running yourself: **AWS fully owns node provisioning, scaling, upgrading, and even the core system add-ons — you interact almost entirely at the pod level.**

```mermaid
graph TD
    subgraph "Traditional EKS (Managed Node Groups or self-run Karpenter)"
    T1["You choose and configure<br/>node group / Karpenter<br/>NodePools"]
    T2["You manage node AMI<br/>updates"]
    T3["You install and manage<br/>core add-ons (CNI, CSI,<br/>DNS) yourself or via<br/>EKS-managed add-ons"]
    end
    subgraph "EKS Auto Mode"
    A1["AWS provisions and scales<br/>nodes AUTOMATICALLY based<br/>on pod demand - Karpenter-<br/>like behavior, but AWS-<br/>operated, not yours to<br/>configure directly"]
    A2["AWS manages node AMI<br/>updates and patching<br/>entirely"]
    A3["Core networking, storage,<br/>and load balancing add-ons<br/>are AWS-managed by default"]
    end
```

**How this differs from running Karpenter yourself, worth stating precisely:** Auto Mode isn't "Karpenter as a product" — it's AWS operating the entire compute layer, including the underlying node OS lifecycle and patching, with Karpenter-style just-in-time provisioning as an implementation detail you don't directly configure. Self-run Karpenter gives you full control over `NodePool`/`EC2NodeClass` definitions, custom AMIs, and node-level customization; **Auto Mode trades that control away entirely in exchange for essentially zero node-layer operational burden** — directly extending the "how much do you want to own vs. hand off" spectrum from Part 5's Fargate/Autopilot discussion.

```bash
# Enabling EKS Auto Mode on cluster creation
eksctl create cluster --name my-cluster --enable-auto-mode
```

**When Auto Mode is the right call, worth stating as a clear decision rule:** teams that want Kubernetes's workload API without owning ANY node-layer operations, and are comfortable with AWS's default node configuration and update cadence. **When it's not:** teams needing custom AMIs (compliance-mandated hardened images), DaemonSet-heavy workloads with specific node-level requirements, or fine-grained control over instance selection beyond what Auto Mode's built-in logic exposes.

---

## EKS Pod Identity — The IRSA Successor

Part 5 covered **IRSA (IAM Roles for Service Accounts)** as EKS's workload-identity mechanism — OIDC federation between a Kubernetes ServiceAccount and an IAM role, eliminating long-lived AWS credentials in pods. **EKS Pod Identity**, released in late 2023, is AWS's newer, simpler successor mechanism, and deserves real treatment here since Part 5 only introduced IRSA at intro depth.

```mermaid
sequenceDiagram
    participant Pod
    participant Agent as EKS Pod Identity Agent<br/>(DaemonSet, runs on every node)
    participant STS as AWS STS

    Pod->>Agent: Requests credentials via<br/>the local Pod Identity<br/>Agent endpoint - no OIDC<br/>token exchange needed
    Agent->>STS: Agent itself calls<br/>AssumeRoleForPodIdentity,<br/>using the association<br/>configured for this<br/>ServiceAccount + cluster
    STS-->>Agent: Temporary AWS credentials
    Agent-->>Pod: Credentials returned directly
```

**The concrete mechanical difference from IRSA, worth naming precisely:** IRSA requires the pod's ServiceAccount token to be exchanged through the cluster's own OIDC identity provider, which means every EKS cluster needs its own registered OIDC provider in IAM, and every IAM role's trust policy has to reference that specific cluster's OIDC provider ARN. **Pod Identity removes the OIDC provider dependency entirely** — a small **Pod Identity Agent DaemonSet** running on each node handles the credential exchange directly against AWS STS's `AssumeRoleForPodIdentity` API, using a much simpler **Pod Identity Association** (cluster + namespace + ServiceAccount → IAM role) instead of an OIDC trust policy.

```bash
# Create a Pod Identity association - dramatically simpler than IRSA's
# OIDC-provider-plus-trust-policy setup
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace default \
  --service-account checkout-service \
  --role-arn arn:aws:iam::123456789012:role/checkout-s3-access
```

```yaml
# No special annotation needed on the ServiceAccount itself with Pod
# Identity - the association lives in AWS, not in the ServiceAccount manifest
apiVersion: v1
kind: ServiceAccount
metadata:
  name: checkout-service
  namespace: default
```

**Why the IAM role trust policy is also simpler with Pod Identity**, a genuinely concrete, citable difference:

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "pods.eks.amazonaws.com"},
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}
```

Compare this to IRSA's trust policy, which must reference the specific cluster's OIDC provider ARN and the exact namespace/ServiceAccount as a `StringEquals` condition on the OIDC provider's `sub` claim — Pod Identity's trust policy is **cluster-agnostic**, meaning the same IAM role's trust policy doesn't need editing when you associate it with a different cluster, a real operational win for organizations running many EKS clusters.

---

## Pod Identity vs IRSA — Choosing and Migrating

```mermaid
graph TB
    subgraph "IRSA"
    I1["Requires per-cluster OIDC<br/>provider registration in IAM"]
    I2["Trust policy references<br/>specific cluster + namespace<br/>+ ServiceAccount"]
    I3["ServiceAccount annotation<br/>required"]
    I4["Works with EKS AND<br/>self-managed/other<br/>Kubernetes clusters using<br/>OIDC federation"]
    end
    subgraph "Pod Identity"
    P1["No per-cluster OIDC<br/>provider needed"]
    P2["Trust policy is cluster-<br/>agnostic - Association<br/>object holds the binding"]
    P3["No ServiceAccount<br/>annotation needed"]
    P4["EKS-ONLY - not portable<br/>to self-managed clusters"]
    end
```

| | IRSA | Pod Identity |
|---|---|---|
| Setup complexity | Higher (OIDC provider + per-cluster trust policy) | Lower (one Association API call) |
| Trust policy portability across clusters | No — cluster-specific | Yes — cluster-agnostic |
| Requires the Pod Identity Agent add-on | No | Yes (an EKS-managed add-on, see below) |
| Portable to non-EKS Kubernetes | Yes (OIDC federation is a generic mechanism) | No — EKS-specific |
| AWS's current recommendation for new EKS workloads | Superseded | **Recommended default for new setups** |

**A strong, senior-level answer on which to use:** for any *new* EKS-only workload, **Pod Identity is AWS's current recommendation** — simpler setup, cluster-agnostic trust policies, and one less moving part (no OIDC provider to manage). IRSA remains fully supported and is still the right choice specifically when a workload identity pattern needs to work identically across EKS *and* a self-managed or non-AWS Kubernetes cluster using the same generic OIDC-federation mechanism, or during a migration window where both must coexist.

**Migrating an existing workload from IRSA to Pod Identity** is genuinely low-risk and can be done incrementally, since both can be associated with the same IAM role simultaneously during a transition:

1. Install the **EKS Pod Identity Agent** add-on on the cluster (see the Add-ons section below).
2. Add the `pods.eks.amazonaws.com` principal to the existing IAM role's trust policy, alongside the existing OIDC trust statement (both can coexist).
3. Create the Pod Identity Association for the same namespace/ServiceAccount.
4. Remove the `eks.amazonaws.com/role-arn` annotation from the ServiceAccount once confirmed working — Pod Identity doesn't need it, and leaving it doesn't break anything either, but a clean migration removes it.
5. Once every workload using that role has migrated, remove the OIDC-specific trust policy statement.

---

## EKS-Managed Add-ons

Core cluster components — the pieces that make basic pod networking, DNS, and storage actually function — can be run either as unmanaged, self-installed Kubernetes resources, or as **EKS-managed add-ons**, where AWS handles versioning, updates, and health monitoring as a first-class EKS API object rather than something you `kubectl apply` and forget about.

```mermaid
graph TD
    Addons["EKS-Managed Add-ons"] --> CNI["Amazon VPC CNI - pod<br/>networking (Part 3/Part 5)"]
    Addons --> CoreDNS["CoreDNS - cluster DNS"]
    Addons --> KubeProxy["kube-proxy - Service<br/>routing (Part 1)"]
    Addons --> EBSCSI["Amazon EBS CSI Driver -<br/>persistent volume<br/>provisioning (Part 3)"]
    Addons --> EFSCSI["Amazon EFS CSI Driver -<br/>shared filesystem storage"]
    Addons --> PodIdentity["EKS Pod Identity Agent<br/>- see above"]
    Addons --> ADOT["ADOT (AWS Distro for<br/>OpenTelemetry) - see<br/>Observability section"]
```

```bash
# List available add-on types and versions for a cluster's Kubernetes version
aws eks describe-addon-versions --kubernetes-version 1.31 \
  --query 'addons[].addonName'

# Install the EBS CSI driver as a managed add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::123456789012:role/ebs-csi-role

# Check add-on health
aws eks describe-addon --cluster-name my-cluster --addon-name aws-ebs-csi-driver \
  --query 'addon.health'
```

**Why choosing managed add-ons over self-installed versions is worth doing by default, a concrete operational argument:** a self-installed CNI/CSI/CoreDNS still requires you to track upstream releases, test compatibility with your specific EKS Kubernetes version, and apply updates manually — a managed add-on surfaces this as a simple version-compatibility check against your cluster's control-plane version, and AWS actively tests each add-on version against each supported EKS Kubernetes version before publishing it as installable.

**A real, worth-knowing exception:** teams running a *third-party* CNI (Cilium or Calico, for instance, chosen for advanced network policy or eBPF-based dataplane features beyond what the VPC CNI offers) deliberately do NOT use the managed VPC CNI add-on — this is a legitimate, common architectural choice, not a mistake, when the workload genuinely needs capabilities the default CNI doesn't provide.

---

## Add-on Update Strategy and Versioning

```mermaid
flowchart TD
    Start{"How should an<br/>add-on conflict with<br/>existing custom<br/>configuration be<br/>handled during update?"} -->|"OVERWRITE"| Over["AWS's default config<br/>wins - any manual field<br/>edits you made are lost"]
    Start -->|"PRESERVE"| Pres["Your existing custom<br/>field values are kept,<br/>only untouched fields<br/>get AWS's defaults"]
```

```bash
# Update an add-on, preserving any custom configuration you've applied
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.19.0-eksbuild.1 \
  --resolve-conflicts PRESERVE
```

**Why `PRESERVE` is almost always the correct choice for the VPC CNI specifically, a real operational trap to know about:** teams commonly customize VPC CNI configuration for custom networking or prefix delegation (see the next section) — updating with `OVERWRITE` silently reverts those customizations back to AWS defaults, which can break pod networking cluster-wide immediately after what looked like a routine update. **This is a genuinely common, real production incident pattern worth naming explicitly in an interview: "always use `PRESERVE` when updating an add-on you've customized, and diff the add-on's actual applied configuration before and after any update."**

---

## Fargate Profiles In Depth

Part 5 introduced EKS Fargate as "serverless containers, no node management" — a **Fargate profile** is the specific mechanism that determines *which pods* actually run on Fargate versus EC2 nodes within the same cluster.

```mermaid
graph TD
    Profile["Fargate Profile - defines<br/>selector rules: namespace<br/>+ optional label matches"] --> Match{"Does a new pod's<br/>namespace/labels<br/>match a profile's<br/>selector?"}
    Match -->|Yes| Fargate["Pod scheduled onto<br/>Fargate - AWS provisions a<br/>dedicated, isolated micro-VM<br/>for exactly this pod"]
    Match -->|No| EC2["Pod scheduled normally<br/>onto EC2 nodes (Managed<br/>Node Group / Karpenter)"]
```

```bash
# Create a Fargate profile - any pod in the "serverless" namespace
# with the label workload=fargate runs on Fargate
eksctl create fargateprofile \
  --cluster my-cluster \
  --name serverless-workloads \
  --namespace serverless \
  --labels workload=fargate
```

**A cluster can mix Fargate and EC2/Karpenter compute freely, and this is a genuinely common real architecture** — stateless, bursty API workloads on Fargate profiles (no node capacity planning needed), while DaemonSet-requiring infrastructure (log shippers, node-level security agents) runs on a small EC2 or Karpenter-managed node group specifically because Fargate's per-pod-isolated model can't support DaemonSets at all (the exact constraint named in Part 5's Problem 1).

**Fargate-specific limits genuinely worth knowing as concrete facts, not vague caveats:** a Fargate pod is capped at a maximum of 4 vCPU / 16 GiB memory per pod (as of current published limits — always verify current values, since these have been raised over time), each Fargate pod gets its own dedicated ENI and security group assignment by default (a real cost and IP-consumption consideration at scale, directly connecting to the VPC CNI IP-exhaustion discussion in Part 5), and **Fargate pods cannot use `hostNetwork`, `hostPort`, privileged containers, or DaemonSets** — all genuinely common EKS interview trip-ups for candidates who assume Fargate is a drop-in replacement for every EC2-based workload.

---

## Custom Networking and IP Prefix Delegation

Part 5 flagged VPC IP exhaustion as a real EKS-specific capacity concern, since the default VPC CNI assigns pods real, routable IPs directly from your VPC's subnet space. Two real, commonly-used mitigations deserve depth here.

```mermaid
graph TD
    Problem["Default VPC CNI: each<br/>pod consumes one real<br/>VPC IP - large clusters<br/>can exhaust subnet<br/>capacity fast"] --> Sol1["Custom Networking:<br/>separate the ENI's<br/>PRIMARY IP (used for<br/>node-to-node traffic)<br/>from pod IPs, letting<br/>pods use a DIFFERENT,<br/>larger secondary CIDR"]
    Problem --> Sol2["Prefix Delegation: assign<br/>an entire /28 IPv4 PREFIX<br/>to an ENI at once instead<br/>of individual IPs -<br/>dramatically increases<br/>the number of pod IPs<br/>each node's ENIs can hold"]
```

```yaml
# Enabling prefix delegation on the VPC CNI add-on config
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  ENABLE_PREFIX_DELEGATION: "true"
  WARM_PREFIX_TARGET: "1"
```

**Why prefix delegation is the more commonly reached-for fix, worth a concrete number:** without it, an `m5.large` instance's ENIs can hold roughly 29 pod IPs total; with prefix delegation enabled, the same instance type can support several hundred pods — a genuinely dramatic increase that directly extends node density (and therefore lowers node count and cost) for pod-dense workloads, without changing instance type at all. **Custom networking** (routing pod traffic through a secondary CIDR block associated with the VPC) is the right tool specifically when the *primary* VPC CIDR itself is running low on free address space across the account, not just per-node ENI capacity — the two techniques solve genuinely different constraints and are often used together.

---

## Security Groups for Pods

A distinctive, EKS-specific capability: assigning **AWS security groups directly to individual pods**, not just to nodes — genuinely useful when different pods on the same node need different network-level access controls at the AWS security-group layer, on top of (not instead of) Kubernetes NetworkPolicy.

```mermaid
graph TD
    Node["EC2 Worker Node -<br/>ONE node-level security<br/>group by default"] --> PodA["Pod A: needs RDS access<br/>- gets its OWN security<br/>group referencing the RDS<br/>SG"]
    Node --> PodB["Pod B: no database<br/>access needed - keeps<br/>only the node's default<br/>security group"]
```

```yaml
# SecurityGroupPolicy - assigns a specific AWS security group to
# matching pods, requires ENABLE_POD_ENI=true on the VPC CNI
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
  name: db-access-pods
spec:
  podSelector:
    matchLabels:
      role: database-client
  securityGroups:
    groupIds:
      - sg-0123456789abcdef0
```

**Why this genuinely matters, distinct from NetworkPolicy, worth stating precisely:** Kubernetes NetworkPolicy (deepened further in Part 3) operates entirely *inside* the cluster's own networking model — it can't directly reference or be evaluated by AWS-side resources like an RDS instance's own security group rules. **Security Groups for Pods lets a specific pod be treated, at the AWS networking layer, exactly like an EC2 instance with its own security group** — meaning an RDS security group rule that only allows traffic from a specific *pod-level* security group (rather than the entire node's broad security group) becomes possible, a meaningfully tighter blast-radius than "any pod on this node can reach the database."

---

## API Endpoint Access Control

Every EKS cluster exposes its Kubernetes API server through an endpoint — by default, publicly reachable from the internet (though still protected by IAM/RBAC authentication, per Part 1's control-plane discussion). This is a real, commonly-tested EKS security configuration choice.

```mermaid
flowchart TD
    Q{"EKS API endpoint<br/>access configuration"} --> Pub["Public only (default):<br/>reachable from the<br/>internet, authenticated<br/>via IAM + RBAC"]
    Q --> PubPriv["Public + Private:<br/>reachable both from the<br/>internet AND from within<br/>the VPC directly - most<br/>common production choice"]
    Q --> PrivOnly["Private only: ONLY<br/>reachable from within the<br/>VPC (or a peered/connected<br/>network) - kubectl from a<br/>laptop requires a bastion,<br/>VPN, or Direct Connect"]
```

```bash
# Restrict the public endpoint to specific CIDR ranges (e.g., your
# office IP range and CI/CD runner IPs), keeping private access on
aws eks update-cluster-config \
  --name my-cluster \
  --resources-vpc-config \
  endpointPublicAccess=true,endpointPrivateAccess=true,publicAccessCidrs="203.0.113.0/24"
```

**A strong, senior-level answer on which to choose, worth stating as an explicit tradeoff:** private-only endpoints minimize the cluster's internet-facing attack surface most aggressively, but add real operational friction — every `kubectl` invocation, including from CI/CD runners, now requires network path into the VPC. **Public + Private with a restricted `publicAccessCidrs` allowlist is the most common real production configuration**, balancing genuine attack-surface reduction against not requiring VPN/bastion infrastructure for every engineer's local `kubectl` access — directly reusing the "public vs private subnet" tradeoff framing from the AWS Cloud Architecture series' networking tutorial, applied here to the control plane specifically.

---

## Multi-Tenancy Patterns on EKS

Running multiple teams or applications ("tenants") on one shared EKS cluster, safely, combines several mechanisms already covered individually — worth assembling into one coherent picture here.

```mermaid
graph TD
    Namespace["Namespace per tenant -<br/>the basic isolation unit<br/>(Part 1-2)"] --> RBAC["RBAC: RoleBindings<br/>scoped to each tenant's<br/>namespace only"]
    Namespace --> Quota["ResourceQuota + LimitRange<br/>per namespace - prevents<br/>one tenant starving<br/>others (Part 2)"]
    Namespace --> NetPol["NetworkPolicy: default-deny<br/>cross-namespace traffic<br/>unless explicitly allowed<br/>(Part 3, deepened)"]
    Namespace --> Identity["Pod Identity / IRSA scoped<br/>per-namespace ServiceAccount<br/>- tenant A's pods can't<br/>assume tenant B's IAM role"]
    Namespace --> PSA["Pod Security Admission<br/>(Part 4/DevSecOps series) -<br/>enforced per namespace"]
```

**Three real multi-tenancy models worth naming as distinct, named patterns, not just "use namespaces":**

| Model | Isolation strength | Real cost |
|---|---|---|
| **Namespace-based (soft multi-tenancy)** | RBAC + NetworkPolicy + ResourceQuota — good for trusted internal teams | Shared control plane and node capacity — a noisy-neighbor or control-plane-level issue can still affect all tenants |
| **Node-group-per-tenant (partial hard multi-tenancy)** | Adds node-level isolation via taints/tolerations + node affinity — tenant workloads never share a physical node | Higher cost (less bin-packing efficiency), more node groups to operate |
| **Cluster-per-tenant (full hard multi-tenancy)** | Complete isolation — separate control plane per tenant | Highest cost and operational overhead — N clusters to upgrade, monitor, and secure instead of one |

**A strong closing interview line on this:** "Namespace-based soft multi-tenancy is the right default for trusted internal teams on a shared platform — it's the least operationally expensive and Kubernetes's own RBAC/NetworkPolicy/ResourceQuota primitives are specifically designed for it. I'd only move to node-group or cluster-per-tenant isolation when there's a genuine untrusted-tenant or strict compliance-boundary requirement that soft multi-tenancy can't satisfy — the cost difference between the three models is real and shouldn't be paid without a concrete reason."

---

## GitOps on EKS

GitOps — declarative infrastructure and application state, continuously reconciled from a Git repository — is covered in full generality in the Automation, CI/CD & GitOps series (`11-automation-cicd-gitops/03-gitops.md`); here's the EKS-specific operational shape.

```mermaid
graph TD
    Git["Git repo - desired<br/>cluster state (Deployments,<br/>Services, ConfigMaps...)"] --> Controller["GitOps controller running<br/>IN the EKS cluster - ArgoCD<br/>or Flux, itself just a set<br/>of Kubernetes controllers<br/>and CRDs (Part 4)"]
    Controller --> Reconcile["Continuously reconciles:<br/>compares Git's desired<br/>state against the cluster's<br/>actual state, applies diffs<br/>automatically"]
    Controller --> Identity2["Controller's own pod uses<br/>Pod Identity/IRSA to pull<br/>from a private Git repo or<br/>ECR - same identity pattern<br/>as any other workload"]
```

```bash
# Installing ArgoCD onto an EKS cluster - it's just Kubernetes
# manifests/a Helm chart, deployed like any other workload
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd --namespace argocd --create-namespace
```

**Why GitOps pairs especially well with EKS specifically, worth stating explicitly:** because EKS's control plane is fully AWS-managed (Part 5), the *only* thing a platform team needs to keep continuously correct is workload and configuration state — GitOps's continuous-reconciliation model fits that responsibility boundary precisely, letting Git become the single source of truth for "what should be running" while AWS owns "is the control plane itself healthy." This also directly closes a real security gap: a GitOps controller with pull-only Git access and a scoped Pod Identity role means **no human or CI system needs standing `kubectl apply` credentials to the cluster at all** — changes flow entirely through a reviewed Git merge, extending the least-privilege discussion from the DevSecOps series' CI/CD pipeline security tutorial (`05-devsecops/05-cicd-pipeline-security-and-supply-chain.md`).

---

## EKS Observability

```mermaid
graph TD
    Cluster["EKS Cluster"] --> CI["Container Insights -<br/>CloudWatch-native metrics/<br/>logs for pods, nodes,<br/>cluster - simplest to<br/>enable, tightly AWS-<br/>integrated"]
    Cluster --> ADOT["ADOT (AWS Distro for<br/>OpenTelemetry) - vendor-<br/>neutral traces/metrics,<br/>can export to CloudWatch,<br/>X-Ray, OR a third-party<br/>backend (Observability<br/>series, Part 4)"]
    Cluster --> FluentBit["Fluent Bit - the standard<br/>EKS log-shipping DaemonSet,<br/>routes container logs to<br/>CloudWatch Logs or another<br/>destination"]
```

```bash
# Enable Container Insights on an existing cluster (as an EKS add-on)
aws eks create-addon --cluster-name my-cluster \
  --addon-name amazon-cloudwatch-observability
```

**Choosing Container Insights vs. ADOT, a genuine, worth-stating tradeoff:** Container Insights is the fastest path to CloudWatch-native dashboards with essentially zero configuration, well-suited to teams already standardized on CloudWatch as their observability backend. **ADOT is the right choice when vendor portability matters** — the same OpenTelemetry instrumentation and collector configuration can export to CloudWatch today and a different backend (Grafana, Datadog, an open-source stack) later without re-instrumenting application code, directly extending the OpenTelemetry discussion in the Observability series (`04-observability`).

**Fluent Bit over the older Fluentd, worth a concrete reason:** Fluent Bit is written in C rather than Ruby, uses meaningfully less memory per node (a real consideration when running as a DaemonSet across every node in a large cluster), and is AWS's own default recommendation for EKS log shipping as of current guidance — Fluentd remains supported but is the older, heavier option.

---

## EKS Cost Optimization

```mermaid
flowchart TD
    Cost{"EKS cost optimization<br/>levers, roughly ordered<br/>by typical impact"} --> Karp["Karpenter consolidation<br/>(above) - eliminates<br/>persistent over-provisioning,<br/>often the single biggest win"]
    Cost --> Spot["Spot for interruption-<br/>tolerant workloads -<br/>commonly 60-90% cheaper<br/>than On-Demand"]
    Cost --> SavingsPlans["Compute Savings Plans -<br/>committed-use discount,<br/>applies automatically to<br/>EC2 AND Fargate usage"]
    Cost --> Fargate2["Fargate: no cost for idle<br/>node capacity, but a real<br/>per-vCPU/GB premium vs<br/>equivalent EC2 - right for<br/>spiky, low-baseline<br/>workloads, wrong for<br/>steady-state high-density<br/>ones"]
    Cost --> RightSize["Vertical right-sizing via<br/>VPA (Part 2) recommendations<br/>feeding into Karpenter's<br/>instance selection"]
```

**A concrete, worked cost comparison, worth internalizing the shape of even without exact current prices:** a steady-state, always-on workload is almost always cheaper on Managed Node Groups or Karpenter with On-Demand/Reserved capacity than on Fargate, because Fargate's per-pod premium compounds continuously; a spiky, bursty workload with long idle periods is frequently cheaper on Fargate, because you pay nothing for idle node capacity sitting unused between bursts — **the right compute model is a function of the workload's utilization shape, not a fixed "Fargate is always more/less expensive" rule.**

**Compute Savings Plans specifically, worth knowing as distinct from EC2-only Reserved Instances:** unlike EC2 Reserved Instances (locked to a specific instance family/region), a Compute Savings Plan is a commitment to a dollar-per-hour spend level, applied automatically across EC2 *and* Fargate *and* Lambda usage — a materially better fit for a Karpenter-managed fleet whose exact instance-type mix changes continuously, since the discount isn't tied to a specific instance type at all.

---

## EKS Upgrade Strategy In Depth

Part 5 covered the general cordon-and-drain node upgrade mechanics common to all managed Kubernetes providers. EKS-specific upgrade strategy deserves more depth given how operationally significant it is.

```mermaid
graph TD
    CP["Control plane upgrade -<br/>ALWAYS happens first,<br/>AWS-managed, typically a<br/>brief API server<br/>availability blip, not<br/>downtime for running<br/>workloads"] --> Nodes["THEN node upgrades -<br/>your responsibility to<br/>trigger (unless using<br/>Auto Mode)"]
    Nodes --> InPlace["In-place: update-<br/>nodegroup-version, AWS<br/>cordons/drains/replaces<br/>nodes within the SAME<br/>node group"]
    Nodes --> BlueGreen["Blue/Green: create a NEW<br/>node group on the new<br/>version, gradually shift<br/>workloads over, delete the<br/>old node group once empty"]
```

**Why Blue/Green node group upgrades are frequently preferred for genuinely critical production workloads over in-place**, a real, concrete operational argument: in-place upgrades replace nodes within the existing node group, meaning a failed or problematic new-version node directly reduces the capacity of your ONLY node group for that workload type mid-upgrade. **Blue/Green keeps the old, known-good node group fully intact and running until the new one is verified healthy**, giving you a genuine, fast rollback path (just shift traffic/scheduling back to the old group) rather than needing to reverse an in-place update.

**Extended Support, worth knowing as a real, named EKS-specific concept:** AWS offers a paid **Extended Support** period for EKS Kubernetes versions past their standard support end-date, giving teams more runway to plan an upgrade rather than being forced onto an unsupported version — a genuinely practical fact for any team managing upgrade cadence against real business constraints (change freezes, major release cycles) rather than always being able to upgrade the moment a version reaches end-of-standard-support.

```bash
# A real upgrade sequence: control plane, then add-ons, then nodes
aws eks update-cluster-version --name my-cluster --kubernetes-version 1.31

# Wait for control plane upgrade to complete, THEN update add-ons
aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni \
  --resolve-conflicts PRESERVE

# THEN update node groups (in-place example)
aws eks update-nodegroup-version --cluster-name my-cluster --nodegroup-name standard-workers
```

**The strict ordering matters, worth stating explicitly:** control plane, then add-ons, then nodes — attempting to upgrade nodes to a Kubernetes version newer than the control plane is rejected outright (nodes can run up to a limited number of minor versions behind the control plane, but never ahead of it), and add-ons compatible with the new control-plane version should be confirmed before rolling nodes forward, since an add-on/node-kernel mismatch is a real, if less common, source of upgrade incidents.

---

## EKS Security Hardening Checklist

A consolidated, EKS-specific checklist, pulling together points made throughout this Part plus direct references back to the DevSecOps series for the generic Kubernetes security material already covered there (`05-devsecops/03-container-and-kubernetes-security.md`):

| Area | Hardening step |
|---|---|
| API endpoint | Restrict `publicAccessCidrs`, or go private-only with VPN/Direct Connect access, per the endpoint access section above |
| Node metadata | Enforce IMDSv2 (`HttpTokens: required`) on every launch template, closing the SSRF-to-credential-theft path |
| Node OS | Consider Bottlerocket for its minimal, immutable, shell-less attack surface |
| Workload identity | Prefer Pod Identity for new workloads; audit and migrate long-lived-credential-using workloads first |
| Network isolation | Default-deny NetworkPolicy per namespace (Part 3), Security Groups for Pods for AWS-side resource access control |
| Add-on updates | Always `PRESERVE` on customized add-ons; diff configuration before/after any update |
| Pod-level security | Pod Security Admission enforced per namespace (DevSecOps series) |
| Multi-tenancy | Match isolation model (namespace/node-group/cluster) to the actual trust boundary needed — don't over- or under-provision isolation |
| Audit logging | Enable EKS control plane logging (API server audit logs) to CloudWatch — off by default, a real, common gap |
| Secrets | KMS envelope encryption for Kubernetes Secrets at the etcd layer (a cluster-creation-time EKS setting), on top of Pod Identity/IRSA to avoid storing secrets in the cluster at all where possible |

```bash
# Enable EKS control plane audit logging - genuinely commonly missed,
# since it's opt-in per log type, not on by default
aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator"],"enabled":true}]}'
```

---

## A Full Worked Example: Migrating from Cluster Autoscaler to Karpenter

A concrete, step-by-step migration — the single most common real EKS modernization project a platform team runs, and a genuinely strong thing to be able to walk through end-to-end in an interview.

```mermaid
flowchart TD
    Start["Existing cluster: fixed<br/>Managed Node Groups +<br/>Cluster Autoscaler"] --> Step1["1. Install Karpenter<br/>alongside the existing<br/>setup - does NOT touch<br/>existing node groups yet"]
    Step1 --> Step2["2. Define NodePool/<br/>EC2NodeClass matching<br/>current instance-type<br/>policy"]
    Step2 --> Step3["3. Scale down (don't<br/>delete) the old node<br/>group's max size -<br/>new pods now land on<br/>Karpenter-provisioned<br/>nodes instead"]
    Step3 --> Step4["4. Drain remaining old-<br/>node-group nodes<br/>gradually, verify<br/>workloads reschedule<br/>cleanly onto Karpenter<br/>nodes"]
    Step4 --> Step5["5. Delete the old node<br/>group and the Cluster<br/>Autoscaler deployment<br/>once empty"]
```

**Step 1 — install Karpenter without touching existing capacity:**

> **Step 1: Install Karpenter via Helm**
> ```bash
> helm registry login public.ecr.aws
> helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
>   --version v1.0.0 \
>   --namespace kube-system \
>   --set settings.clusterName=my-cluster \
>   --set settings.interruptionQueue=my-cluster-karpenter
> ```
>
> **Step 2: Apply a NodePool matching the existing node group's instance policy**
> ```bash
> kubectl apply -f nodepool-general-purpose.yaml
> kubectl apply -f ec2nodeclass-default.yaml
> ```
>
> **Step 3: Confirm Karpenter is actually provisioning nodes for new pending pods**
> ```bash
> kubectl get nodeclaims
> kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f
> ```
>
> **Step 4: Reduce the old Managed Node Group's desired/max size gradually**
> ```bash
> aws eks update-nodegroup-config --cluster-name my-cluster \
>   --nodegroup-name standard-workers \
>   --scaling-config minSize=0,maxSize=0,desiredSize=0
> ```
>
> **Step 5: Once the old node group is empty, remove it and the Cluster Autoscaler deployment**
> ```bash
> eksctl delete nodegroup --cluster my-cluster --name standard-workers
> kubectl delete deployment cluster-autoscaler -n kube-system
> ```

**Why the gradual, parallel-running approach matters, worth stating explicitly:** running Karpenter alongside the still-active old node group during the transition means every step is independently verifiable and reversible — if Karpenter-provisioned nodes show a problem, the old node group is still there to absorb load, and the migration can pause or roll back at any step rather than being an all-or-nothing cutover. This mirrors the same blue/green risk-reduction principle from the upgrade-strategy section above, applied to a compute-model migration instead of a version upgrade.

**A real, worth-naming pitfall during this migration:** forgetting to also remove the Cluster Autoscaler's IAM permissions and its own node-group `autoscaling:*` tags once fully migrated — a leftover, un-deleted Cluster Autoscaler deployment with stale node-group references will generate continuous, noisy error logs (harmless, but a genuine operational annoyance and a sign the migration wasn't fully cleaned up) even after all real capacity has moved to Karpenter.

---

## Part 7 CLI Cheat Sheet

```bash
# Managed Node Groups
eksctl create nodegroup --cluster my-cluster --name workers --node-type m6i.xlarge
eksctl get nodegroup --cluster my-cluster
aws eks update-nodegroup-version --cluster-name my-cluster --nodegroup-name workers
aws eks update-nodegroup-config --cluster-name my-cluster --nodegroup-name workers \
  --scaling-config minSize=2,maxSize=10,desiredSize=3

# Karpenter
kubectl get nodepools
kubectl get nodeclaims
kubectl describe nodepool general-purpose
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

# Pod Identity
aws eks create-pod-identity-association --cluster-name my-cluster \
  --namespace default --service-account my-sa --role-arn arn:aws:iam::123456789012:role/my-role
aws eks list-pod-identity-associations --cluster-name my-cluster
aws eks delete-pod-identity-association --cluster-name my-cluster --association-id a-abc123

# Add-ons
aws eks list-addons --cluster-name my-cluster
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni
aws eks update-addon --cluster-name my-cluster --addon-name vpc-cni \
  --addon-version v1.19.0-eksbuild.1 --resolve-conflicts PRESERVE

# Fargate profiles
eksctl create fargateprofile --cluster my-cluster --name serverless --namespace serverless
eksctl get fargateprofile --cluster my-cluster

# Cluster and endpoint configuration
aws eks update-cluster-version --name my-cluster --kubernetes-version 1.31
aws eks update-cluster-config --name my-cluster \
  --resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="203.0.113.0/24"
aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit"],"enabled":true}]}'

# Diagnostics
kubectl get nodes -o wide --label-columns=karpenter.sh/capacity-type,node.kubernetes.io/instance-type
aws eks describe-cluster --name my-cluster --query 'cluster.status'
aws eks describe-nodegroup --cluster-name my-cluster --nodegroup-name workers --query 'nodegroup.health'
```

---

## Common Mistakes

| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Updating a customized add-on with `OVERWRITE` instead of `PRESERVE` | Silently reverts custom configuration (e.g. VPC CNI prefix delegation settings), can break pod networking cluster-wide immediately after the update | Always use `--resolve-conflicts PRESERVE` on any add-on you've customized, and diff its applied config before/after |
| Assuming EKS Auto Mode is "just Karpenter" | Auto Mode is AWS operating the entire node layer including OS patching, not a self-configurable Karpenter deployment — very different control/customization tradeoff | Choose Auto Mode only when full node-layer control isn't needed; use self-run Karpenter when custom AMIs or fine-grained NodePool control matter |
| Scheduling a DaemonSet-dependent workload (log shippers, node agents) onto a Fargate profile | Fargate's per-pod-isolated model doesn't support DaemonSets at all | Run DaemonSet-requiring infrastructure on EC2/Karpenter-managed nodes, keep Fargate for stateless, DaemonSet-independent workloads |
| Leaving the EKS control plane's audit logging disabled | It's opt-in, not default — a real, common gap that removes a critical forensic signal during a security incident | Enable `api`, `audit`, and `authenticator` control plane logging at cluster creation or via `update-cluster-config` |
| Choosing cluster-per-tenant multi-tenancy by default "to be safe" | Multiplies operational overhead (N clusters to upgrade, patch, and monitor) without a concrete compliance/trust requirement demanding that level of isolation | Default to namespace-based soft multi-tenancy for trusted internal teams; escalate isolation level only for a genuine, named requirement |

---

## Worked Practice Problems

**Problem 1:** A platform team runs EKS with a fixed set of Managed Node Groups sized for peak traffic, and observes significant EC2 spend during off-peak hours when actual pod resource usage is a fraction of provisioned node capacity. What would you recommend, and why?

*Answer:* Introduce Karpenter (or migrate fully to it) instead of relying solely on fixed-shape Managed Node Groups. Karpenter's continuous consolidation loop actively removes or downsizes underutilized nodes in near-real-time — rather than a fixed node group sized for peak and left running at low utilization off-peak, Karpenter provisions capacity that tracks actual unschedulable-pod demand and aggressively right-sizes as that demand drops. Combined with a Spot-weighted NodePool for interruption-tolerant workloads and a Compute Savings Plan for the remaining On-Demand baseline, this directly targets the described waste — the traditional Cluster Autoscaler's conservative, timer-based scale-down wouldn't capture nearly as much of this savings, since it only removes empty nodes on a fixed delay rather than continuously right-sizing underutilized ones.

**Problem 2:** A security review flags that an EKS cluster's IAM role trust policies must be individually edited every time a workload identity needs to be replicated to a second, disaster-recovery EKS cluster in another region. What's the concrete fix, and why does it work?

*Answer:* Migrate the affected workloads from IRSA to EKS Pod Identity. IRSA's trust policies are cluster-specific — each references the exact OIDC provider ARN of one specific cluster, so replicating a workload's identity to a second cluster genuinely requires editing (or duplicating) the IAM role's trust policy to add the new cluster's OIDC provider. Pod Identity's trust policy instead just trusts the generic `pods.eks.amazonaws.com` service principal — cluster-agnostic — with the actual cluster+namespace+ServiceAccount binding held in a lightweight Pod Identity Association object per cluster instead of baked into the IAM trust policy itself. Migrating removes the need to touch the trust policy at all when adding a DR cluster; only a new Association needs to be created there.

**Problem 3:** An engineer requests full cluster-per-tenant isolation for every team on a new internal developer platform built on EKS, citing "maximum security." The platform team pushes back. What's the reasoning, and what would you propose instead?

*Answer:* Cluster-per-tenant is the most operationally expensive multi-tenancy model — every additional cluster is a separate control plane to upgrade, patch, monitor, and secure, multiplying the platform team's ongoing operational burden roughly linearly with tenant count, without a correspondingly large increase in actual isolation guarantee for *trusted internal teams* (as opposed to genuinely untrusted third parties). The right question isn't "what's maximally isolated" but "what isolation does this specific trust boundary actually require." For trusted internal teams, namespace-based soft multi-tenancy — RBAC scoped per namespace, default-deny NetworkPolicy, ResourceQuota/LimitRange, and per-namespace Pod Identity associations — gives strong, real isolation at a fraction of the operational cost, and is Kubernetes's own primitives working as designed. Escalating to node-group-per-tenant or cluster-per-tenant should be reserved for a genuine, named requirement (a specific compliance boundary, or a tenant that's actually untrusted), not applied uniformly "to be safe."

---

## Summary and What's Next

- **Managed Node Groups** wrap an AWS-owned Auto Scaling Group with automated bootstrap, graceful termination, and AMI version tracking — launch templates and custom AMIs (including Bottlerocket) give you control beyond the defaults when needed.
- **Karpenter** replaces fixed-shape node groups with just-in-time, right-sized provisioning driven by actual unschedulable pod requirements, plus continuous consolidation that a timer-based Cluster Autoscaler can't match — a genuinely major EC2 cost lever.
- **EKS Auto Mode** takes managed compute one step further than Karpenter-you-run-yourself: AWS owns the entire node layer, including OS patching, in exchange for reduced customization.
- **EKS Pod Identity** is AWS's current recommended successor to IRSA — simpler setup, cluster-agnostic trust policies — while IRSA remains the right choice for cross-platform OIDC-federation portability.
- **EKS-managed add-ons** (VPC CNI, CoreDNS, kube-proxy, EBS/EFS CSI, Pod Identity Agent) centralize version tracking and health monitoring — always update customized add-ons with `PRESERVE`, never `OVERWRITE`.
- **Fargate profiles**, **custom networking/prefix delegation**, and **Security Groups for Pods** round out EKS's distinctive compute and networking mechanisms beyond what Part 5 introduced.
- **Multi-tenancy, GitOps, observability, cost optimization, upgrade strategy, and security hardening** are the operational disciplines that separate "a cluster is running" from a production-grade EKS platform — each reuses primitives from Parts 1-6 and the DevSecOps/Automation series, applied specifically to EKS's own tooling.

This is the final part of the Kubernetes Deep Dive series. For the AWS-native compute services that sit alongside EKS (ECS, Fargate for ECS, Lambda), continue to the AWS Cloud Architecture series' dedicated service deep-dive chapters.