Secrets Management & IAM

Table of Contents#

1. Why Secrets Are the Highest-Value Target
2. What Counts as a Secret
3. The Sins of Secret Management
4. Secret Scanning — Catching Leaks Before They Happen
5. Secret Scanning in Practice: Gitleaks and TruffleHog
6. What to Do the Moment a Secret Leaks
7. Dedicated Secrets Managers — Why Env Vars Aren't Enough
8. HashiCorp Vault — Architecture and Core Concepts
9. Vault in Practice — Commands and Policies
10. Dynamic Secrets — Vault's Killer Feature
11. Cloud-Native Alternatives: AWS Secrets Manager & KMS
12. Envelope Encryption, Explained Simply
13. IAM — Identity and Access Management
14. The Principle of Least Privilege, Applied Concretely
15. A Worked IAM Policy Example
16. Service Accounts, Workload Identity, and Why Not to Use Long-Lived Keys
17. Secret and Key Rotation
18. Common Mistakes
19. Worked Practice Problems
20. Summary and What's Next

---

Why Secrets Are the Highest-Value Target#

Of every category of security work covered in this series, secrets deserve special attention for one simple reason: a single leaked secret can bypass every other control you've built. All the RBAC, network policies, and scanning from Part 3 mean nothing if an attacker simply finds a valid database password or a cloud admin key sitting in plain text somewhere.

Diagram

A simple analogy: you can install the strongest lock, the best alarm system, and security cameras on your house — but if you leave a copy of your key under the doormat, none of it matters. Secrets management is about never leaving the key under the doormat.

---

What Counts as a Secret#

A broader list than most people initially think of — worth having ready in an interview.

Diagram

---

The Sins of Secret Management#

A memorable checklist of what NOT to do:

Diagram

The genuinely important, commonly-missed point about "removing" a secret from git: deleting a line from a file and committing that change does not remove the secret from the repository — it's still fully readable in the git history, in every prior commit, for anyone with clone access. The only real fix is treating the secret as permanently compromised (rotate it immediately) and, if truly necessary, rewriting git history entirely (a disruptive, last-resort operation) to purge it.

Copy
# This does NOT remove the secret from history — it's still
# fully recoverable from prior commits
git rm secrets.env
git commit -m "remove secrets file"

# The secret is STILL here:
git log --all --full-history -- secrets.env
git show <old-commit-hash>:secrets.env

---

Secret Scanning — Catching Leaks Before They Happen#

Exactly the shift-left principle from Part 1, applied specifically to secrets: catch a hardcoded credential before it's ever committed, or at latest, the moment it's pushed — not days later during manual review.

Diagram

---

Secret Scanning in Practice: Gitleaks and TruffleHog#

Copy
# Gitleaks — scan a repo's current files AND its full git history
gitleaks detect --source . --verbose

# Scan only a specific commit range (e.g. in a CI pull-request job)
gitleaks detect --source . --log-opts="HEAD~10..HEAD"

# Set up gitleaks as a pre-commit hook (using the pre-commit framework)

Copy
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

Copy
pip install pre-commit
pre-commit install    # installs the git hook locally
# Now every `git commit` automatically runs gitleaks first

Copy
# TruffleHog — another very widely used option, notable for
# actually VERIFYING many secret types against their real APIs
# (reduces false positives dramatically — it doesn't just pattern-
# match, it checks whether the key is ACTUALLY still valid)
trufflehog git file://. --only-verified

Why TruffleHog's "verification" feature is worth knowing specifically: a plain pattern-matching scanner will flag anything that looks like an AWS key format, even if it's a fake example in a test fixture or documentation. TruffleHog can actually attempt to use certain credential types against their real provider APIs to confirm whether they're genuinely live and valid — dramatically cutting down on false positives, directly addressing the alert-fatigue problem raised throughout this series.

---

What to Do the Moment a Secret Leaks#

A concrete, step-by-step incident response worth having memorized — this is a very common scenario-based interview question.

Diagram

The single most important, most-often-missed step: rotate first, investigate second. A common but wrong instinct is to spend time figuring out "was this actually used maliciously" before rotating — but every minute the secret remains valid is a minute an attacker (who may have already found it, completely independent of your own discovery) can use it. Rotation is fast, cheap, and reversible in impact; delay is not.

---

Dedicated Secrets Managers — Why Env Vars Aren't Enough#

Environment variables are a common, simple way to inject a secret into an application — but they have real, specific limitations worth naming.

Diagram

A dedicated secrets manager (Vault, AWS Secrets Manager, etc.) fixes all four of these: secrets are fetched at runtime via an authenticated API call (not baked into the environment permanently), access is centrally audited, rotation can be automated, and — as covered below — some systems can even generate short-lived, dynamically-created credentials instead of long-lived static ones.

---

HashiCorp Vault — Architecture and Core Concepts#

Vault is the most widely used, tool-agnostic secrets management platform — genuinely important to understand at an architectural level, not just as a product name to recognize.

Diagram

Key Concepts#

---

Vault in Practice — Commands and Policies#

Copy
# Start a local Vault dev server (NEVER use dev mode in production —
# it's unsealed automatically and stores everything in memory)
vault server -dev

# Write a static secret
vault kv put secret/checkout/db-credentials \
  username="checkout_app" \
  password="s3cr3t-generated-value"

# Read it back
vault kv get secret/checkout/db-credentials

A Vault policy implementing least privilege — the checkout-service identity can only read its own secrets, nothing else:

Copy
# checkout-policy.hcl
path "secret/data/checkout/*" {
  capabilities = ["read"]
}

# Explicitly NOT granted: write, delete, or any access to
# secret/data/payments/* or any other service's secrets

Copy
vault policy write checkout-policy checkout-policy.hcl

# Set up Kubernetes auth so pods can authenticate using their
# own ServiceAccount token (no static credential needed at all)
vault auth enable kubernetes
vault write auth/kubernetes/role/checkout-service \
  bound_service_account_names=checkout-sa \
  bound_service_account_namespaces=checkout \
  policies=checkout-policy \
  ttl=1h

Why the Kubernetes auth method is worth knowing specifically: it means an application pod never needs a long-lived, static Vault credential baked in anywhere at all — it authenticates using its own Kubernetes-issued identity (its ServiceAccount token), which Vault verifies directly against the Kubernetes API. This closes the loop with Part 3's RBAC discussion: the same least-privilege identity system (Kubernetes ServiceAccounts) is reused to control access to secrets, not just to the Kubernetes API itself.

---

Dynamic Secrets — Vault's Killer Feature#

This is genuinely one of the most impressive, high-value concepts in this entire tutorial — worth understanding deeply, since it's a favorite advanced interview topic.

Diagram

The core idea, in plain terms: instead of one static, shared, long-lived database password that every instance of your application uses forever, Vault can dynamically create a brand-new, unique database user on demand, hand it to exactly one requesting application instance, and automatically delete that specific user once its lease expires.

Diagram

Why this is such a strong interview answer to "how would you reduce the blast radius of a leaked database credential": "Instead of one long-lived shared password, use Vault's dynamic secrets engine so every application instance gets its own unique, automatically-expiring credential. A leak now has a bounded lifetime (it self-expires) instead of remaining valid indefinitely until someone manually notices and rotates it — this shrinks the actual damage a leak can do, rather than just trying to prevent leaks from happening in the first place, which you can never fully guarantee."

---

Cloud-Native Alternatives: AWS Secrets Manager & KMS#

Vault is powerful but adds real operational overhead (you have to run and maintain it). Cloud-native alternatives are often simpler for teams fully committed to one cloud provider.

Copy
# AWS Secrets Manager — store a secret
aws secretsmanager create-secret \
  --name checkout/db-credentials \
  --secret-string '{"username":"checkout_app","password":"s3cr3t"}'

# Retrieve it (application code typically uses the SDK, not the CLI,
# for this — shown here for clarity)
aws secretsmanager get-secret-value --secret-id checkout/db-credentials

# Enable automatic rotation on a schedule, using a Lambda function
# AWS provides pre-built rotation templates for common databases
aws secretsmanager rotate-secret \
  --secret-id checkout/db-credentials \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:SecretsManagerRDSRotation \
  --rotation-rules AutomaticallyAfterDays=30

---

Envelope Encryption, Explained Simply#

A commonly-tested concept about how cloud KMS (Key Management Service) systems actually encrypt data efficiently.

Diagram

Why this two-layer approach exists, rather than just encrypting everything directly with the KMS key: calling out to a remote KMS for every single byte of a large file would be incredibly slow (a network round-trip per encryption operation). Instead, a fast, local, one-time-use key (the DEK) does the actual heavy-lifting encryption of the real data, and only that small DEK itself needs to be encrypted by the KMS (the KEK) — one fast network call, regardless of how large the underlying data is. This is exactly how AWS KMS, Google Cloud KMS, and Vault's Transit secrets engine all work under the hood, and being able to explain why this two-layer design exists (not just naming it) is a genuine, strong interview signal.

---

IAM — Identity and Access Management#

IAM governs who (or what) can do what, to which resources, across your cloud/infrastructure — the identity-and-permissions counterpart to Kubernetes RBAC from Part 3, but at the cloud provider level.

Diagram

---

The Principle of Least Privilege, Applied Concretely#

This exact phrase — "principle of least privilege" — has come up repeatedly throughout this series (RBAC in Part 3, Vault policies above) because it's genuinely the single most important, most universally applicable security principle across every layer of a system.

Diagram

---

A Worked IAM Policy Example#

Before — an overly broad AWS IAM policy (a genuinely common real-world mistake):

Copy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

This grants every possible S3 action (read, write, delete, change permissions) on every bucket in the entire account — a common shortcut taken "to get things working," and a serious, unnecessary risk.

After — a least-privilege policy for the same actual use case (an application that only needs to read specific files from one specific bucket):

Copy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::checkout-app-assets/*"
    }
  ]
}

Copy
# Verify what an IAM role can actually do (AWS's IAM Policy Simulator,
# or more practically, the CLI dry-run/what-if approach)
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/checkout-app-role \
  --action-names s3:DeleteObject \
  --resource-arns arn:aws:s3:::checkout-app-assets/file.txt
# Result: implicitDeny (correctly denied — was never granted)

---

Service Accounts, Workload Identity, and Why Not to Use Long-Lived Keys#

A specific, commonly-tested best practice: avoid long-lived, static IAM access keys (like an AWS access key ID/secret pair) wherever possible, in favor of short-lived, automatically-issued credentials tied to a workload's identity.

Diagram

Copy
# On an AWS EC2 instance (or in a properly configured pod using
# IAM Roles for Service Accounts / IRSA), an application can get
# temporary credentials automatically, with NOTHING stored anywhere:
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/checkout-app-role
# Returns a temporary AccessKeyId, SecretAccessKey, SessionToken,
# and Expiration — automatically refreshed before it expires,
# by the AWS SDK, with zero developer effort

The strongest possible interview answer regarding credentials: "Wherever the platform supports it, I'd avoid static, long-lived credentials entirely in favor of workload identity mechanisms — AWS IAM roles for EC2/ECS/EKS, GCP Workload Identity, Azure Managed Identities. These are automatically issued, automatically rotated, short-lived, and never need to be stored anywhere at all — which eliminates an entire category of leak risk (there's simply nothing sitting in a config file or secrets manager to leak in the first place)."

---

Secret and Key Rotation#

Diagram

A practical distinction worth naming: manual rotation ("someone remembers to change the password every 90 days") is fragile and often skipped under deadline pressure. Automated rotation (via Vault's dynamic secrets, or AWS Secrets Manager's Lambda-based rotation) removes the human-memory dependency entirely — directly connecting back to the toil elimination principle from the SRE Fundamentals series: a manually repeated task with no lasting value is toil, and rotation is a textbook example of something that should be automated rather than relied upon as a recurring human chore.

---

Common Mistakes#

---

Worked Practice Problems#

Problem 1: A gitleaks scan finds a hardcoded database password in a commit from 8 months ago, in a public open-source repository. The current codebase no longer contains it (it was removed 3 months ago). Is this still an active risk, and what would you do?

Answer: Yes, absolutely still an active risk — anyone who cloned the repository at any point in the last 8 months has that password in their local git history, permanently, regardless of what the current HEAD contains, and a public repo means this could be an unknown, unbounded number of people. Immediate action: rotate the credential right now, treating it as compromised, without spending time first determining whether it was actually misused. Given it's a public repo, purging it from history (e.g., via git filter-repo or BFG Repo-Cleaner) is far less valuable than the rotation itself, since the data has likely already been cloned/cached elsewhere beyond your control — rotation is the actual fix; history rewriting is a secondary cleanup step.

Problem 2: Your team currently gives every application instance the exact same static database password, stored in a config file deployed to every server. Propose a better architecture, and explain the concrete security benefit.

Answer: Move to a dynamic secrets model using Vault's database secrets engine — each application instance authenticates to Vault (e.g., via its Kubernetes ServiceAccount identity) and receives its own uniquely-generated, short-lived database credential, automatically revoked after a defined lease period (e.g., 1 hour). Concrete benefits: a leaked credential now has a bounded lifetime instead of being valid indefinitely; every database action can be traced back to the specific application instance whose unique credential performed it (impossible with one shared password); and rotating credentials happens automatically and continuously rather than depending on someone remembering to do it manually.

Problem 3: An application currently authenticates to AWS using a long-lived access key pair stored as environment variables in its deployment config. It runs on EKS (managed Kubernetes on AWS). What would you change, and why is it strictly better?

Answer: Switch to IAM Roles for Service Accounts (IRSA), AWS's workload identity mechanism for EKS — the pod's Kubernetes ServiceAccount is associated with an IAM role, and the AWS SDK automatically fetches short-lived, auto-rotating temporary credentials with zero static keys stored anywhere at all. This is strictly better because there's no long-lived secret to leak in the first place (nothing sits in an environment variable or config file to be accidentally logged, committed, or exposed), the credentials automatically expire and refresh (typically hourly), and access can be scoped per-ServiceAccount using the same least-privilege IAM policy principles covered above.

---

Summary and What's Next#

• A single leaked secret can bypass every other security control — this makes secrets management uniquely high-stakes among all the topics in this series.
• Secret scanning (Gitleaks, TruffleHog) should run both as a local pre-commit hook AND in CI, as defense in depth — catching leaks in seconds rather than days.
• Removing a secret from a git file's latest version does not remove it from history — the only real fix for a leaked secret is immediate rotation, investigation second.
• Environment variables are a common but limited way to deliver secrets — dedicated secrets managers (Vault, AWS Secrets Manager) add centralized audit trails, automated rotation, and (with Vault specifically) dynamic secrets — unique, automatically-expiring credentials per workload instance, dramatically shrinking the blast radius of any leak.
• Envelope encryption (a fast local data key, itself encrypted by a KMS-held key) is how cloud KMS systems efficiently encrypt data of any size with only one remote call.
• IAM and least privilege are the cloud-level counterpart to Kubernetes RBAC — always scope policies to exactly the actions and resources needed, never broad wildcards "to be safe."
• Workload identity mechanisms (AWS IAM roles, IRSA, GCP Workload Identity) eliminate an entire category of risk by removing the need for long-lived, storable static credentials altogether.
• Rotation should be automated, not a manually-remembered chore — exactly the toil-elimination principle from the SRE Fundamentals series applied to security operations.

Continue to Part 5 (05-cicd-pipeline-security-and-supply-chain.md) for CI/CD pipeline hardening, software supply chain security (SBOM, SLSA, artifact signing), and Infrastructure-as-Code scanning.