# AWS Cloud Architecture — Part 19: Machine Learning & AI on AWS

> **Series:** AWS Cloud Architecture (19 of 19)
> **Part 1:** `01-fundamentals-and-account-structure.md` — Fundamentals & Account Structure
> **Part 2:** `02-iam-and-identity.md` — IAM & Identity
> **Part 3:** `03-compute-ec2-and-autoscaling.md` — Compute: EC2 & Auto Scaling
> **Part 4:** `04-networking-vpc-deep-dive.md` — Networking: VPC Deep Dive
> **Part 5:** `05-storage-s3-ebs-efs.md` — Storage: S3, EBS & EFS
> **Part 6:** `06-managed-databases-and-data-services.md` — Managed Databases & Data Services
> **Part 7:** `07-containers-and-serverless.md` — Containers & Serverless
> **Part 8:** `08-load-balancing-cdn-and-dns.md` — Load Balancing, CDN & DNS
> **Part 9:** `09-security-and-compliance.md` — Security & Compliance
> **Part 10:** `10-monitoring-logging-and-tracing.md` — Monitoring, Logging & Tracing
> **Part 11:** `11-cicd-iac-and-messaging.md` — CI/CD, IaC & Messaging
> **Part 12:** `12-multi-region-dr-migration-and-cheatsheet.md` — Multi-Region, DR, Migration & Cheat Sheet
> **Part 13:** `13-api-gateway-integration-and-identity.md` — API Gateway, Cognito & Event Integration
> **Part 14:** `14-beanstalk-sam-and-developer-tooling.md` — Elastic Beanstalk, SAM & Developer Tooling
> **Part 15:** `15-systems-manager-and-fleet-operations.md` — Systems Manager & Fleet Operations
> **Part 16:** `16-cost-optimization-and-finops.md` — Cost Optimization & FinOps
> **Part 17:** `17-migration-and-modernization.md` — Migration & Modernization
> **Part 18:** `18-data-analytics-and-engineering.md` — Data Analytics & Engineering
> **Part 19:** This file — Machine Learning & AI on AWS
> **Questions:** `questions.md`

Assumes you're comfortable with the full data platform from Part 18 (especially the Glue Data Catalog and
S3-based data lake), IAM (Part 2), and containers (Part 7) — SageMaker leans on all three directly, and
this part is the final one in the series, completing coverage across every Associate and Professional
certification this series set out to address.

## Table of Contents

1. [Why This Part Exists](#why-this-part-exists)
2. [Two Distinct Problems: Custom ML and Pre-Built AI](#two-distinct-problems-custom-ml-and-pre-built-ai)
3. [SageMaker, the Big Picture](#sagemaker-the-big-picture)
4. [SageMaker Unified Studio](#sagemaker-unified-studio)
5. [Data Preparation: SageMaker Data Wrangler and Processing Jobs](#data-preparation-sagemaker-data-wrangler-and-processing-jobs)
6. [SageMaker Feature Store, in Depth](#sagemaker-feature-store-in-depth)
7. [Training a Model: Estimators and Training Jobs](#training-a-model-estimators-and-training-jobs)
8. [Built-In Algorithms vs Bring-Your-Own](#built-in-algorithms-vs-bring-your-own)
9. [Distributed Training and Specialized Hardware](#distributed-training-and-specialized-hardware)
10. [Automatic Model Tuning (Hyperparameter Optimization)](#automatic-model-tuning-hyperparameter-optimization)
11. [SageMaker Notebooks and Studio Environments](#sagemaker-notebooks-and-studio-environments)
12. [Deploying a Model: The Four Inference Options](#deploying-a-model-the-four-inference-options)
13. [Model Evaluation Metrics](#model-evaluation-metrics)
14. [Data Splitting and Avoiding Leakage](#data-splitting-and-avoiding-leakage)
15. [Inference Components — Multi-Model Endpoints](#inference-components--multi-model-endpoints)
16. [SageMaker Pipelines — MLOps as Code](#sagemaker-pipelines--mlops-as-code)
17. [The Model Registry](#the-model-registry)
18. [Model Monitor — Detecting Drift](#model-monitor--detecting-drift)
19. [A Full ML Lifecycle, Visually](#a-full-ml-lifecycle-visually)
20. [Amazon Bedrock — Foundation Models as a Service](#amazon-bedrock--foundation-models-as-a-service)
21. [Bedrock Knowledge Bases — Retrieval-Augmented Generation](#bedrock-knowledge-bases--retrieval-augmented-generation)
22. [Bedrock Agents](#bedrock-agents)
23. [Bedrock Guardrails](#bedrock-guardrails)
24. [SageMaker JumpStart](#sagemaker-jumpstart)
25. [Fine-Tuning Foundation Models](#fine-tuning-foundation-models)
26. [Choosing Between Bedrock and SageMaker](#choosing-between-bedrock-and-sagemaker)
27. [Bedrock Model Evaluation](#bedrock-model-evaluation)
28. [Prompt Engineering Fundamentals](#prompt-engineering-fundamentals)
29. [Vector Databases Beyond OpenSearch](#vector-databases-beyond-opensearch)
30. [Pre-Built AI Services: Vision](#pre-built-ai-services-vision)
31. [Pre-Built AI Services: Language](#pre-built-ai-services-language)
32. [Pre-Built AI Services: Search and Forecasting](#pre-built-ai-services-search-and-forecasting)
33. [Choosing Between a Pre-Built AI Service, Bedrock, and Custom SageMaker](#choosing-between-a-pre-built-ai-service-bedrock-and-custom-sagemaker)
34. [Multi-Model and A/B Testing in Production](#multi-model-and-ab-testing-in-production)
35. [Shadow Testing](#shadow-testing)
36. [Security for ML Workloads](#security-for-ml-workloads)
37. [Cost Optimization for ML Workloads](#cost-optimization-for-ml-workloads)
38. [Cross-Series Callback: Everything ML Sits On Top Of](#cross-series-callback-everything-ml-sits-on-top-of)
39. [Responsible AI: Bias, Explainability, and Governance](#responsible-ai-bias-explainability-and-governance)
40. [A Currency Note: MLA-C01's Transition](#a-currency-note-mla-c01s-transition)
41. [AgentCore and the Frontier of Agentic AI](#agentcore-and-the-frontier-of-agentic-ai)
42. [A Full Worked Example: Adding AI to the Order-Status Platform](#a-full-worked-example-adding-ai-to-the-order-status-platform)
43. [A Closing Thought on Escalation Discipline](#a-closing-thought-on-escalation-discipline)
44. [ML/AI Best Practices — The Consolidated Checklist](#mlai-best-practices--the-consolidated-checklist)
45. [Part 19 CLI Cheat Sheet](#part-19-cli-cheat-sheet)
46. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
47. [Worked Practice Problems](#worked-practice-problems)
48. [Where to Go From Here](#where-to-go-from-here)
49. [Series Summary — Nineteen Parts, One Coherent Picture](#series-summary--nineteen-parts-one-coherent-picture)

---

## Why This Part Exists

Every part of this series has been building toward the same underlying skill: recognizing which specific
AWS service answers a given requirement, and why. This final part applies that same discipline to ML/AI —
squarely MLA-C01's exam scope, and the newest, fastest-moving corner of the AWS service catalog covered
across this whole series. The throughline worth holding onto: **most real ML/AI requirements don't need a
custom-trained model at all** — a huge fraction of MLA-C01 (and real production decisions) is choosing the
least custom, least operationally expensive option that actually meets the requirement, escalating toward
custom SageMaker training only when a pre-built service or a foundation model genuinely can't do the job.

## Two Distinct Problems: Custom ML and Pre-Built AI

Before touching a single service, separate two categorically different needs this part covers:
**custom machine learning** — training a model on an organization's own data for a problem specific enough
that no pre-built service solves it (SageMaker's whole domain) — and **pre-built AI** — calling a managed
API that already solves a common, well-understood problem (detecting objects in an image, extracting text
from a document, translating a sentence) without training anything at all. A genuinely common, costly
mistake: standing up custom SageMaker training for a problem Rekognition or Textract already solves out of
the box.

## SageMaker, the Big Picture

**SageMaker** (formally **Amazon SageMaker AI** as of the platform's 2026 restructuring, distinguishing
the ML-building toolkit from the broader SageMaker Unified Studio surface below) spans the full ML
lifecycle: data preparation, training, tuning, deployment, and ongoing monitoring — a managed
infrastructure layer removing the undifferentiated heavy lifting of provisioning GPU clusters, managing
training job orchestration, and building deployment infrastructure by hand, the same value proposition
every managed service in this series has offered relative to its self-hosted equivalent.

```mermaid
flowchart LR
    Prep["Prepare<br/>Data Wrangler,<br/>Processing Jobs,<br/>Feature Store"] --> Train["Train<br/>Estimators,<br/>built-in algos,<br/>distributed training"]
    Train --> Tune["Tune<br/>Automatic Model<br/>Tuning"]
    Tune --> Deploy["Deploy<br/>Real-time, Serverless,<br/>Async, Batch"]
    Deploy --> Monitor["Monitor<br/>Model Monitor,<br/>drift detection"]
    Monitor -.->|"drift detected"| Train

    classDef stage fill:#1d4ed8,stroke:#1e3a8a,color:#ffffff
    classDef loop fill:#7c3aed,stroke:#4c1d95,color:#ffffff
    class Prep,Train,Tune,Deploy stage
    class Monitor loop
```

## SageMaker Unified Studio

**SageMaker Unified Studio** is AWS's 2026 convergence of previously separate interfaces (SageMaker
Studio, parts of the Glue/Redshift/EMR consoles) into one governed workspace — a data engineer building a
Part 18 Glue pipeline and a data scientist training a model can now work inside the same project, with
Iceberg-based **SageMaker Lakehouse** integration letting a training job stream data directly from sources
like Redshift without a separate export step. This is worth recognizing specifically as the "why do these
data-platform and ML services feel connected now" answer — Part 18's data platform isn't a separate system
a model consumes at arm's length; it's increasingly the same governed workspace.

## Data Preparation: SageMaker Data Wrangler and Processing Jobs

**SageMaker Data Wrangler** provides a visual interface for ML-specific data preparation (handling
missing values, encoding categorical variables, balancing an imbalanced dataset) — the ML-focused sibling
of Part 18's Glue DataBrew, aimed at a data scientist preparing features rather than a data engineer
cleaning general-purpose data. **Processing Jobs** run arbitrary preprocessing/postprocessing code
(feature engineering, evaluation) on managed infrastructure that spins up for the job and terminates
after — the SageMaker-native equivalent of a Glue job, scoped specifically to the ML pipeline's own
pre/post-training steps.

## SageMaker Feature Store, in Depth

Part 18 flagged Feature Store as the boundary between the data platform and ML. In full: Feature Store
maintains both an **online store** (low-latency lookups for real-time inference — a model scoring a live
request needs a customer's current feature values in milliseconds) and an **offline store** (a full
historical record in S3, used for training and batch scoring), keeping both in sync automatically. This
solves the exact training-serving skew problem named in Part 18 — a feature computed one way during
training and a subtly different way during inference is a genuinely common, hard-to-debug source of model
performance degradation in production, and Feature Store's single computation path for both use cases
removes that entire failure class structurally.

## Training a Model: Estimators and Training Jobs

A SageMaker **training job** provisions the requested compute (CPU or GPU instances, sized to the
workload), runs training code against data pulled from S3, writes the trained model artifact back to S3,
and tears the compute down automatically once training completes — meaning a team never pays for idle GPU
capacity between training runs, the same "no idle infrastructure" economics as every serverless service in
this series, applied to what would traditionally be a persistent, expensive GPU cluster. An **Estimator**
(in the SageMaker SDK) is the code-level abstraction wrapping a training job's configuration — algorithm,
instance type/count, hyperparameters — regardless of whether the underlying algorithm is a SageMaker
built-in, a supported framework (PyTorch, TensorFlow, scikit-learn), or fully custom code in a Docker
container.

## Built-In Algorithms vs Bring-Your-Own

| Option | What it means | Fits |
|---|---|---|
| **SageMaker built-in algorithms** | Pre-implemented, optimized algorithms (XGBoost, Linear Learner, Image Classification) | Common, well-understood problem shapes — fastest path to a working model |
| **Framework containers** | AWS-maintained containers for PyTorch, TensorFlow, scikit-learn, etc. | Custom model code, standard framework, no custom container to build |
| **Bring-your-own container** | A fully custom Docker image implementing SageMaker's training/inference contract | A genuinely novel algorithm, an unsupported framework, or specific dependency requirements |

This is a strict escalation ladder — reach for a built-in algorithm first, a framework container when the
problem needs custom model code in a standard framework, and a fully custom container only when neither
of the first two options actually fits, since each step up trades faster time-to-working-model for more
operational ownership.

## Distributed Training and Specialized Hardware

For models too large or datasets too big for a single instance, SageMaker supports **distributed
training** — data parallelism (splitting the dataset across instances, each holding a full model copy) and
model parallelism (splitting the model itself across instances, for models too large to fit on one). For
specialized hardware, **AWS Trainium** (AWS's own purpose-built ML training silicon) offers a meaningfully
better price/performance ratio than comparable GPU instances for supported workloads — the same
Graviton-style "AWS's own silicon beats general-purpose hardware on price/performance for the workloads it
targets" pattern Part 16 already established for general compute, applied here to ML training specifically.

## Automatic Model Tuning (Hyperparameter Optimization)

**Automatic Model Tuning** runs many training jobs across a defined hyperparameter search space (each
parameter set as continuous, integer, or categorical), using **Bayesian optimization** by default (each
successive job informed by prior results, converging toward good hyperparameters faster than exhaustive
grid search) or grid search when the space is small enough to search exhaustively. **Early stopping**
terminates a clearly underperforming training job before it finishes, avoiding wasted compute spend on
runs that were never going to produce a competitive result — a direct, automatic cost lever built into the
tuning process itself rather than something a team has to remember to configure separately.

## SageMaker Notebooks and Studio Environments

Before training anything at scale, a data scientist typically works interactively — **SageMaker Studio**
(the IDE-style notebook environment, now part of Unified Studio above) provides managed Jupyter notebooks
with direct access to the same training/deployment infrastructure, so moving from "experimenting in a
notebook" to "running a real training job" doesn't require re-provisioning environments or copying code
between systems. **Local Mode** lets a training/inference container run on the notebook instance itself
first, for fast iteration on code correctness before submitting a real, billed training job against a
larger dataset — the ML-specific equivalent of Part 14's `sam local` fast-iteration loop, catching an
obvious bug before it costs real training compute time.

## Deploying a Model: The Four Inference Options

| Option | Latency | Scales to zero? | Fits |
|---|---|---|---|
| **Real-time endpoint** | Sub-second, persistent | No | A live application needing consistent, low-latency predictions |
| **Serverless inference** | Sub-second after a cold start | Yes | Intermittent traffic where paying for idle endpoint capacity isn't justified |
| **Async inference** | Seconds to minutes, queued | Yes (scales down between requests) | Large payloads or longer processing time, with the caller polling or receiving a callback |
| **Batch transform** | No live endpoint at all — offline job | N/A | Scoring an entire dataset at once, no real-time requirement |

This maps directly onto the exact same Lambda-vs-Fargate-vs-batch decision framework Part 7 already
established for general compute — real-time endpoints are SageMaker's "always-on" option, serverless
inference is its Lambda-equivalent, and batch transform is its scheduled-batch-job equivalent, just applied
to model inference specifically instead of general application logic.

## Model Evaluation Metrics

Choosing the right evaluation metric is a genuinely testable MLA-C01 topic, since the wrong metric can make
a bad model look good. For classification: **accuracy** (overall correct-prediction rate) is misleading on
an imbalanced dataset — a fraud model predicting "not fraud" every time can still score 99% accuracy on a
1%-fraud dataset while being completely useless; **precision** (of predicted positives, how many were
actually positive) and **recall** (of actual positives, how many were caught) trade off against each other
and matter differently by use case — a medical screening model favors recall (missing a real case is worse
than a false alarm), a spam filter favors precision (wrongly blocking real email is worse than missing some
spam); **F1 score** balances the two; and **AUC-ROC** evaluates a classifier's ranking quality across every
possible threshold rather than one fixed cutoff. For regression: **RMSE** and **MAE** measure prediction
error magnitude, with RMSE penalizing large errors more heavily than MAE.

## Data Splitting and Avoiding Leakage

Standard practice splits data into **training** (fitting the model), **validation** (tuning
hyperparameters and comparing candidate models during development), and **test** (a final, untouched
evaluation of the chosen model, used exactly once to estimate real-world performance) sets — reusing the
test set for any tuning decision quietly turns it into another validation set, producing an overly
optimistic performance estimate that won't hold up in production. **Data leakage** — information from
outside the training set (often, inadvertently, from the future relative to what the model would know at
prediction time) leaking into training features — is a related, insidious failure mode: a model that looks
excellent in evaluation but performs far worse in production is a strong signal to check for leakage
specifically, often before looking anywhere else.

## Inference Components — Multi-Model Endpoints

**Inference Components**, SageMaker's current standard for endpoint efficiency, let multiple distinct
models share a single underlying GPU instance rather than each model requiring its own dedicated,
often-underutilized endpoint — dramatically improving utilization and lowering cost for an organization
serving many models with individually modest traffic. This is the SageMaker-specific instance of the same
"stop paying for idle capacity" instinct behind Fargate's per-task billing (Part 7) or Lambda's
per-invocation billing — applied to GPU inference capacity, which is meaningfully more expensive per hour
than the general-purpose compute this series has mostly discussed until now.

## SageMaker Pipelines — MLOps as Code

**SageMaker Pipelines** defines an ML workflow — data processing, training, evaluation, conditional model
registration — as a **directed acyclic graph (DAG)** of steps, expressed declaratively via the SageMaker
SDK, with each run's full lineage (which data, which code version, which resulting model) tracked
automatically. This is MLOps' equivalent of Part 11's CI/CD pipelines and Part 18's Step Functions/Glue
Workflows orchestration — the same "turn a manual, error-prone sequence of steps into a repeatable,
auditable, automated workflow" principle this series has applied to infrastructure deployment and data
pipelines, now applied to the ML training-and-deployment lifecycle itself.

## The Model Registry

The **Model Registry** catalogs trained models as versioned **Model Package Groups** — each training run
producing a new version, carrying metadata (training data, hyperparameters, evaluation metrics) and an
**approval status** (Pending, Approved, Rejected) that gates whether a version is eligible for production
deployment. This is the ML-specific analog of a container image registry (Part 7's ECR) combined with
CodeDeploy's approval gating (Part 11/15's Change Manager) — a deliberate checkpoint between "a model
finished training" and "a model is actually serving production traffic," rather than the two being the
same automatic event.

## Model Monitor — Detecting Drift

**Model Monitor** continuously analyzes a live endpoint's actual inputs and outputs against a captured
baseline, detecting **data drift** (the distribution of incoming requests diverging from what the model
was trained on) and **model quality drift** (the model's actual prediction accuracy degrading over time,
when ground truth becomes available to compare against). This is the ML-specific instance of the same
"detect degradation before it becomes a full outage" instinct behind every alarm and health check this
series has covered — a model silently getting worse at its job is a real production incident, just one
without an obvious error rate or latency spike to alert on directly, which is exactly why Model Monitor
exists as a distinct, purpose-built capability rather than something ordinary CloudWatch alarms alone would
catch.

## A Full ML Lifecycle, Visually

```mermaid
sequenceDiagram
    participant DS as Data Scientist
    participant SM as SageMaker Pipelines
    participant Train as Training Job
    participant Reg as Model Registry
    participant EP as Endpoint
    participant Mon as Model Monitor

    DS->>SM: Trigger pipeline (new data or code)
    SM->>Train: Run processing + training steps
    Train-->>SM: Trained model artifact + metrics
    SM->>Reg: Register new model version (Pending)
    Reg->>Reg: Approval (manual or automated gate)
    Reg->>EP: Deploy approved version
    EP-->>Mon: Live inference traffic
    Mon->>Mon: Compare against baseline
    Mon-->>DS: Drift detected alert
    DS->>SM: Trigger retraining pipeline
```

## Amazon Bedrock — Foundation Models as a Service

**Bedrock** provides API access to foundation models from multiple providers (Anthropic, Meta, Mistral,
Cohere, Amazon's own Titan/Nova, and others via Bedrock Marketplace) without provisioning or managing any
underlying infrastructure — the generative-AI-specific instance of "consume a managed API instead of
running your own infrastructure" this whole series has repeated across every domain. Bedrock has grown well
past a simple model-invocation proxy into a fuller platform: its own RAG capability (Knowledge Bases,
below), an agent runtime (Agents, below), a safety layer (Guardrails, below), evaluation tooling, and — as
of 2026 — **AgentCore**, a separate microVM-based runtime specifically for running more complex, longer-
lived agentic workloads beyond what the core Agents capability targets.

## Bedrock Knowledge Bases — Retrieval-Augmented Generation

**Knowledge Bases** implements **RAG (Retrieval-Augmented Generation)** as a fully managed capability:
ingesting an organization's own documents, chunking them, generating embeddings, storing them in a vector
store (OpenSearch Serverless or a supported alternative), and — at query time — retrieving the most
relevant chunks to inject into the foundation model's prompt before generating a response. This is the
concrete mechanism behind "a chatbot that answers questions using our own internal documentation" without
fine-tuning a model at all — the model itself stays generic; the organization's specific knowledge is
supplied as retrieved context on every request instead.

## Bedrock Agents

**Bedrock Agents** layers reasoning and multi-step task execution on top of a chosen foundation model —
an agent can call **action groups** (Lambda functions, Part 13's whole toolkit) to take real actions
(look up an order status, call an internal API) and retrieve from a Knowledge Base, turning a
single-turn chat model into something that can plan and execute a multi-step task autonomously. This is
Bedrock's low-code answer to "an LLM that can actually do things, not just answer questions" — the
foundation model handles reasoning about what to do next; action groups are the concrete mechanism
connecting that reasoning to real AWS resources and external systems.

## Bedrock Guardrails

**Guardrails** provides a configurable safety layer applied to both the prompt going into a model and the
response coming out — filtering harmful content, blocking specific denied topics, redacting PII, and
checking for factual grounding against provided context — attachable to Agents and Knowledge Bases
directly, or invoked standalone against any model call. This is the generative-AI-specific instance of the
same defense-in-depth instinct Part 9 established for general security: a foundation model's own training
provides some inherent safety behavior, but Guardrails adds an explicit, configurable, auditable
enforcement layer on top, rather than relying on the model's implicit behavior alone.

## SageMaker JumpStart

**JumpStart** is a hub of pre-trained, open-source foundation and task-specific models (many of the same
model families available through Bedrock, plus additional open-source options), deployable directly to a
SageMaker endpoint or fine-tunable against an organization's own data with a few clicks. Where Bedrock
offers foundation models as a fully managed, no-infrastructure API call, JumpStart deploys those (or
similar) models onto **infrastructure the organization controls directly** — the right reach when a team
needs more control over the serving infrastructure, model customization depth, or data residency than
Bedrock's fully-managed model provides, at the cost of taking on the infrastructure operational ownership
Bedrock otherwise removes entirely.

## Fine-Tuning Foundation Models

Beyond RAG (supplying context at query time, no model change), both Bedrock and SageMaker/JumpStart support
genuine **fine-tuning** — further training a foundation model on an organization's own labeled examples,
adjusting the model's actual weights rather than just its input context. This is a heavier, more
expensive, more operationally involved option than RAG, appropriate specifically when a model needs to
adopt a particular *style*, *format*, or *specialized behavior* consistently (not just recall specific
facts, which RAG already handles well) — the practical rule of thumb worth internalizing: reach for RAG
first for "the model needs to know about our data," and reserve fine-tuning for "the model needs to behave
differently," since RAG is cheaper, faster to iterate on, and easier to keep current as underlying data
changes.

## Choosing Between Bedrock and SageMaker

| | Bedrock | SageMaker |
|---|---|---|
| Training | No custom training — consume existing foundation models | Full custom training on your own data |
| Best fit | Generative AI (text, chat, RAG, agents) using an existing model | A custom prediction problem (fraud scoring, demand forecasting) needing a model trained on your own labeled data |
| Time to first result | Minutes — call an API | Days to weeks — data prep, training, tuning |
| Operational ownership | Minimal — AWS manages the model | Full ML lifecycle ownership |

A genuinely common real pattern: Bedrock for the generative/conversational surface of an application, and
SageMaker for a specific, custom predictive problem (say, fraud scoring) feeding structured decisions into
that same application — the two aren't mutually exclusive, and many real architectures use both for
different parts of the same product.

## Bedrock Model Evaluation

Choosing among Bedrock's many available foundation models for a specific use case is itself a real
decision worth tooling, not guessing at — **Bedrock Model Evaluation** runs a candidate set of models
against a defined dataset and task, scoring them on relevant metrics (accuracy, robustness, toxicity, or a
custom metric) using either automatic scoring or human reviewers, producing a comparable basis for
selecting a model rather than picking one based on general reputation alone. This matters concretely
because model choice is a genuine cost/quality/latency tradeoff — a larger, more capable model costs more
per token and responds more slowly than a smaller one, and the right choice depends entirely on whether
the specific task actually needs that extra capability.

## Prompt Engineering Fundamentals

Worth a brief, concrete treatment since it's directly testable: effective prompting for a foundation model
generally benefits from a clear **system prompt** (establishing the model's role and constraints once, up
front), **few-shot examples** (showing the model a couple of example input/output pairs directly in the
prompt, which measurably improves output consistency for many tasks without any fine-tuning at all), and
explicit **output format instructions** (asking for JSON, a specific structure) when the response needs to
be machine-parsed downstream rather than just read by a human. **Temperature** and **top-p** are the two
most commonly tuned inference parameters — lower temperature produces more deterministic, focused output
(appropriate for factual/extraction tasks); higher temperature produces more varied, creative output
(appropriate for brainstorming or creative-writing tasks) — the same "no single fixed setting is right for
every task" logic that runs through this entire part.

## Vector Databases Beyond OpenSearch

Bedrock Knowledge Bases defaults to OpenSearch Serverless as its vector store, but it's worth knowing the
option isn't exclusive: **Aurora PostgreSQL with the `pgvector` extension** (Part 6) and **Amazon
MemoryDB** both support vector similarity search and are supported alternative Knowledge Bases backends —
a genuinely relevant choice for a team that already runs Aurora and would rather extend an existing,
familiar database than stand up a separate OpenSearch Serverless collection purely for vector storage. The
underlying capability (storing embeddings, finding nearest-neighbor matches) is the same regardless of
which store is chosen; the decision is really about which operational surface a team already knows and
wants to consolidate onto.

## Pre-Built AI Services: Vision

**Amazon Rekognition** analyzes images and video — object/scene detection, facial analysis, content
moderation (flagging unsafe content), and text-in-image detection — as a direct API call, no training
required for its standard capabilities (custom object detection for a domain-specific need is also
supported, but distinct from the zero-training default use). **Amazon Textract** extracts text and
structured data (forms, tables) from documents — meaningfully more capable than plain OCR, since it
understands document structure (which text belongs to which form field) rather than returning an
undifferentiated block of extracted text.

## Pre-Built AI Services: Language

**Amazon Comprehend** extracts meaning from text — sentiment, named entities, key phrases, PII detection —
answering "what does this text mean" rather than merely "what does this text say." **Amazon Translate**
provides real-time and batch language translation. **Amazon Transcribe** converts speech to text
(including streaming, real-time transcription). **Amazon Polly** does the reverse — text to natural-
sounding speech. Together, these cover the standard language-AI surface without requiring any of them to
be trained on an organization's own data first.

## Pre-Built AI Services: Search and Forecasting

**Amazon Kendra** provides intelligent, natural-language enterprise search across an organization's
documents — conceptually adjacent to Bedrock Knowledge Bases' retrieval step, but as a standalone search
product rather than a RAG pipeline feeding a generative model. **Amazon Forecast** produces time-series
forecasts (demand, inventory) using built-in algorithms tuned for forecasting specifically, without
requiring a team to build and tune a custom forecasting model in SageMaker from scratch.

## Choosing Between a Pre-Built AI Service, Bedrock, and Custom SageMaker

```mermaid
flowchart TD
    Start{"Does a pre-built<br/>AI service solve<br/>this exact problem?"}
    Start -->|"Yes — image/video,<br/>text, search, forecast"| PreBuilt["Use Rekognition,<br/>Textract, Comprehend,<br/>Kendra, or Forecast"]
    Start -->|"No, but it's a<br/>generative/conversational<br/>problem"| BedrockCheck{"Existing foundation<br/>model + prompt/RAG<br/>sufficient?"}
    BedrockCheck -->|"Yes"| Bedrock["Use Bedrock<br/>(model + Knowledge Base + Agent)"]
    BedrockCheck -->|"No — needs<br/>domain-specific<br/>fine-tuning"| SageMakerFT["SageMaker: fine-tune<br/>a foundation model"]
    Start -->|"No — a custom<br/>prediction problem"| SageMakerCustom["SageMaker: train a<br/>custom model from scratch"]

    classDef decision fill:#7c3aed,stroke:#4c1d95,color:#ffffff
    classDef prebuilt fill:#0f766e,stroke:#134e4a,color:#ffffff
    classDef bedrock fill:#1d4ed8,stroke:#1e3a8a,color:#ffffff
    classDef sagemaker fill:#b45309,stroke:#78350f,color:#ffffff
    class Start,BedrockCheck decision
    class PreBuilt prebuilt
    class Bedrock bedrock
    class SageMakerFT,SageMakerCustom sagemaker
```

This decision tree is the single most exam-relevant and real-world-relevant judgment call in this entire
part — reaching for custom SageMaker training when a pre-built service or Bedrock would have solved the
problem faster, cheaper, and with less ongoing operational ownership is the single most common
architectural overreach in ML/AI projects.

## Multi-Model and A/B Testing in Production

Deploying a new model version rarely means an instant full cutover, for exactly the same blast-radius
reasons Part 14 established for application deployments. A SageMaker endpoint supports **production
variants** — multiple model versions behind the same endpoint, with configurable traffic-weight splitting
between them — enabling **A/B testing** (comparing a challenger model's real-world performance against the
current production model on live traffic before fully committing) or a gradual, canary-style rollout of a
new version. This is the direct ML-specific application of Part 14's Lambda alias weighting and Part 7's
ECS traffic-shifting patterns, just expressed through SageMaker's own endpoint variant mechanism instead.

## Shadow Testing

A related, distinct pattern worth naming separately: **shadow testing** (sometimes called shadow
deployment) runs a challenger model against real production traffic **in parallel** with the current
model, logging the challenger's predictions for offline comparison, without ever actually serving the
challenger's output to a real user or system. This is a strictly lower-risk validation step than A/B
testing, appropriate when a team wants real-traffic validation before taking on any risk of the challenger
model's output actually reaching production decisions at all — the ML equivalent of MGN's test-launch
capability from Part 17, validating against reality without any production impact.

## Security for ML Workloads

Every security pattern this series has established applies directly: IAM least privilege for training job
execution roles and inference endpoint invocation permissions (Part 2), VPC-isolated training jobs and
endpoints for sensitive data (Part 4's private subnet pattern, directly supported by SageMaker), encryption
at rest for training data and model artifacts via KMS (Part 9), and CloudTrail logging every SageMaker/
Bedrock API call for audit. Bedrock Guardrails, above, adds a generative-AI-specific layer on top of this
same foundation, not a replacement for it — a Bedrock application still needs the same IAM/VPC/KMS
discipline as everything else in this series, plus Guardrails for content-specific safety.

## Cost Optimization for ML Workloads

Part 16's toolkit applies directly, with ML-specific levers worth naming: **Managed Spot Training** runs
training jobs on Spot capacity (Part 3) for fault-tolerant training workloads, at meaningfully lower cost
than On-Demand, with SageMaker handling checkpoint/resume automatically around interruptions; **serverless
inference** avoids paying for an idle real-time endpoint during low-traffic periods; **Inference
Components**, above, improve GPU utilization directly; and choosing the right inference option (real-time
vs serverless vs batch) from the four-option table above is itself the single biggest ML-specific cost
lever, the same "match the compute shape to the actual traffic pattern" discipline Part 7 established for
general application compute.

## Cross-Series Callback: Everything ML Sits On Top Of

Worth naming explicitly, since it's easy to experience this part as an entirely separate island: SageMaker
training jobs run on the same EC2/managed-compute foundation from Part 3, secured by the same IAM roles
and VPC isolation from Parts 2 and 4, storing artifacts in the same S3 covered in Part 5, monitored via
the same CloudWatch stack from Part 10, and deployed with the same cost-conscious, right-sized-instance
discipline from Part 16. None of this part's services are a fundamentally different kind of AWS resource —
they're the exact same building blocks this entire series has already covered, applied to the specific
problem of training and serving models rather than running general application workloads.

## Responsible AI: Bias, Explainability, and Governance

**SageMaker Clarify** detects bias in training data and model predictions (a model performing meaningfully
worse for one demographic group than another, for instance) and provides explainability reports (which
features most influenced a specific prediction) — increasingly a genuine compliance requirement, not just
a best practice, for models used in regulated decisions (lending, hiring, insurance). This is worth naming
explicitly as its own category: a model can be technically accurate in aggregate while still being unfair
or non-compliant in ways that only surface through deliberate bias analysis, not through ordinary accuracy
metrics alone.

## A Currency Note: MLA-C01's Transition

Worth stating for anyone using this part to prepare for the specific exam: as of this writing (August
2026), **registration for MLA-C02 opens September 1, 2026**, with the last day to sit MLA-C01 in English on
September 28, 2026 — meaning this part's content, current as of today, sits right at that transition point.
The underlying AWS services and architectural decisions covered here don't change with the exam version
number; only the specific task-statement wording and weightings might shift in C02. Confirm which version
is actually being administered before a real exam attempt, the same currency-checking discipline this
whole series has applied to every other service.

## AgentCore and the Frontier of Agentic AI

Worth a forward-looking mention, since it represents where this space is actively moving as of 2026:
**AgentCore** is a separate, microVM-based runtime specifically for agentic workloads that outgrow Bedrock
Agents' core capability — longer-running, more complex multi-step tasks needing stronger isolation between
concurrent agent sessions than a shared, lighter-weight agent runtime provides. The pattern worth
recognizing rather than memorizing exhaustively: as agentic AI use cases mature past simple single-purpose
assistants toward genuinely autonomous, longer-lived task execution, AWS is building dedicated
infrastructure for that specific shape of workload — the same evolutionary pattern this series has already
seen play out for containers (Part 7's progression from EC2 to ECS to Fargate) and migration tooling (Part
17's evolution from plain MGN to the fuller AWS Transform platform).

## A Full Worked Example: Adding AI to the Order-Status Platform

Extending the order-status platform from Parts 13-16 with AI capabilities:

1. **Pre-built first**: Customer support tickets (free-text) get sentiment and key-phrase analysis via
   Comprehend, flagging genuinely upset customers for priority routing — no custom training needed at all,
   solving the problem with a direct API call.
2. **Document processing**: Return-request photos and receipts uploaded by customers get processed through
   Textract, extracting order numbers and amounts automatically rather than requiring manual data entry —
   again, no training required.
3. **Generative support**: A Bedrock Knowledge Base, built from the product catalog and FAQ documentation
   already living in the Part 18 data lake, powers a customer-facing chat assistant answering product
   questions — with Guardrails configured to block off-topic requests and redact any accidentally-shared
   PII from responses.
4. **Custom prediction**: A genuinely custom problem — predicting which orders are likely to result in a
   return, based on the organization's own historical order data — doesn't fit any pre-built service, so a
   SageMaker XGBoost model is trained on curated features from the Part 18 Feature Store, tuned via
   Automatic Model Tuning, and deployed as a serverless inference endpoint (traffic is intermittent,
   scoring only happens at order-confirmation time).
5. **MLOps**: The return-prediction model runs through SageMaker Pipelines, registering each new version in
   the Model Registry with a manual approval gate before production deployment, and Model Monitor watches
   the live endpoint for drift as the organization's product mix changes over a season.
6. **Governance**: SageMaker Clarify's bias report on the return-prediction model confirms it isn't
   inadvertently correlating with a protected customer attribute before it's approved for production use.
7. **Safe rollout**: A new challenger version of the return-prediction model runs in shadow mode against
   real order traffic for two weeks, logging predictions without affecting any real decision, before an
   A/B split gradually shifts a fraction of live traffic to it via production variants.
8. **Model selection**: Before committing to a specific foundation model for the chat assistant, Bedrock
   Model Evaluation compares three candidate models against a curated set of real customer questions,
   balancing response quality against per-token cost and latency for the platform's actual traffic volume.
9. **Cost discipline**: Training runs for the return-prediction model use Managed Spot Training, and the
   chat assistant's Knowledge Base vector store reuses the platform's existing Aurora PostgreSQL cluster
   with `pgvector` rather than standing up a separate OpenSearch Serverless collection, consolidating onto
   infrastructure the team already operates and monitors.

## A Closing Thought on Escalation Discipline

The decision tree from earlier in this part — pre-built service, then Bedrock, then custom SageMaker only
as a last resort — is worth internalizing as a general habit, not just an ML-specific one. It's the same
escalation discipline this entire series has modeled repeatedly: reach for a managed service before
self-hosting (Part 6's RDS over self-managed databases), reach for Serverless before Provisioned (Part 16's
cost framework), reach for an existing pattern before inventing a new one (this rule's own pattern-
consistency principle). ML/AI is simply the newest domain where that same discipline applies, and very
often the domain where skipping it — jumping straight to custom training out of excitement for the
technology — is most tempting and most costly to walk back later.

## ML/AI Best Practices — The Consolidated Checklist

- Check whether a pre-built AI service (Rekognition, Textract, Comprehend, Kendra, Forecast) already
  solves the problem before reaching for custom SageMaker training.
- Check whether Bedrock (an existing foundation model, optionally with RAG via Knowledge Bases) solves a
  generative/conversational need before fine-tuning or training a custom model.
- Use Feature Store's online/offline store pair to eliminate training-serving skew, rather than computing
  features separately for training and inference.
- Match the inference deployment option (real-time, serverless, async, batch) to the actual traffic
  pattern — this is the single biggest ML-specific cost lever.
- Gate production model deployment behind Model Registry approval, not automatic deployment straight from
  a training job.
- Run Model Monitor against every production endpoint — a silently degrading model is a real incident
  without an obvious error-rate signal to alert on otherwise.
- Attach Bedrock Guardrails to any customer-facing generative AI surface, not just relying on a model's
  implicit safety behavior.
- Run SageMaker Clarify bias/explainability analysis on any model feeding a regulated or high-stakes
  decision before production approval.
- Use Managed Spot Training for fault-tolerant training workloads to reduce training cost meaningfully.
- Reserve the test set for one final, untouched evaluation — never reuse it for tuning decisions, or it
  quietly becomes another validation set with an overly optimistic performance estimate.
- Choose an evaluation metric (precision, recall, F1, AUC) that actually reflects the use case's real
  cost of a false positive vs a false negative, not accuracy by default on an imbalanced dataset.
- Default to RAG (Knowledge Bases) before fine-tuning a foundation model — RAG is cheaper, faster to
  iterate, and easier to keep current as underlying data changes.
- Validate a challenger model via shadow testing or a gradual A/B rollout before a full production cutover.
- Run Bedrock Model Evaluation before committing to a specific foundation model, rather than choosing by
  general reputation alone.
- Confirm which exam version (MLA-C01 vs its successor) is actually being administered before relying on
  exact task-statement wording from any single source, including this chapter.
- Reach for AgentCore only once a genuinely long-running, complex agentic workload outgrows Bedrock
  Agents' core capability — not as a default starting point for a simple assistant.

## Part 19 CLI Cheat Sheet

| Task | Command |
|---|---|
| Start a SageMaker training job | `aws sagemaker create-training-job --training-job-name <name> --algorithm-specification <spec-json> --role-arn <arn> --input-data-config <config-json> --output-data-config <config-json> --resource-config <config-json> --stopping-condition MaxRuntimeInSeconds=3600` |
| Start a hyperparameter tuning job | `aws sagemaker create-hyper-parameter-tuning-job --hyper-parameter-tuning-job-name <name> --hyper-parameter-tuning-job-config <config-json> --training-job-definition <def-json>` |
| Create a real-time endpoint | `aws sagemaker create-endpoint --endpoint-name <name> --endpoint-config-name <config-name>` |
| Register a model version | `aws sagemaker create-model-package --model-package-group-name <group> --inference-specification <spec-json>` |
| Start a SageMaker pipeline execution | `aws sagemaker start-pipeline-execution --pipeline-name <name>` |
| Invoke a Bedrock model | `aws bedrock-runtime invoke-model --model-id <model-id> --body <json-payload> output.json` |
| Create a Bedrock Knowledge Base | `aws bedrock-agent create-knowledge-base --name <name> --role-arn <arn> --knowledge-base-configuration <config-json>` |
| Detect entities with Comprehend | `aws comprehend detect-entities --text "sample text" --language-code en` |
| Analyze a document with Textract | `aws textract analyze-document --document '{"S3Object":{"Bucket":"bucket","Name":"doc.pdf"}}' --feature-types "FORMS"` |
| Detect labels in an image with Rekognition | `aws rekognition detect-labels --image '{"S3Object":{"Bucket":"bucket","Name":"image.jpg"}}'` |

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Training a custom SageMaker model for a problem Rekognition/Textract/Comprehend already solves | Reinvents a maintained, already-optimized capability at far higher cost and time investment | Check pre-built AI services first, every time |
| Computing features differently for training vs real-time inference | Produces training-serving skew — a common, hard-to-debug source of production model degradation | Use Feature Store's shared online/offline computation path |
| Deploying every model behind an always-on real-time endpoint regardless of traffic pattern | Pays for idle GPU capacity between requests for intermittent workloads | Match the deployment option to actual traffic — serverless/async/batch for non-constant traffic |
| Deploying a newly trained model straight to production with no approval gate | No checkpoint to catch a regression before it reaches live traffic | Gate deployment behind Model Registry approval |
| Assuming a model's initial accuracy holds indefinitely with no ongoing monitoring | Data drift and model quality drift are real, common, and silent without active monitoring | Run Model Monitor against every production endpoint |
| Treating a foundation model's built-in safety behavior as sufficient for a customer-facing application | Implicit model behavior isn't configurable, auditable, or guaranteed | Attach Bedrock Guardrails explicitly for content filtering and PII redaction |
| Judging a fraud/imbalanced-classification model purely by accuracy | A model that always predicts the majority class can score high accuracy while being useless | Use precision/recall/F1/AUC, matched to the actual cost of false positives vs false negatives |
| Reusing the test set during hyperparameter tuning | Quietly turns the test set into another validation set, producing an overly optimistic final estimate | Reserve the test set for one final, untouched evaluation only |
| Reaching for fine-tuning when the actual need is "the model should know about our documents" | Fine-tuning is slower, more expensive, and harder to keep current than RAG for that specific need | Default to Knowledge Bases/RAG; reserve fine-tuning for changing the model's behavior or style |

## Worked Practice Problems

**Problem 1:** A team needs to extract structured data (line items, totals) from scanned invoices
uploaded by customers, and initially proposes training a custom SageMaker model for this. What's the
faster, cheaper alternative, and why does it fit better?

*Answer:* Amazon Textract, specifically its forms/tables analysis capability, already extracts structured
data from documents including understanding which text belongs to which field — exactly the invoice-
extraction problem described, with zero training data, zero training time, and a direct API call. Custom
SageMaker training would require collecting and labeling a training dataset of invoices, meaningfully more
time and cost investment, for a problem a pre-built service already solves well.

**Problem 2:** A fraud-detection model's production accuracy has quietly degraded over three months, only
discovered when a manual quarterly review happened to catch it. What SageMaker capability would have
caught this automatically, and what's the underlying phenomenon it detects?

*Answer:* SageMaker Model Monitor, configured against the live endpoint from day one, would have
automatically detected either data drift (incoming transaction patterns diverging from the training
distribution) or model quality drift (actual prediction accuracy declining once ground-truth fraud
outcomes became available) well before a quarterly manual review — the entire point of continuous
monitoring is catching this kind of silent degradation without depending on a human noticing it
eventually.

**Problem 3:** A customer-facing chatbot built on Bedrock needs to answer questions using a company's
internal product documentation, without the underlying foundation model ever being retrained or
fine-tuned. What Bedrock capability provides this, and what's the mechanism?

*Answer:* Bedrock Knowledge Bases, implementing Retrieval-Augmented Generation. The mechanism: the
company's documents are chunked, embedded, and stored in a vector store; at query time, the most relevant
chunks are retrieved and injected into the foundation model's prompt as context before it generates a
response. The model itself stays generic and untouched — the organization's specific knowledge is supplied
fresh on every request rather than baked into the model's own weights.

**Problem 4:** A newly trained challenger model shows better offline evaluation metrics than the current
production model, but the team is hesitant to trust that offline improvement will hold up against real,
messy production traffic. What deployment pattern lets them validate this with zero risk to real users
before committing to any traffic shift?

*Answer:* Shadow testing. Running the challenger against real production traffic in parallel, logging its
predictions for offline comparison without ever serving its output to an actual user or downstream system,
validates the model against genuine production data distribution while carrying zero risk — only once
shadow results confirm the offline improvement holds up against real traffic does it make sense to move to
an actual A/B traffic split via production variants.

## Where to Go From Here

For a reader using this series specifically toward certification, the practical next step is matching this
part's content against whichever exam guide is actually current at attempt time (Part 17's currency-
checking discipline applies here as much as anywhere) and working through official AWS practice questions
for the target exam — this series builds the conceptual foundation and the real architectural judgment an
exam (and real production work) actually tests, but exam-format familiarity still benefits from practice
against the real question style. For a reader using this series for production work rather than
certification, the decision frameworks throughout this part — pre-built vs Bedrock vs custom SageMaker,
RAG vs fine-tuning, which inference deployment option matches the traffic pattern — are the parts worth
returning to directly when a real ML/AI requirement actually shows up, more so than any individual
service's full feature list.

## Series Summary — Nineteen Parts, One Coherent Picture

This series set out to cover every AWS service and topic tested across the Associate and Professional
certifications — SAA-C03, DVA-C02, SOA-C02, DEA-C01, MLA-C01, SAP-C02, and DOP-C02 — and, across nineteen
parts, it does: account structure and IAM (Parts 1-2), compute and networking (Parts 3-4), storage and
databases (Parts 5-6), containers and serverless (Part 7), edge and delivery (Part 8), security (Part 9),
observability (Part 10), CI/CD and messaging (Part 11), multi-region and DR (Part 12), API-layer identity
and event integration (Part 13), developer tooling and deployment safety (Part 14), fleet operations (Part
15), cost and FinOps (Part 16), migration (Part 17), the data platform (Part 18), and finally ML/AI (this
part). The single idea worth carrying forward past any individual service's details: **every part of this
series has repeated the same underlying discipline** — understand what a service actually does and why it
exists, recognize the real tradeoff against its alternatives, and choose the option that fits the actual
requirement rather than the most familiar or most impressive one. That discipline, more than any single
service's API surface, is what a genuinely senior AWS practitioner — and a well-prepared exam candidate —
actually has.