Assumes you're comfortable with IAM roles and policies (Part 2), Lambda (Part 7), and SQS/SNS/EventBridge at the level Part 11 introduced them — this part goes much deeper on the pieces that sit in front of an application: who's allowed to call it, and how its internal services talk to each other.
Table of Contents#
- Why This Part Exists
- Two Different Problems Wearing Similar Names
- API Gateway — Three Products, One Name
- REST APIs — The Full-Featured Option
- HTTP APIs — Lightweight and Cheap
- REST vs HTTP APIs — Head to Head
- WebSocket APIs — Persistent, Two-Way Connections
- Integration Types — What's Actually Behind the Route
- Mapping Templates and Payload Transformation
- Request Validation and Models (REST APIs)
- Observability for APIs: Logs, Metrics, and Tracing
- Authorizers — IAM, Lambda, Cognito, and JWT
- Throttling, Usage Plans, and API Keys
- Caching at the API Gateway Layer
- Private APIs and VPC Links
- Endpoint Types: Edge-Optimized, Regional, and Private
- CORS — A Persistent Source of Confusion
- Canary Releases and Stage Variables
- Cognito — Identity for Customer-Facing Applications
- Cognito User Pools — Core Concepts
- User Pool Authentication Flows
- Hosted UI and Federation
- Identity Pools — From Token to AWS Credentials
- User Pools and Identity Pools, Wired Together
- Securing an API Gateway Route With Cognito, Worked
- Cognito Lambda Triggers — Customizing the Auth Lifecycle
- Advanced Security: Adaptive Authentication and Compromised Credentials
- Cognito vs IAM vs IAM Identity Center — Choosing the Right Identity System
- AppSync — GraphQL APIs on AWS
- AppSync Resolvers — Unit and Pipeline
- AppSync Real-Time Subscriptions
- AppSync Caching and Data Source Depth
- REST/HTTP APIs vs GraphQL — Choosing
- Step Functions, Revisited in Depth
- Standard vs Express Workflows
- Service Integrations and the Callback Pattern
- Error Handling: Retry and Catch
- The Map State and Distributed Map — Large-Scale Parallelism
- EventBridge, Revisited in Depth
- EventBridge Schema Registry
- EventBridge Pipes — Point-to-Point Integration
- Cross-Account and Cross-Region Event Buses
- Choosing Among SQS, SNS, EventBridge, Step Functions, and AppSync
- Amazon MQ — When Protocol Compatibility Matters
- Idempotency for Public APIs
- A Full Worked Example: A Public Order-Status API
- Security Checklist for Public APIs
- API and Event Integration Best Practices — The Consolidated Checklist
- Part 13 CLI Cheat Sheet
- Common Mistakes and Interview Traps
- Worked Practice Problems
- Summary and What's Next
Why This Part Exists#
Part 7 built Lambda functions and containers. Part 11 introduced SQS, SNS, and EventBridge as the plumbing that lets those services talk to each other asynchronously. What's been missing is the layer that sits in front of all of it — the thing an external client, a mobile app, or a partner integration actually calls, and the thing that decides whether the caller is who they claim to be. That's this part: API Gateway as the front door, Cognito as the identity system behind that door, AppSync as an alternative front door for GraphQL, and a much deeper pass on the event-driven orchestration tools Part 11 only introduced at a survey level.
This maps directly to the "API-first" and "identity as a first-class architectural concern" ideas that show up constantly in real platform work: an air-traffic control tower doesn't just route planes, it verifies every aircraft's clearance before it's allowed anywhere near the runway. API Gateway is that tower for a system's edge; Cognito is the clearance registry it checks against.
Two Different Problems Wearing Similar Names#
Before touching any service, separate two questions that get conflated constantly, including on both the DVA-C02 and SAP-C02 exams:
- Authentication — "who is this caller?" Cognito user pools, IAM users/roles, federated identity providers all answer this.
- Authorization — "what is this caller allowed to do?" IAM policies (Part 2), Cognito identity-pool role mappings, and API Gateway resource policies all answer this — separately, after authentication has already established an identity.
Note
Every identity system in this part produces the same end product for AWS API calls: a JSON Web Token (JWT) or a set of temporary IAM credentials. The system generating that token differs (Cognito, a corporate IdP via IAM Identity Center from Part 1, a Lambda authorizer's own logic), but what API Gateway and downstream AWS services actually check is always one of those two things.
API Gateway — Three Products, One Name#
"API Gateway" isn't one product with optional features — it's three genuinely different products sharing a console and a name: REST APIs, HTTP APIs, and WebSocket APIs. Picking the wrong one for a new project is a common, expensive-to-unwind mistake, because they don't share configuration and migrating between them means rebuilding routes and integrations from scratch.
REST APIs — The Full-Featured Option#
The original API Gateway product (sometimes called "REST APIs" or, in older docs, just "API Gateway"). Full feature set: request/response validation against a JSON Schema model, per-stage caching, API keys and usage plans, resource policies (including making an API private to a VPC), WAF integration, and mapping templates for arbitrary request/response transformation. This depth is also why it's the most expensive and most complex of the three, and why AWS's own current guidance is "use it only when you specifically need one of these features."
HTTP APIs — Lightweight and Cheap#
A newer, deliberately minimal product built for the common serverless case: a client calling a Lambda function or an HTTP backend, with JWT-based auth and CORS handling built in natively (no custom authorizer needed for the common case), and native integration with IAM Identity Center / OIDC / OAuth 2.0 providers. It does not support request validation, per-stage response caching, or API keys/usage plans as of this writing — if a project turns out to need those later, that's a genuine migration, not a config flag.
Tip
Default to HTTP APIs for anything new unless a specific REST-API-only feature is already a known requirement. Confirmed current pricing gap: HTTP APIs run roughly 70% cheaper per million requests than REST APIs for equivalent traffic, and the DVA-C02 exam explicitly expects "cheapest option that meets requirements" reasoning — reaching for REST APIs by habit when nothing in the requirements needs its extra feature set is a real, gradeable mistake on that exam, not just a cost nitpick in production.
REST vs HTTP APIs — Head to Head#
| Capability | REST API | HTTP API |
|---|---|---|
| Lambda/HTTP proxy integration | Yes | Yes |
| Native JWT authorizer | No (needs a Lambda authorizer) | Yes, built in |
| Request/response validation | Yes | No |
| Per-stage response caching | Yes | No |
| API keys and usage plans | Yes | No |
| Private API (VPC-only) | Yes | Yes |
| WAF integration | Yes | No |
| Mapping templates (VTL) | Yes | Limited (parameter mapping only) |
| Relative cost per million requests | Baseline | ~70% cheaper |
WebSocket APIs — Persistent, Two-Way Connections#
Unlike REST/HTTP APIs, which are strictly request-response, a WebSocket API holds a persistent
connection open, so either side can push a message at any time without the client polling. Routes are
keyed by message content rather than by URL path: $connect fires once when a client opens the
connection, $disconnect fires when it closes, $default catches anything not matched by a custom route,
and any number of custom routes match on a field inside the incoming message body. Connection state
(who's connected, what data to associate with a connection ID) has to be stored somewhere durable —
DynamoDB is the standard choice, since Lambda itself is stateless between invocations.
A live chat feature, a real-time dashboard, or a stock-ticker-style price feed are the canonical fits — anywhere the server needs to initiate the message, not just respond to one.
Integration Types — What's Actually Behind the Route#
Every route in API Gateway maps to an integration type, and picking the right one avoids writing glue code that AWS Gateway would otherwise do for free:
| Integration type | What it does |
|---|---|
| Lambda proxy | Passes the entire request through to Lambda as an event object; Lambda's response shape (statusCode, headers, body) is passed back verbatim. The default choice for serverless backends. |
| Lambda custom (non-proxy) | API Gateway transforms the request/response via mapping templates before/after Lambda runs. More setup, more control — mostly a REST-API-only pattern now. |
| HTTP proxy | Forwards the request essentially unchanged to an existing HTTP backend (an ALB-fronted service, a third-party API). |
| AWS service integration | Calls another AWS service's API directly (e.g., putting a message straight onto SQS, or invoking Step Functions) with zero compute in between. |
| Mock | Returns a canned response with no backend call at all — useful for CORS preflight OPTIONS responses or building out a contract before the backend exists. |
Mapping Templates and Payload Transformation#
REST APIs (and, to a lesser degree, HTTP APIs) can rewrite a request or response body in flight using
Velocity Template Language (VTL) mapping templates. The classic use case: exposing a clean, versioned
public API contract while the actual backend (say, a legacy SOAP service, or a differently-shaped internal
schema) doesn't match that contract at all. This is powerful and also genuinely painful to debug — VTL
errors show up as unhelpful Execution failed due to configuration error messages with limited context,
which is precisely why most new serverless-first designs prefer Lambda proxy integration and do the
transformation in ordinary application code instead, reserving VTL for the cases (AWS service integration
with no Lambda in the loop at all) where there's no other option.
Request Validation and Models (REST APIs)#
REST APIs can reject a malformed request before it ever reaches the backend, by validating the request
body against a JSON Schema model, and/or validating required query-string parameters and headers are
present — configured declaratively on the method, no application code involved. This moves a whole class
of defensive "is this field present and the right type" checks out of every Lambda function and into the
API layer itself, where a bad request fails fast with a 400 and never consumes any compute at all. HTTP
APIs, true to their minimal design, don't support this — a Lambda-side validation library (or a shared
validation layer, Part 7) is the equivalent there.
Observability for APIs: Logs, Metrics, and Tracing#
API Gateway integrates directly with the observability stack from Part 10 rather than inventing its own:
execution logs (detailed per-request debug logging, useful during development, expensive to leave on
in production at high volume), access logs (a configurable-format log line per request — method,
path, status, latency — the one worth always leaving on), CloudWatch metrics (Count, 4XXError,
5XXError, Latency, IntegrationLatency — the last one specifically isolating backend latency from API
Gateway's own overhead), and native X-Ray tracing support that stitches an API Gateway request into the
same trace as the Lambda/backend work it triggers, so a single trace shows the full request lifecycle
end to end rather than two disconnected fragments.
Authorizers — IAM, Lambda, Cognito, and JWT#
| Authorizer type | How it verifies the caller | Fits |
|---|---|---|
| IAM authorization | Caller signs the request with SigV4 using real IAM credentials; API Gateway checks the resulting identity against an IAM policy. | Service-to-service calls already inside AWS, internal tooling. |
| Lambda authorizer | A Lambda function receives the request (or just its token) and returns an IAM policy document deciding allow/deny, optionally with a caching TTL. | Custom auth logic (a legacy session token, an API key stored outside Cognito, header-based tenant routing). |
| Cognito user pool authorizer | API Gateway validates a Cognito-issued JWT directly, no Lambda in the loop. | REST APIs backed by Cognito-authenticated users. |
| JWT authorizer (HTTP APIs) | Native JWT validation against any OIDC-compliant issuer (Cognito, or a third-party IdP), configured declaratively. | HTTP APIs — the built-in replacement for a Lambda authorizer's most common use case. |
Important
A Lambda authorizer's response is cached by default (5 minutes unless configured otherwise), keyed on
the caller-supplied identity source (usually the Authorization header value). Revoking a compromised
token doesn't take effect until that cache entry expires unless the cache is deliberately invalidated —
a real operational gap worth knowing about before an incident, not during one.
Throttling, Usage Plans, and API Keys#
API Gateway enforces two levels of throttling: an account-level steady-state/burst limit shared across every API in the account/region, and a per-method or per-stage limit set explicitly. REST APIs add usage plans: a named bundle of throttle and quota limits (e.g., "1000 requests/day, 10 requests/second burst") associated with one or more API keys, letting an operator hand different partners different tiers of access to the same API without deploying separate infrastructure. An API key by itself is not an authentication mechanism — it identifies which usage plan applies, nothing more; a key alone grants no authorization, which is a genuinely common point of confusion worth stating explicitly.
Caching at the API Gateway Layer#
REST APIs support an optional, per-stage response cache (sized from 0.5 GB to 237 GB), keyed by request parameters and TTL-controlled per method. This trades a small added cost for the cache itself against a real reduction in backend load and latency for read-heavy, slowly-changing responses (a product catalog listing, a configuration payload) — but it's a full response cache at the edge closest to the client, a different layer entirely from CloudFront's caching (Part 8) or DAX in front of DynamoDB (Part 6). All three can legitimately be layered on the same request path, each solving a different part of the latency budget.
Private APIs and VPC Links#
A REST (or HTTP) API can be made private — reachable only from inside a VPC via an interface VPC endpoint (Part 4), never from the public internet — the standard shape for an internal platform API that should never be internet-facing at all. Going the other direction, a VPC Link lets a public-facing API Gateway route traffic into a private VPC resource (an internal ALB, an NLB fronting an ECS service) without that backend ever needing a public IP or an internet gateway route — the API Gateway layer is the only piece of the path exposed to the internet.
Endpoint Types: Edge-Optimized, Regional, and Private#
A REST API chooses one of three endpoint types, and the choice is about where traffic enters AWS's network, not about functionality: edge-optimized routes client requests through the nearest CloudFront edge location first (Part 8), reducing latency for geographically distributed clients at the cost of an extra network hop for a client that's already close to the API's home region; regional serves traffic directly from the API's own region, the better default when clients are concentrated near that region already, or when the API sits behind a separately managed CloudFront distribution (giving full control over caching/WAF/custom domains instead of the semi-managed edge-optimized setup); private, as covered above, is reachable only from inside a VPC. For a genuinely global user base needing true multi-region failover (Part 12's patterns), the real answer is usually regional APIs deployed in multiple regions, fronted by Route 53 latency-based or failover routing — edge-optimized alone doesn't provide multi-region backend failover, only faster network entry to a single backend region.
CORS — A Persistent Source of Confusion#
Cross-Origin Resource Sharing trips up more API Gateway builds than almost anything else, because the
error surfaces in the browser console with no useful detail (No 'Access-Control-Allow-Origin' header is present) that points nowhere near the actual misconfiguration. What's actually required, for both HTTP
and REST APIs, is two separate things working together: the browser's preflight OPTIONS request must get
a response listing the allowed origin/methods/headers, and the actual GET/POST/etc. response
itself must also carry an Access-Control-Allow-Origin header — configuring only one of the two is the
single most common cause of "it works in Postman but not the browser," since Postman never sends a
preflight request at all.
HTTP APIs support CORS as a native, declarative configuration block (allowed origins, methods, headers,
credentials, max-age) with no extra integration required. REST APIs need either a manually configured
OPTIONS Mock integration returning the right headers, or — for a Lambda proxy integration — the
Lambda function itself must include the CORS headers on every response it returns, preflight and real
alike, since a proxy integration passes the response through exactly as the function shaped it.
Warning
Setting Access-Control-Allow-Origin: * alongside Access-Control-Allow-Credentials: true is invalid
per the CORS spec and will be silently rejected by browsers — a wildcard origin cannot be combined with
credentialed requests (cookies, Authorization headers carrying session state). An API that needs both
wildcard-style flexibility and credentialed calls has to enumerate allowed origins explicitly instead.
Canary Releases and Stage Variables#
A REST API stage supports a canary release: a configurable percentage of traffic routed to a newly
deployed version while the rest continues hitting the previous one, with independent CloudWatch metrics
per canary vs. baseline traffic — the same progressive-delivery idea from Part 7's ECS/Lambda deployment
strategies, applied at the API Gateway layer itself rather than at the compute layer behind it. Stage
variables (REST APIs) are named key-value pairs scoped to a single stage, commonly used to point the same
API definition at different backend Lambda aliases or HTTP endpoints per environment (dev, staging,
prod) without maintaining separate API definitions — the API Gateway equivalent of an environment
variable, resolved at request time via ${stageVariables.lambdaAlias} inside an integration's
configuration.
Cognito — Identity for Customer-Facing Applications#
Part 2 covered IAM in full depth — but IAM identities (users, roles) are fundamentally for AWS principals: engineers, CI/CD pipelines, EC2 instances. They were never meant to model millions of end-users signing up for a consumer app. Amazon Cognito is the purpose-built answer to that different problem: a managed identity system for the humans (or devices) using an application, entirely separate from the IAM identities of the people operating the AWS account itself.
Cognito User Pools — Core Concepts#
A user pool is a managed user directory: sign-up, sign-in, password policies, MFA, account recovery, and custom user attributes, all without running a database of password hashes yourself. Successful authentication against a user pool returns three JWTs: an ID token (claims about who the user is — name, email, custom attributes), an access token (scoped for calling your own backend APIs), and a refresh token (used to get new ID/access tokens without forcing the user to log in again). Each token has its own separate, configurable expiry — access and ID tokens are short-lived by design (default one hour), refresh tokens live far longer (default 30 days), a deliberate tradeoff between session convenience and the blast radius of a leaked token.
User Pool Authentication Flows#
| Flow | How it works | Fits |
|---|---|---|
| USER_SRP_AUTH | Secure Remote Password protocol — the password itself never crosses the network, even encrypted. | Default for first-party web/mobile clients — the recommended flow. |
| USER_PASSWORD_AUTH | Plain username/password sent directly (over TLS). | Server-side / trusted-backend flows where SRP's extra round trips aren't practical. |
| Custom auth flow | A chain of Lambda triggers (DefineAuthChallenge, CreateAuthChallenge, VerifyAuthChallengeResponse) implementing arbitrary logic. | Passwordless flows (magic link, OTP via SMS/email), CAPTCHA-gated login, adaptive/step-up auth. |
| Refresh token flow | Exchange a valid refresh token for new ID/access tokens. | Keeping a session alive without re-prompting for credentials. |
Hosted UI and Federation#
A user pool can host a fully-managed sign-up/sign-in web page (the Hosted UI) so an application never has to build its own login form or handle raw credentials at all — a real reduction in the app's PCI/PII handling surface. The same user pool can federate with external identity providers: social logins (Google, Facebook, Apple, Amazon), or enterprise SAML/OIDC IdPs — letting a B2B SaaS product accept "sign in with your company's Okta/Azure AD" without building that integration from scratch. Federated and native (username/password) users land in the same user pool and produce the same token shape downstream, so the rest of the application never needs to know which path a given user came in through.
Identity Pools — From Token to AWS Credentials#
A user pool proves who someone is. An identity pool (a genuinely different, commonly confused resource) is what turns that proof into temporary AWS credentials — a short-lived access key/secret key/session token pair scoped by an IAM role, obtained via STS, exactly like the role-assumption mechanics covered in Part 2. This is what lets a mobile app upload a file straight to a specific S3 prefix, or read straight from a DynamoDB table, without a backend server in the middle brokering that access at all. Identity pools also support unauthenticated (guest) identities — a distinct, separately-scoped IAM role for users who haven't signed in yet, useful for letting a mobile app do something minimal (log analytics events, browse public content) before requiring a login.
User Pools and Identity Pools, Wired Together#
The identity pool's IAM role can use policy variables (Part 2's ABAC pattern) keyed on
cognito-identity.amazonaws.com:sub — the user's unique Cognito identity ID — to scope S3/DynamoDB access
to exactly that user's own data with one shared role definition, rather than provisioning a role per user.
Securing an API Gateway Route With Cognito, Worked#
- Create a user pool; enable the Hosted UI or a native client integration.
- Attach a Cognito user pool authorizer (REST API) or a JWT authorizer pointed at the pool's issuer URL (HTTP API) to the route.
- The client signs in against the user pool, gets an ID/access token.
- The client calls the API with
Authorization: Bearer <token>. - API Gateway validates the token's signature and expiry against the user pool's public keys — before the request ever reaches Lambda; an invalid or expired token never executes any application code.
- The Lambda function receives the decoded claims (user ID, custom attributes, scopes) in the event's
requestContext.authorizer— enough to make per-user authorization decisions without a second call back to Cognito.
Cognito Lambda Triggers — Customizing the Auth Lifecycle#
A user pool can invoke a Lambda function at over a dozen points in the authentication lifecycle, letting an application inject custom logic without forking Cognito's own behavior:
| Trigger | Fires when | Common use |
|---|---|---|
PreSignUp | Before a new account is created | Auto-confirm trusted domains, block disposable-email signups, validate an invite code |
PostConfirmation | Right after a user confirms their account | Create a matching row in an application database, send a welcome event onto EventBridge |
PreTokenGeneration | Before ID/access tokens are issued (including on refresh) | Inject custom claims (a tenant ID, a subscription tier) into the token so downstream services never need a second lookup |
PreAuthentication | Before a sign-in attempt is validated | Custom risk checks, blocking known-bad IPs |
CustomMessage | Before a verification/invite email or SMS is sent | Fully custom email templates and branding |
DefineAuthChallenge / CreateAuthChallenge / VerifyAuthChallengeResponse | During a custom auth flow | Passwordless (OTP, magic link) login flows |
PreTokenGeneration is worth calling out specifically: it's what makes Cognito viable for genuinely
multi-tenant SaaS applications, since a tenant ID baked into the access token means every downstream
Lambda/API can authorize purely from the token's claims, with zero additional database round trip per
request.
Advanced Security: Adaptive Authentication and Compromised Credentials#
User pools with the Plus feature tier enabled add adaptive authentication: each sign-in attempt is scored for risk (unfamiliar device, impossible-travel location, known bad IP reputation) and can trigger step-up MFA or an outright block on high-risk attempts, without the application writing any custom risk logic itself. A separate, always-available check, compromised credentials detection, blocks sign-in or sign-up attempts using a username/password pair AWS has identified in a known public credential-leak dataset — the managed equivalent of the "have I been pwned" pattern, wired directly into the sign-in flow rather than bolted on separately.
Cognito vs IAM vs IAM Identity Center — Choosing the Right Identity System#
| System | Identities it manages | Typical caller |
|---|---|---|
| IAM users/roles (Part 2) | AWS-account-scoped principals | CI/CD pipelines, service-to-service, break-glass access |
| IAM Identity Center (Part 1) | Workforce (employees) across a whole Organization | Engineers logging into the AWS Console/CLI across many accounts |
| Cognito | End-users of an application your company built | Customers of a mobile app, users of a SaaS product's web UI |
A genuinely common architectural mistake: provisioning individual IAM users for an application's customers. It doesn't scale (IAM has hard account-level identity limits nowhere near consumer-app volume), it conflates two categorically different trust boundaries, and it's precisely the anti-pattern Cognito exists to replace.
AppSync — GraphQL APIs on AWS#
AppSync is API Gateway's GraphQL-shaped sibling: a managed GraphQL API that can resolve fields from DynamoDB, Lambda, RDS (via the RDS Data API), OpenSearch, or any HTTP endpoint — often several of them in a single client request. Where a REST/HTTP API typically means "one round trip per resource," a GraphQL client can request exactly the fields it needs, from however many underlying data sources those fields actually live in, in one round trip — a real advantage for mobile clients on constrained networks pulling data assembled from multiple services.
AppSync Resolvers — Unit and Pipeline#
A resolver is the piece of configuration that maps one GraphQL field to a data source and a request/response transformation, written in APPSYNC_JS (JavaScript) or the older VTL. A unit resolver talks to exactly one data source. A pipeline resolver chains several AppSync functions in sequence — validate input, call one data source, feed its result into a second data source — all within a single GraphQL field resolution, without needing a Step Functions state machine or an orchestrating Lambda just to sequence two internal calls.
AppSync Real-Time Subscriptions#
Beyond queries (read) and mutations (write), GraphQL defines subscriptions — a client subscribes to a
field, and AppSync pushes an update over a managed WebSocket connection whenever a matching mutation
happens, with no connection-management code required on either side (unlike a hand-rolled WebSocket API,
where $connect/$disconnect bookkeeping is the caller's responsibility). This is the same "push, not
poll" capability a raw WebSocket API provides, but scoped to GraphQL's type system and wired in
automatically.
AppSync Caching and Data Source Depth#
AppSync supports its own server-side response cache, either full-query or per-resolver, backed transparently by a managed cluster the operator sizes but never patches — conceptually the same tradeoff as API Gateway's stage cache (Part 13's earlier section), applied to GraphQL field resolution instead of whole HTTP responses. Beyond DynamoDB and Lambda, AppSync data sources also include OpenSearch (full-text search fields inside a GraphQL schema), the RDS Data API (querying Aurora Serverless without managing a persistent connection pool from inside a resolver), and HTTP data sources for wrapping an existing REST API behind a GraphQL facade — a common modernization step when a GraphQL layer needs to sit in front of services that were never rebuilt as GraphQL-native.
REST/HTTP APIs vs GraphQL — Choosing#
| Consideration | REST/HTTP API (API Gateway) | GraphQL (AppSync) |
|---|---|---|
| Client fetches exactly the fields it needs | No — fixed response shape per endpoint | Yes — client specifies the shape |
| Aggregating multiple backend sources in one call | Requires a BFF/orchestrating Lambda | Native, via pipeline resolvers |
| Caching | Mature (per-stage cache, CloudFront) | Possible but less mature tooling |
| Learning curve / ecosystem maturity | Lower — plain HTTP semantics | Higher — GraphQL schema design discipline |
| Real-time push | WebSocket API (separate product) | Built in via subscriptions |
Neither replaces the other outright — a public partner-facing API with a stable, documented contract still usually favors REST/HTTP; a mobile app assembling a dashboard from five different data types in one screen is the case GraphQL was built for.
Step Functions, Revisited in Depth#
Part 7 introduced Step Functions as "orchestrating multiple Lambdas." The full picture: Step Functions is a managed state machine, defined in Amazon States Language (ASL), that can orchestrate not just Lambda but any of over 220 AWS service API actions directly — starting a Glue job (Part 18), running an ECS task, publishing to SNS — with retry, error handling, and parallel/choice branching declared in the state machine definition itself instead of hand-written in application code.
Standard vs Express Workflows#
| Standard | Express | |
|---|---|---|
| Max duration | Up to 1 year | Up to 5 minutes |
| Execution semantics | Exactly-once | At-least-once (async) or at-most-once (sync) |
| Execution history | Full, visual, retained 90 days, free | CloudWatch Logs only, at extra cost |
| Pricing model | Per state transition | Per invocation + duration (GB-seconds) |
| Fits | Order fulfillment, approval workflows, long ETL jobs — anything needing a durable audit trail | High-volume event processing, streaming transformation, IoT ingestion |
The choice is rarely close once the durations and volumes are known: a workflow that must survive for days and be individually auditable belongs on Standard; a workflow firing thousands of times a second for sub-second tasks belongs on Express, where Standard's per-state-transition pricing would be far more expensive.
Service Integrations and the Callback Pattern#
Most AWS service integrations in Step Functions support one of three request-response patterns:
Request Response (fire the API call, immediately continue), Run a Job (.sync) (wait for a
service's own asynchronous job to finish — an ECS task, a Glue job, a Batch job — polling internally so the
workflow doesn't need custom polling logic), and Wait for Callback (.waitForTaskToken) (pause the
state machine entirely until an external system calls back SendTaskSuccess/SendTaskFailure with a
token, the standard pattern for a human-approval step or a long-running third-party process with no native
Step Functions integration).
Error Handling: Retry and Catch#
Every state can declare a Retry block (which errors to retry, backoff interval, max attempts, backoff
rate) and a Catch block (which errors route to which fallback state) directly in the ASL definition —
the same resilience patterns from Part 8/Part 4 of the broader course (circuit breakers, exponential
backoff), expressed declaratively instead of hand-coded in every Lambda function that might fail.
The Map State and Distributed Map — Large-Scale Parallelism#
Step Functions' Map state iterates a state machine's sub-workflow over every item in an input array,
running iterations concurrently up to a configurable concurrency limit — the declarative equivalent of a
for loop with a worker pool, without writing the pool management. The standard Map state holds its
entire input array in the state's own execution history, which caps it at roughly 40 items of meaningfully
sized payload before hitting Step Functions' 256 KB state-size limit. Distributed Map, a newer mode
built for genuinely large-scale fan-out, instead reads its input directly from an S3 object (a manifest
file, or every object under a prefix) and can launch up to 10,000 parallel child workflow executions,
each running independently with its own execution history — the pattern for processing every file dropped
into an S3 bucket, or reprocessing millions of existing database rows, without hitting the standard Map
state's size ceiling at all.
EventBridge, Revisited in Depth#
Part 11 covered EventBridge's core loop: producers publish to a bus, rules match events by pattern, and matched events fan out to up to five targets each. Two pieces worth going deeper on for real production use and for SAP-C02-level architecture questions: schema management, and the newer point-to-point Pipes product.
EventBridge Schema Registry#
The schema registry stores the shape of events flowing through a bus — either discovered automatically from real traffic, or defined manually as OpenAPI 3 or JSON Schema — and can generate strongly-typed code bindings for consuming applications in several languages. This solves a real, easy-to-underestimate problem in event-driven architectures: without a shared schema contract, a producer team can silently change an event's shape and break every consumer that assumed the old shape, with no compile-time warning anywhere. The registry itself is free; it's a governance tool, not a runtime dependency.
EventBridge Pipes — Point-to-Point Integration#
Pipes are a different shape than rules entirely: one source, one target, no bus and no many-to-many routing in between, but with built-in filtering, enrichment (calling a Lambda/Step Functions/API Destination to transform the payload before delivery), and support for source types rules don't have direct access to — an SQS queue, a Kinesis/DynamoDB stream, or an MSK topic (Part 18) feeding directly into a target with no polling Lambda required to bridge them. Where a rule answers "route this event to whichever of several interested targets," a pipe answers "reliably move and transform everything from this one specific source into this one specific target" — a narrower but often simpler and cheaper tool for that narrower job.
Cross-Account and Cross-Region Event Buses#
A single organization's event-driven architecture rarely lives in one account. EventBridge supports cross-account event delivery: a rule in the producing account's bus can target the default bus of a different account directly (with a resource-based policy on the receiving bus granting that specific sender permission), which is how a security-tooling account (Part 1/Part 9's log-archive-style pattern) can centrally receive GuardDuty/Config compliance events from every workload account without each one individually pushing to a shared queue. Cross-region delivery works the same way via a rule targeting a bus ARN in another region — useful for centralizing operational events from a multi-region deployment (Part 12) into one region's monitoring/alerting stack, though it's worth remembering this is a one-directional push, not a replicated bus — each region's bus still only sees what's been explicitly routed to it.
Choosing Among SQS, SNS, EventBridge, Step Functions, and AppSync#
| Need | Reach for |
|---|---|
| Decouple two services, one consumer processes each message | SQS |
| Fan out one message to many independent subscribers | SNS (often with SQS behind each subscriber) |
| Route events by content/source across many possible consumers, with schema governance | EventBridge (rules) |
| Move a stream/queue's contents into exactly one target, with transformation | EventBridge Pipes |
| Orchestrate a multi-step, stateful process with retries, branching, and an audit trail | Step Functions |
| Serve a client that needs exactly-the-fields-it-asks-for from multiple sources, with real-time push | AppSync |
| Bridge an existing app that already speaks AMQP/JMS/MQTT, not worth rewriting | Amazon MQ |
| Fan out processing across thousands of independent items from an S3 manifest | Step Functions Distributed Map |
Amazon MQ — When Protocol Compatibility Matters#
Amazon MQ is a managed broker running actual Apache ActiveMQ or RabbitMQ, speaking industry-standard protocols (AMQP 0-9-1, AMQP 1.0, MQTT, OpenWire, STOMP) rather than a proprietary AWS API. It exists for exactly one reason, and AWS's own guidance is explicit about it: migrating an existing application that already speaks one of those protocols, without rewriting producer/consumer code to target SQS/SNS's API. For anything greenfield, SQS/SNS/EventBridge are the default — they scale further, cost less at low-to- moderate throughput, and require zero broker capacity planning, none of which Amazon MQ removes (it still bills per broker-hour, not per message, so it can actually be cheaper only at sustained high, steady throughput). For availability, a broker can be deployed single-instance (cheapest, no built-in failover) or active/standby across two AZs, where the standby takes over automatically on a failure — the same Multi-AZ shape RDS uses (Part 6), applied to a message broker instead of a database.
Idempotency for Public APIs#
A mobile client on a flaky connection will retry a request it never got a response for — and a naive
POST /orders handler that creates a new order on every invocation will happily create duplicates from
that single user action. The standard fix, expected knowledge on DVA-C02's troubleshooting domain: require
an idempotency key (a client-generated UUID) on the request, and have the backend check a fast-lookup
store (DynamoDB, with a short TTL) for that key before doing any real work — if it's already been seen,
return the original response instead of repeating the side effect. This is application-level logic, not
something API Gateway provides natively, but it belongs in this part because the failure mode it prevents
is specifically a public-API-and-retry problem, not a general backend concern: a well-behaved internal
service call inside a VPC rarely needs it, while anything reachable from an unreliable client network
almost always does.
Tip
Step Functions' Standard workflows are exactly-once by design (revisited earlier in this part) — which is one more reason a long-running, side-effect-heavy public API operation (charging a payment, for instance) is often better modeled as starting a Standard workflow idempotently (keyed the same way) than as a single Lambda doing everything inline.
A Full Worked Example: A Public Order-Status API#
A retail platform needs to let its mobile app show a customer their order status in real time, without exposing internal fulfillment systems directly.
- Identity: A Cognito user pool handles customer sign-up/sign-in via the Hosted UI, federated with Google and Apple sign-in for lower-friction onboarding.
- Read path: An HTTP API, secured with a JWT authorizer pointed at the user pool, exposes
GET /orders/{id}— a Lambda proxy integration reads from a DynamoDB table (Part 6) scoped to the caller's ownsubclaim. - Live updates: Rather than polling, the mobile app opens an AppSync subscription on
onOrderStatusChanged— internally, a fulfillment-side Step Functions workflow publishes an EventBridge event on each status transition, which a small Lambda target turns into an AppSync mutation, fanning out to every subscribed client instantly. - Internal orchestration: The fulfillment workflow itself (validate → charge → reserve inventory → notify) runs as the Standard Step Functions state machine diagrammed above — long-running by fulfillment standards (minutes to hours), needing the full audit trail for customer-support investigations.
- Direct upload: When a customer needs to upload a return-request photo, the app exchanges its
Cognito ID token at the identity pool for temporary AWS credentials scoped (via a policy variable on
sub) to only that customer's own S3 prefix, uploading directly with no backend server or Lambda in the path at all.
Security Checklist for Public APIs#
- Every public route has an explicit authorizer — no route silently left open by omission.
- Throttling limits set at both the account and per-method level, sized for real expected traffic, not left at defaults.
- WAF attached (REST APIs) for anything internet-facing and handling sensitive data (Part 9).
- Cognito user pool has MFA enabled for anything beyond a low-sensitivity consumer app.
- Identity pool IAM roles use
sub-scoped policy variables — never one broad shared role for every authenticated identity. - Lambda authorizer cache TTL is understood and acceptable for the token-revocation risk it implies.
- CloudTrail (Part 9) is capturing API Gateway management-plane changes; access logging is enabled on every stage.
- No secrets or long-lived credentials embedded in a mobile/web client — only short-lived tokens ever leave the identity system.
API and Event Integration Best Practices — The Consolidated Checklist#
- Default new APIs to HTTP APIs; move to REST only when a specific feature actually requires it.
- Never build a custom auth Lambda for the common JWT-validation case — HTTP APIs' native JWT authorizer and REST APIs' Cognito authorizer both cover it without custom code.
- Keep IAM identities for AWS principals and Cognito identities for application end-users strictly separate — never provision IAM users to represent app customers.
- Scope every identity-pool IAM role with
sub-based policy variables rather than one shared broad role. - Treat API keys as a usage-plan selector only, never as an authentication mechanism on their own.
- Pick Standard vs Express Step Functions workflows by duration and volume, not by habit — the wrong choice is either an unnecessary per-state cost or a hard 5-minute ceiling hit in production.
- Reach for EventBridge Pipes only for genuine one-source-to-one-target plumbing; use rules the moment more than one consumer might ever care about an event type.
- Attach WAF (Part 9) to any public REST API handling sensitive data; HTTP APIs need a CloudFront distribution in front for the same protection, since HTTP APIs don't support WAF directly.
- Confirm CORS is configured on both the preflight response and the real response — testing only via a
tool that skips preflight (like a bare
curlor Postman call) hides the most common failure mode.
Part 13 CLI Cheat Sheet#
| Task | Command |
|---|---|
| Create an HTTP API | aws apigatewayv2 create-api --name my-api --protocol-type HTTP --target <lambda-arn> |
| Create a JWT authorizer | aws apigatewayv2 create-authorizer --api-id <id> --authorizer-type JWT --identity-source '$request.header.Authorization' --jwt-configuration Audience=<client-id>,Issuer=<issuer-url> |
| Deploy a REST API stage | aws apigateway create-deployment --rest-api-id <id> --stage-name prod |
| Create a Cognito user pool | aws cognito-idp create-user-pool --pool-name my-pool |
| Create a user pool client | aws cognito-idp create-user-pool-client --user-pool-id <id> --client-name web-app |
| Create an identity pool | aws cognito-identity create-identity-pool --identity-pool-name my-identities --allow-unauthenticated-identities |
| Start a Step Functions execution | aws stepfunctions start-execution --state-machine-arn <arn> --input '{"orderId":"123"}' |
| Describe an execution's history | aws stepfunctions get-execution-history --execution-arn <arn> |
| Create an EventBridge pipe | aws pipes create-pipe --name my-pipe --source <arn> --target <arn> --role-arn <role-arn> |
| Create an AppSync API | aws appsync create-graphql-api --name my-api --authentication-type AMAZON_COGNITO_USER_POOLS |
Common Mistakes and Interview Traps#
| Mistake | Why it's wrong | Correct approach |
|---|---|---|
| Defaulting to REST APIs for a new serverless project | Pays ~3.5x more per request for features never used | Default to HTTP APIs; move to REST only when a specific feature (caching, usage plans) is actually needed |
| Provisioning IAM users for application end-users | Doesn't scale, conflates workforce and customer identity | Use Cognito user pools for application identities |
| Treating an API key as authentication | An API key only selects a usage plan — it grants no identity or authorization on its own | Pair a usage plan/API key with a real authorizer |
| Using Step Functions Standard for a sub-second, high-volume event pipeline | Per-state-transition pricing gets expensive fast at that volume | Use Express workflows for high-throughput, short-duration work |
| Assuming a Lambda authorizer's deny takes effect instantly | Results are cached (5 min default); a revoked token can still pass until the cache entry expires | Set an appropriately short cache TTL for anything security-sensitive, or explicitly invalidate |
| Building a hand-rolled WebSocket connection table just to push GraphQL updates | AppSync subscriptions already do this natively | Use AppSync subscriptions when the API is already GraphQL |
| Choosing Amazon MQ for a new, greenfield queuing need | Pays for broker-hour capacity even during idle periods; more operational surface than SQS | Default to SQS/SNS/EventBridge; reach for MQ only for protocol-compatible migrations |
Worked Practice Problems#
Problem 1: A mobile app needs to let authenticated users upload profile photos directly to S3, and the team is currently routing every upload through a Lambda function that just proxies bytes from the client to S3. What's the architectural improvement, and what AWS service makes it possible?
Answer: Route the upload directly from the client to S3, bypassing the Lambda proxy entirely. A Cognito
identity pool, fed by the user pool's ID token, issues temporary AWS credentials scoped (via a sub-based
policy variable) to only that user's own S3 prefix — the client uploads with those credentials directly.
This removes Lambda's payload-size limits and cold-start latency from the upload path, and removes the
cost of running compute purely to relay bytes it never needed to touch.
Problem 2: An order-processing workflow needs to pause and wait for a human manager's approval on any order over $10,000, potentially for hours, before continuing. What Step Functions pattern fits, and why not just have the Lambda function poll a database for the approval?
Answer: The .waitForTaskToken (Wait for Callback) integration pattern. The state machine pauses
entirely — consuming no compute, incurring no polling cost or complexity — until an external system (an
approval UI's backend) calls SendTaskSuccess or SendTaskFailure with the token issued when the state
paused. Polling from inside a Lambda would either need a slow, wasteful loop or an entirely separate
scheduling mechanism to re-invoke itself, reinventing what the callback pattern already provides natively.
Problem 3: A partner integration needs webhook-style events whenever an order's status changes, but different partners care about different subsets of event types, and the list of partners changes frequently. Would EventBridge rules or EventBridge Pipes fit better, and why?
Answer: Rules. Pipes are one-source-to-one-target; adding or removing partners would mean creating or deleting individual pipes per partner with no shared routing logic. A bus with per-partner rules, each matching on the event types that partner cares about and targeting that partner's own SQS queue or API destination, handles an arbitrary and changing number of many-to-many subscribers without restructuring the pipeline — exactly the shape EventBridge rules were built for.
Summary and What's Next#
API Gateway is the front door — three different products depending on whether the API needs REST's full feature set, HTTP's low cost and native JWT auth, or WebSocket's persistent push. Cognito is the identity system behind that door for customer-facing applications: user pools authenticate, identity pools convert that authentication into real temporary AWS credentials. AppSync offers the same front-door role for GraphQL clients that need to assemble data from multiple sources in one round trip. And Step Functions/EventBridge, revisited in depth, round out the event-driven orchestration toolkit Part 11 only introduced — Standard vs Express workflows for stateful orchestration, schema registry and Pipes for governed and point-to-point event delivery.
Part 14 moves from the request path to the deployment path: Elastic Beanstalk as a simpler PaaS-style alternative to the container/serverless patterns already covered, the AWS SAM framework for building and deploying serverless applications with a tighter feedback loop than raw CloudFormation, and the developer productivity tools (CodeArtifact, CodeGuru, Cloud9/CloudShell) that round out the DVA-C02 deployment domain.