Data Analytics & Engineering

Assumes you're comfortable with S3 (Part 5), the database services and the brief Redshift/Athena/Glue mentions in Part 6, and Kinesis's basic shape from Part 11 — this part goes to the full depth DEA-C01 expects across data ingestion, transformation, storage, and governance.

Table of Contents#

1. Why This Part Exists
2. The Modern Data Architecture, at a Glance
3. AWS Glue — The Serverless ETL Backbone
4. The Glue Data Catalog
5. Glue Crawlers
6. Glue Jobs: Spark, Python Shell, and Ray
7. Glue Studio and Visual ETL
8. Glue DataBrew — No-Code Data Preparation
9. Glue Job Bookmarks — Incremental Processing
10. Kinesis Data Streams — Real-Time Ingestion
11. Shards, Capacity Modes, and the 2026 On-Demand Advantage
12. Streaming Ingestion, Visually
13. Amazon Data Firehose — Managed Delivery
14. Kinesis Data Streams vs Firehose vs Managed Service for Apache Flink
15. DynamoDB Streams as a Data Source
16. Schema Registry for Streaming Data
17. Amazon MSK — Managed Kafka
18. Choosing a Streaming Service
19. Amazon EMR — Big Data Processing at Scale
20. EMR Serverless
21. EMR vs Glue — Choosing
22. Amazon Redshift, Revisited in Depth
23. Redshift Serverless vs Provisioned (RA3)
24. Redshift Spectrum
25. Zero-ETL Integrations
26. AWS Lake Formation — Centralized Data Governance
27. Lake Formation Fine-Grained Access Control
28. Partitioning Strategy in Depth
29. Redshift Workload Management and Concurrency Scaling
30. Athena Federated Query
31. Amazon Athena, Revisited in Depth
32. Table Formats: Apache Iceberg and the Data Lakehouse
33. OpenSearch Service for Log and Search Analytics
34. Amazon QuickSight — BI and Visualization
35. Amazon Q in QuickSight — Generative BI
36. Data Lake Zones: Raw, Curated, and Consumption
37. Batch vs Streaming: A Unifying Decision Framework
38. Orchestrating Data Pipelines: Step Functions vs Glue Workflows vs MWAA
39. SageMaker Feature Store and the ML/Analytics Boundary
40. Data Quality and Schema Evolution
41. Cross-Account Data Sharing
42. Security and Governance for the Data Platform
43. Cost Optimization for the Data Platform
44. A Closing Note: This Is the Series' Own Architecture, Applied to Data
45. A Full Worked Example: A Real-Time Order Analytics Pipeline
46. Data Platform Best Practices — The Consolidated Checklist
47. A Note on Data Contracts
48. Part 18 CLI Cheat Sheet
49. Common Mistakes and Interview Traps
50. Worked Practice Problems
51. Summary and What's Next

---

Why This Part Exists#

Part 6 introduced Redshift, Athena, and Glue as brief, practical mentions inside a broader databases part; Part 11 introduced Kinesis at survey depth. This part is where all of them get the depth DEA-C01 actually expects — ingestion (batch and streaming), transformation, storage, cataloging, governance, and consumption, as one coherent pipeline rather than a set of disconnected service mentions. The throughline worth holding onto across this whole part: data moves through stages (ingest → store → catalog → transform → serve), and almost every service here occupies exactly one of those stages — confusing which stage a service belongs to is the single most common source of architectural mistakes on both the exam and in real pipeline design.

The Modern Data Architecture, at a Glance#

Every service covered in this part maps onto one of these five stages — worth referring back to this diagram whenever a new service's role feels unclear.

AWS Glue — The Serverless ETL Backbone#

AWS Glue is a fully managed, serverless data integration service spanning discovery (crawlers), metadata (the Data Catalog), and transformation (ETL jobs) — the closest thing this part has to a single central hub, since the Data Catalog specifically is consumed by nearly every other service in this part (Athena, Redshift Spectrum, EMR, Lake Formation all read from it). Glue bills per DPU-hour (Data Processing Unit) for job execution, with no idle cost for infrastructure sitting unused between runs — the same serverless economic shape Part 7 established for Lambda, applied to data processing instead of application compute.

The Glue Data Catalog#

The Data Catalog is a persistent, Hive-metastore-compatible metadata store — table definitions, schemas, partition information — decoupled from the actual data sitting in S3. This decoupling is the architectural key to this entire part: multiple compute engines (Athena, Redshift Spectrum, EMR, even third-party tools) can all query the exact same underlying S3 data through the exact same catalog entry, without each engine needing its own separate metadata layer or a costly data copy — a single source of truth for "what does this data actually look like," queried by however many different consuming engines a pipeline needs.

Glue Crawlers#

A crawler scans a data source (S3, a JDBC-connected database, DynamoDB) and automatically infers schema — column names, types, partition structure — populating the Data Catalog without a person hand- writing table DDL. Crawlers can run on a schedule to pick up schema drift automatically (a new column appearing in incoming data, for instance) or on demand after a known structural change. This is specifically the "discovery" stage of the pipeline — a crawler doesn't move or transform any data itself, it only builds and maintains the metadata describing data that already exists.

Glue Jobs: Spark, Python Shell, and Ray#

A Glue job is the actual transformation step, available in three runtime flavors: Spark (the default and most common — distributed processing for genuinely large datasets, using Glue's own Spark-based ETL library on top of open-source Apache Spark), Python Shell (a single-node Python script, appropriate for smaller transformation tasks or simple orchestration logic that doesn't need distributed processing at all), and Ray (a newer runtime option for Python-native distributed workloads, particularly relevant for ML data-preparation pipelines feeding into Part 19's SageMaker material). Picking the right runtime is a real cost and complexity decision — reaching for Spark on a dataset small enough for Python Shell wastes both DPU cost and unnecessary distributed-systems complexity.

Glue Studio and Visual ETL#

Glue Studio provides a visual, drag-and-drop interface for building ETL jobs — source, transform, and target nodes connected on a canvas, with Glue generating the underlying Spark code automatically. This lowers the barrier for a data analyst who understands the transformation logic needed but isn't necessarily a Spark/Python expert, while still producing genuine, inspectable, version-controllable code underneath — not a black box the visual tool alone can maintain.

Glue DataBrew — No-Code Data Preparation#

DataBrew is a separate, purpose-built tool for data cleaning and normalization specifically — over 250 pre-built transformations (handling missing values, standardizing formats, removing duplicates) accessible through a visual interface with no code at all, aimed at data analysts rather than engineers. Where a Glue Studio job is a general-purpose ETL pipeline, DataBrew is narrowly focused on the "my data is messy and I need to clean it before anything else can happen" problem specifically — a real, common, and often underestimated fraction of any data pipeline's total effort.

Glue Job Bookmarks — Incremental Processing#

A recurring Glue job re-run against a growing S3 prefix has a real problem worth solving deliberately: reprocessing every file on every run wastes both time and DPU cost. Job bookmarks solve this natively — Glue tracks which S3 objects (or JDBC rows, via a tracking column) have already been processed and automatically skips them on the next run, processing only genuinely new data. This is the Glue-native equivalent of DataSync's incremental transfer behavior (Part 17) and CodeBuild's dependency caching (Part 16) — the same "don't redo work that's already done" principle, expressed through Glue's own job configuration rather than a separate mechanism.

Kinesis Data Streams — Real-Time Ingestion#

Kinesis Data Streams is a durable, replayable stream a team builds custom consumers against — data written to the stream stays available for a configurable retention window (24 hours by default, extendable to 365 days), meaning multiple independent consumers can each read the same data at their own pace, and a consumer that falls behind or needs to reprocess historical data can simply re-read from an earlier point in the stream. This replayability is the single biggest architectural difference from SQS (Part 11) — an SQS message disappears once consumed; a Kinesis record stays available to every consumer until its retention window expires.

Shards, Capacity Modes, and the 2026 On-Demand Advantage#

Throughput in Kinesis Data Streams is governed by shards — each providing 1 MB/s (or 1,000 records/s) of write capacity and 2 MB/s of read capacity, with enhanced fan-out giving each consumer a dedicated 2 MB/s per shard rather than consumers competing for shared read throughput. Two capacity modes exist: Provisioned (shard count set explicitly, cost-predictable, appropriate for steady, well-understood traffic) and On-Demand (Kinesis manages shard scaling automatically, billed by actual throughput consumed, appropriate for spiky or unpredictable traffic). The 2025/2026 On-Demand Advantage enhancement raised on-demand's practical ceiling — larger record sizes (up to 10 MiB) and support for more enhanced-fan-out consumers — meaningfully narrowing the gap that used to push high-throughput, predictable workloads toward Provisioned mode by default.

Streaming Ingestion, Visually#

The same raw stream feeds an arbitrary number of independent consumers, each reading at its own pace — Firehose delivering to S3 for later batch analysis, and two entirely separate real-time applications reading the identical events for completely different purposes, none of them interfering with each other.

Amazon Data Firehose — Managed Delivery#

Amazon Data Firehose (the current name for what was previously called Kinesis Data Firehose) is a fully managed delivery service — no shards to size, no consumer code to write or maintain. Firehose buffers incoming records (by size or time interval) and writes them to a destination — S3, Redshift, OpenSearch, a third-party HTTP endpoint — with optional inline transformation via a Lambda function and optional format conversion (JSON to Parquet, for instance) applied automatically during delivery. Where Kinesis Data Streams is the tool when a team needs to build custom, potentially multiple, replay-capable consumers, Firehose is the tool when the actual need is simpler: get streaming data reliably into a storage/analytics destination, with AWS managing everything in between.

Kinesis Data Streams vs Firehose vs Managed Service for Apache Flink#

DynamoDB Streams as a Data Source#

Part 6 introduced DynamoDB Streams for triggering Lambda on table changes. It's also a legitimate, commonly-used ingestion source for this part's pipelines: a DynamoDB Streams event can feed directly into Kinesis Data Streams (via Kinesis Data Streams for DynamoDB, a direct integration bypassing a Lambda-based relay entirely) or trigger a Lambda that writes into Firehose — turning every write to an operational DynamoDB table into a real-time event feeding the exact same analytics pipeline this part describes, without the source application ever needing to know an analytics pipeline exists downstream at all. This is a genuinely clean pattern for capturing operational data for analytics without adding any burden to the application team owning the operational table.

Schema Registry for Streaming Data#

Just as the Glue Data Catalog governs schema for data at rest, the Glue Schema Registry governs schema for data in motion — producers and consumers of a Kinesis or MSK stream register and validate against a shared schema (Avro, JSON Schema, or Protobuf), preventing the same "a producer silently changes the event shape and breaks every consumer with no warning" problem Part 13's EventBridge Schema Registry solves for event-driven architectures, applied here to high-throughput streaming data specifically. A schema-registry-enforced stream fails fast at the producer if a write doesn't conform, rather than letting malformed data flow downstream and corrupt a transformation stage or analytics query far from its actual source.

Amazon MSK — Managed Kafka#

Amazon MSK (Managed Streaming for Apache Kafka) runs actual Apache Kafka, for organizations with an existing Kafka investment (consumer code, tooling, operational expertise) that want the protocol compatibility without self-managing Kafka brokers — the same "protocol compatibility over rewriting everything" value proposition Part 13 established for Amazon MQ, applied to streaming instead of traditional message queuing. MSK Serverless removes broker provisioning and capacity planning entirely, scaling automatically with actual traffic — the Kafka-compatible answer to Kinesis's own on-demand mode, for a team that specifically needs Kafka's ecosystem and semantics rather than Kinesis's AWS-native API.

Choosing a Streaming Service#

Amazon EMR — Big Data Processing at Scale#

EMR runs open-source big-data frameworks (Apache Spark, Hive, Trino, Flink, Presto) on managed clusters, with AWS-optimized runtimes delivering meaningfully better performance than vanilla open-source builds of the same frameworks. EMR is the right tool specifically when a workload needs the full, open-source framework ecosystem — custom Spark libraries, existing Hive-based pipelines, Presto/Trino federated queries — rather than Glue's more constrained, Glue-specific Spark environment. Clusters can run on EC2 (full control over instance types, including Spot for cost savings on fault-tolerant stages), on EKS (for teams standardizing all compute on Kubernetes, Part 7), or Serverless (below).

EMR Serverless#

EMR Serverless removes cluster provisioning and capacity planning entirely — submit a Spark or Hive job, EMR Serverless allocates and scales workers automatically, billing per vCPU/memory-second actually consumed. 2026 enhancements worth knowing specifically: interactive sessions with Spark Connect ( developing and running Spark applications from a notebook — SageMaker Unified Studio, Jupyter, VS Code — against a live EMR Serverless session rather than only submitting batch jobs), a new 32 vCPU/244 GB worker configuration delivering meaningfully faster execution on shuffle-heavy, multi-table join queries, and full Spark 4.0 support across every EMR deployment option. EMR Serverless has also removed the need to separately provision local worker storage, directly reducing processing cost for storage-heavy jobs.

EMR vs Glue — Choosing#

A team migrating an existing on-premises Hadoop/Spark ecosystem (Part 17) very often lands on EMR specifically because it preserves framework and tooling compatibility that a Glue-only migration would break; a team building new, standard ETL pipelines from scratch very often defaults to Glue for its lower operational overhead.

Amazon Redshift, Revisited in Depth#

Part 6 introduced Redshift briefly as "data warehousing." The deeper picture: Redshift is a columnar, MPP (massively parallel processing) data warehouse — data stored by column rather than by row, which dramatically accelerates the kind of aggregate, scan-heavy analytical queries a warehouse is built for (summing a column across billions of rows) compared to a row-oriented OLTP database like RDS, at the cost of being a poor fit for OLTP's own single-row read/write pattern. This columnar-vs-row distinction is the single most fundamental "why does this warehouse exist separately from our OLTP database" answer, worth having crisp for both the exam and real architecture conversations.

Redshift Serverless vs Provisioned (RA3)#

Both deployment types share RA3's core architectural advantage — storage and compute scale independently, meaning storage growth doesn't force an unnecessary compute upsize the way older, storage-coupled node types once did.

Redshift Spectrum#

Redshift Spectrum queries data sitting directly in S3 — no loading into Redshift's own storage required — joining that external data against tables that are loaded into Redshift in the same query. This is the practical mechanism behind a common real pattern: keep hot, frequently-queried data loaded into Redshift proper for maximum query performance, while cold or rarely-queried historical data stays in S3 and gets queried through Spectrum only when actually needed — avoiding the cost of loading and storing data inside the warehouse that's rarely touched. Worth noting precisely: RA3/DC2 clusters run Spectrum queries on a separate, dedicated fleet outside the cluster's own compute, while Redshift Serverless runs its Spectrum-equivalent data-lake queries on the cluster's own integrated compute — a real architectural difference between the two deployment types, not just a billing distinction.

Zero-ETL Integrations#

A genuinely significant modernization AWS has been building out: zero-ETL integrations, starting with Aurora-to-Redshift, replicate transactional data from a source database into Redshift automatically, in near real-time, via Change Data Capture, with no pipeline to build or maintain and no additional AWS charge for the integration itself. This directly replaces what used to require a hand-built DMS (Part 6/17) or Glue-based CDC pipeline for the specific, common case of "I want my OLTP data queryable in my warehouse with minimal lag" — worth recognizing as the current, actively-expanding-to-more-source-services answer to a problem this part's other tools (Glue, DMS) used to be the only way to solve.

AWS Lake Formation — Centralized Data Governance#

Lake Formation layers centralized governance on top of the Glue Data Catalog, S3, Athena, Redshift Spectrum, and EMR — becoming the single control plane through which data access across the entire data lake is granted, audited, and revoked, rather than managing S3 bucket policies and IAM policies separately per consuming service. Every access request — whether it originates from Athena, Redshift Spectrum, or EMR — passes through both IAM and Lake Formation permission checks; a request only succeeds if it clears both layers, the same dual-gate pattern Part 9 established for KMS (an IAM policy AND a key policy both required).

Lake Formation Fine-Grained Access Control#

Beyond simple table-level grants, Lake Formation's fine-grained access control (FGAC) manages permissions down to the column, row, and cell level — a analyst role might see every column of a customer table except a masked ssn column, or see only rows matching their own region, all enforced centrally at the Lake Formation layer rather than requiring separate, redundant filtering logic built into every consuming query or application. This is the concrete mechanism that makes "comply with data regulations while still enabling broad self-service analytics access" achievable without either locking the data lake down entirely or trusting every individual query author to filter sensitive columns correctly on their own.

Partitioning Strategy in Depth#

Choosing a partition scheme is one of the highest-leverage decisions in the entire platform, worth more depth than a single mention. The standard pattern is Hive-style partitioning — encoding partition values directly into the S3 key prefix (s3://bucket/table/year=2026/month=08/day=27/), which both Athena and Redshift Spectrum understand natively via partition projection (computing valid partitions from a defined pattern rather than requiring an explicit MSCK REPAIR TABLE/crawler run after every new partition appears). The real design tradeoff: partitioning too coarsely (by year only) doesn't meaningfully reduce scan cost for a query filtering by day; partitioning too finely (by minute) produces an enormous number of small files, which hurts performance in the opposite direction (file-open overhead dominating over actual data scanned) — the right granularity matches the actual, dominant query pattern a dataset's consumers use, not an arbitrary default.

Redshift Workload Management and Concurrency Scaling#

For provisioned Redshift clusters specifically, Workload Management (WLM) queues incoming queries into separate resource pools — preventing one team's long-running analytical query from starving a dashboard's fast, latency-sensitive queries of cluster resources, the same "isolate noisy neighbors" instinct behind ECS task placement (Part 7) or RDS read replica traffic splitting (Part 6), applied to a warehouse's query queue. Concurrency Scaling transparently adds temporary, additional cluster capacity during a burst of concurrent queries, then removes it once the burst passes, billed only for the burst duration — a Redshift-specific instance of the same elastic-scaling instinct behind EC2 Auto Scaling (Part 3), applied to query concurrency rather than compute capacity.

Athena Federated Query#

Athena Federated Query extends Athena beyond S3 entirely — via Lambda-based data source connectors, a single Athena query can join data sitting in S3 against a live RDS table, a DynamoDB table, or even a non-AWS data source, without first copying any of it into S3. This is genuinely useful for the common "I need to join my data lake against a live operational table for one ad hoc investigation" case, without standing up a full ETL pipeline just to answer a single question — though it's worth treating as an ad hoc convenience rather than a production pipeline pattern, since federated queries carry real performance overhead compared to querying data that's already sitting natively in S3.

Amazon Athena, Revisited in Depth#

Part 6 introduced Athena as "S3 + Athena for data lakes." The deeper picture: Athena is a serverless, interactive SQL query engine (based on Trino/Presto under the hood) that queries data directly in S3 through the Glue Data Catalog, billed per byte scanned — meaning query cost and performance are both directly driven by how much data a query actually has to read, which is precisely why partitioning (organizing S3 data by a query-relevant column, like date, into separate prefixes) and columnar formats (Parquet, ORC) matter so much for Athena specifically: a well-partitioned, columnar dataset can let a query skip scanning the vast majority of irrelevant data entirely, directly reducing both cost and latency.

Table Formats: Apache Iceberg and the Data Lakehouse#

A genuinely important modernization concept: Apache Iceberg (and similar open table formats like Hudi and Delta Lake) adds database-like capabilities — ACID transactions, schema evolution, time travel (querying data as it existed at a prior point), and efficient upserts/deletes — directly on top of files sitting in S3, without needing to load that data into a traditional warehouse first. This is the technical foundation of the data lakehouse pattern: a single S3-based data lake that behaves like a warehouse for the operations that matter (consistent reads, schema changes, row-level updates) while remaining fundamentally open, cheap object storage underneath — Kinesis's 2025/2026 native Iceberg integration (mentioned above) is a direct, current example of this pattern extending into the streaming-ingestion layer itself, not just batch storage.

OpenSearch Service for Log and Search Analytics#

Worth a specific mention alongside Athena/Redshift: Amazon OpenSearch Service (the managed fork continuing Elasticsearch/Kibana's open-source lineage) fills a different niche — full-text search and near-real-time log analytics, rather than structured SQL analytics over a data lake. Part 10's CloudWatch Logs Insights covers ad hoc log querying within CloudWatch itself; OpenSearch is the right reach specifically when a team needs full-text search capability, custom relevance scoring, or a longer-lived, more richly queryable log/event store than CloudWatch Logs is built to be — Firehose (above) can deliver directly into OpenSearch as one of its native destination types, making "stream events into a searchable index" a config-only pipeline rather than custom integration code.

Amazon QuickSight — BI and Visualization#

QuickSight is AWS's serverless business intelligence service — dashboards, visualizations, and reports built over data from Redshift, Athena, RDS, S3, and third-party sources, with a genuinely different billing model than most BI tools: per-user and per-session pricing based on actual dashboard interaction, rather than a flat per-seat license regardless of usage. Its SPICE in-memory engine caches query results for fast, repeated dashboard interaction without re-querying the underlying data source on every click — the BI-layer equivalent of ElastiCache (Part 6) sitting in front of a database, applied to dashboard query results specifically.

Amazon Q in QuickSight — Generative BI#

Amazon Q in QuickSight adds natural-language Q&A directly against a dashboard's data ("what were our top three products by revenue last quarter"), automatically generated executive summaries, and generative data stories — narrative, automatically-written explanations of what a dashboard's data actually shows, aimed at a business stakeholder who wants the takeaway without manually exploring the underlying visualization themselves. This is the BI-layer instance of the same generative-AI-assisted tooling wave Part 14 already covered for developer tooling (Amazon Q Developer) — a consistent AWS pattern worth recognizing: Q as a family of generative-AI assistants, each specialized for a different surface.

Data Lake Zones: Raw, Curated, and Consumption#

A well-organized data lake typically separates S3 storage into distinct zones, each with its own access controls and purpose: a raw (or "bronze") zone holding data exactly as ingested, immutable and unprocessed — the permanent source of truth a pipeline can always replay from if a downstream transformation bug is discovered; a curated (or "silver") zone holding cleaned, validated, schema-enforced data after transformation; and a consumption (or "gold") zone holding business-level, often pre-aggregated data shaped specifically for a known consuming use case (a QuickSight dashboard, a specific team's reporting need). This zone separation is what makes "we found a bug in our transformation logic three months later" a recoverable situation — since raw data was never mutated, a corrected transformation can simply re-run against it — rather than a permanent, unrecoverable data-quality incident.

Batch vs Streaming: A Unifying Decision Framework#

Pulling this part's ingestion options together into one decision: batch processing (Glue jobs, EMR steps run on a schedule) fits when some latency between an event happening and it being reflected in analytics is genuinely acceptable — hours, not seconds — and batch's simpler operational model and lower cost per byte processed is the right tradeoff. Streaming (Kinesis, MSK, Firehose) fits when a business requirement genuinely needs near-real-time visibility — fraud detection, a live operational dashboard, anything where minutes of latency has real business cost. A common, sensible hybrid: stream raw ingestion into S3 via Firehose (getting data available quickly) while still running batch Glue/EMR transformation jobs against that same data on a schedule for the deeper, less time-sensitive analytical processing — not an either/or choice at the whole-pipeline level, but a per-stage one.

Orchestrating Data Pipelines: Step Functions vs Glue Workflows vs MWAA#

Three genuine options for sequencing a multi-step data pipeline, and picking the right one matters: Step Functions (Part 13), already covered in depth, orchestrates arbitrary AWS service calls including Glue jobs and EMR steps, with the same Standard/Express and error-handling machinery already familiar; Glue Workflows provide native, Glue-specific orchestration for a pipeline that's entirely Glue jobs and crawlers, with less setup than Step Functions for that narrower, Glue-only case; and Amazon MWAA (Managed Workflows for Apache Airflow) runs actual, open-source Apache Airflow, appropriate specifically for a team with an existing Airflow investment (DAGs already written, an Airflow-specific ecosystem of plugins) that wants to preserve that investment rather than rewrite pipelines in Step Functions' ASL or Glue Workflows' own model.

SageMaker Feature Store and the ML/Analytics Boundary#

Worth flagging before Part 19 covers it properly: SageMaker Feature Store sits at the boundary between this part's data platform and Part 19's machine learning material — a centralized repository for curated ML features (derived, model-ready values computed from raw data, like a customer's rolling 30-day purchase count) that both training pipelines and real-time inference can read consistently, avoiding the classic "training used one feature computation, production inference used a subtly different one" bug class. It's built directly on top of this part's data platform (commonly fed by the same Glue/EMR transformation jobs already covered), which is exactly why it's mentioned here rather than left as a complete surprise when Part 19 picks it up in full.

Data Quality and Schema Evolution#

AWS Glue Data Quality evaluates incoming data against defined rules (completeness, uniqueness, value ranges) using Data Quality Definition Language (DQDL), failing or flagging a pipeline run when incoming data doesn't meet expectations — catching a garbage-in problem at the ingestion boundary rather than discovering corrupted downstream analytics after the fact. Schema evolution — a source system adding a new column, or changing a column's type — is handled gracefully by crawlers re-running and updating the catalog, combined with table-format features (Iceberg's schema evolution support, above) that let existing queries keep working against evolved schemas without breaking, rather than a schema change silently corrupting or halting the entire pipeline.

Cross-Account Data Sharing#

A genuinely common real-world requirement: sharing curated data with another team's account, or an external partner, without duplicating it. Lake Formation cross-account sharing grants another AWS account read access to specific catalog tables/databases, backed by AWS Resource Access Manager (Part 15's RAM), the same cross-account resource-sharing mechanism already familiar from Image Builder and CodeArtifact domains. Redshift data sharing, separately, lets a producer cluster share live query access to its own data with a consumer cluster — including across accounts — without physically copying or ETL-ing any data between them. Both mechanisms solve the same underlying problem (avoid data duplication across organizational boundaries) at their respective layers — Lake Formation for the data-lake/catalog layer, Redshift data sharing for the warehouse layer specifically.

Security and Governance for the Data Platform#

Every security pattern this series has already established applies directly to the data platform, not as new material: encryption at rest for S3/Redshift/Glue (Part 9's KMS patterns), IAM least-privilege for every Glue job's execution role and every Athena/Redshift querying principal (Part 2), VPC endpoints for Glue/Athena/Redshift traffic that shouldn't traverse the public internet (Part 4), and CloudTrail logging every catalog and data-access API call for audit (Part 9). Lake Formation's own governance layer, above, doesn't replace any of this — it adds a data-specific, fine-grained authorization layer on top of the same IAM/KMS/VPC foundation already covered.

Cost Optimization for the Data Platform#

Part 16's full toolkit applies directly here, with a few data-platform-specific levers worth naming: partition and compress data (Parquet/ORC plus sensible partitioning directly reduces Athena's per-byte-scanned cost and Redshift Spectrum's scan cost); choose Serverless over Provisioned for variable-workload Redshift/EMR/Glue usage, avoiding paying for idle capacity; right-size EMR clusters using Spot for fault-tolerant stages (the same Spot discipline from Part 3, applied to big-data compute); and set Athena query result reuse/caching where applicable to avoid re-scanning identical data for repeated queries. A data platform's cost profile is unusually sensitive to data layout specifically — the single highest-leverage cost action in this part is very often "partition and compress the data better," not a purchasing-model change.

A Closing Note: This Is the Series' Own Architecture, Applied to Data#

Worth naming explicitly since it's easy to miss while learning each service individually: this part's overall shape — decoupled stages connected through a durable intermediate store, each stage independently scalable and independently replaceable — is the exact same architectural instinct Part 7's event-driven compute and Part 13's EventBridge material already established, just expressed through S3 and the Glue Data Catalog instead of SQS and an event bus. A data platform isn't a fundamentally different kind of system from everything else in this series; it's the same decoupling and single-responsibility principles, applied to a pipeline whose "events" happen to be data files and table partitions rather than application messages.

A Full Worked Example: A Real-Time Order Analytics Pipeline#

A retail platform building real-time order analytics on top of the order-status API from Part 13:

1. Ingest: Order events publish onto Kinesis Data Streams (On-Demand mode, given genuinely spiky traffic around promotional events), with Amazon Data Firehose consuming the same stream and delivering raw events as Parquet into an S3 "raw" prefix, partitioned by date.
2. Catalog: A Glue crawler runs nightly against the raw prefix, keeping the Data Catalog's schema current as new event fields are added over time.
3. Transform: A scheduled Glue Spark job joins raw order events against a product-reference table, producing a cleaned, enriched "curated" dataset in a separate S3 prefix, also Parquet, partitioned by date and region.
4. Govern: Lake Formation grants the analytics team read access to the curated prefix's non-sensitive columns while masking a customer-PII column via column-level FGAC, with every access logged to CloudTrail.
5. Serve — ad hoc: Data analysts query the curated dataset directly via Athena for ad hoc investigation, benefiting directly from the partitioning and Parquet format's reduced byte-scanned cost.
6. Serve — warehouse: The same curated data also zero-ETL-replicates from an Aurora order-summary table into Redshift Serverless for the finance team's recurring, heavier BI workload, queried through QuickSight dashboards with Amazon Q enabled for natural-language exploration.
7. Orchestrate: The nightly crawler-then-transform sequence runs as a Step Functions Standard workflow (Part 13), with a CloudWatch alarm (Part 10) on job failure feeding the same OpsCenter (Part 15) incident path as every other operational alert in the account.
8. Zone separation: The raw Parquet prefix from step 1 is treated as immutable — when a bug is later found in the product-reference join logic, the fix is simply a corrected Glue job re-run against the still-intact raw data, not a data-recovery incident.
9. Data quality: A Glue Data Quality rule flags any incoming event missing a required order_id field, routing the flagged batch to a quarantine prefix for manual review rather than silently polluting the curated dataset downstream.
10. Cost review: A Part 16-style monthly review confirms Athena's scanned-byte cost stayed flat despite a 40% growth in raw event volume — direct evidence the partitioning and Parquet-format choices from step 1 are doing their job as the pipeline scales.

Data Platform Best Practices — The Consolidated Checklist#

• Treat the Glue Data Catalog as the single source of truth for schema — avoid multiple engines maintaining their own separate, potentially drifting metadata.
• Partition and use columnar formats (Parquet/ORC) for anything queried via Athena or Redshift Spectrum — this is the highest-leverage cost and performance lever in the whole platform.
• Choose Kinesis Data Streams only when multiple independent, replay-capable consumers are genuinely needed; default to Firehose for simple delivery.
• Default to Glue for standard ETL; reach for EMR specifically when an existing framework ecosystem or Hadoop/Spark-specific tooling investment demands it.
• Prefer Serverless deployment options (Redshift Serverless, EMR Serverless, Glue) for variable workloads; reserve Provisioned/RA3 for steady, well-understood, high-utilization workloads.
• Route all data-lake access through Lake Formation's governance layer rather than managing S3/IAM policies separately per consuming service.
• Evaluate zero-ETL integrations before building a custom CDC pipeline for a supported source/target pair.
• Define Data Quality rules at the ingestion boundary, not just downstream — catch bad data before it reaches analytics, not after.
• Keep a raw, immutable zone in the data lake — a corrected transformation should always be able to re-run against original data, not just a mutated copy.
• Register schemas via Glue Schema Registry for high-throughput streaming data, so a producer's silent shape change fails fast rather than corrupting downstream consumers.
• Use Job Bookmarks for recurring Glue jobs against a growing source, rather than reprocessing the full dataset on every run.
• Choose partition granularity by the actual dominant query pattern, not an arbitrary default — too coarse wastes scan cost, too fine produces excessive small files.

A Note on Data Contracts#

A final, forward-looking practice worth naming: a data contract formalizes the agreement between a data producer and its consumers — the schema, update frequency, and quality guarantees a dataset commits to — as an explicit, versioned artifact rather than an implicit assumption discovered only when something breaks. The Glue Schema Registry (streaming) and Data Quality rules (batch) covered above are the concrete enforcement mechanisms; a data contract is the organizational practice of writing that agreement down and treating a breaking change to it as a deliberate, communicated decision — the same API-versioning discipline Part 13 established for public APIs, applied here to internal data interfaces between teams.

Part 18 CLI Cheat Sheet#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team's Athena queries against a 500 GB dataset of unpartitioned JSON files are slow and expensive, and query cost scales with roughly the full dataset size even when a query only needs one day's data. What's the fix, and why does it address both problems simultaneously?

Answer: Convert the data to a columnar format (Parquet) and partition it by date. Both changes attack the same root cause — Athena bills and performs based on bytes actually scanned. Partitioning by date lets a query targeting one day skip scanning every other day's data entirely; columnar format lets a query reading only specific columns skip scanning irrelevant columns within the data it does read. Together they can reduce both cost and latency by orders of magnitude for a typical filtered, column-selective query.

Problem 2: A team needs three independent applications to each process the same stream of incoming order events, at their own pace, with the ability for one of them to reprocess the last 24 hours of events after a bug fix. Would Firehose or Kinesis Data Streams fit better, and why?

Answer: Kinesis Data Streams. Firehose delivers and forgets — it has no concept of multiple independent consumers or replay. Kinesis Data Streams' retention window (extendable well past 24 hours) and support for multiple independent consumer applications, each tracking its own read position, directly provide both requirements: independent per-application pacing and the ability to reprocess recent history after a fix, neither of which Firehose's simple delivery model supports.

Problem 3: A data platform team wants an analyst role to query a customer table freely for aggregate analysis, but must ensure that role never sees the raw email or phone columns, without maintaining separate, redundant filtering logic in every tool (Athena, a BI dashboard, an ad hoc notebook) that might query the table. What's the correct mechanism?

Answer: Lake Formation column-level fine-grained access control, applied once at the governance layer rather than in each individual consuming tool. Every access path — Athena, Redshift Spectrum, EMR — passes through the same Lake Formation permission check, so masking the sensitive columns centrally guarantees consistent enforcement regardless of which specific tool an analyst happens to query through, without duplicating the filtering logic per tool.

Problem 4: A recurring Glue job processing a daily-growing S3 prefix takes noticeably longer and costs more DPU-hours every month, even though the actual amount of genuinely new data each day stays roughly constant. What's the likely misconfiguration, and what fixes it?

Answer: Job Bookmarks are likely disabled or not functioning, causing the job to reprocess the entire growing dataset from scratch on every run instead of only the new data since the last successful run. Enabling and correctly configuring Job Bookmarks (ensuring the job's transformation code doesn't inadvertently disable bookmark tracking) restores the expected behavior — each run's cost and duration should track the volume of genuinely new data, not the ever-growing total.

Summary and What's Next#

Data flows through five stages — ingest, store, catalog, transform, serve — and nearly every service in this part occupies exactly one of them: Kinesis/Firehose/MSK for ingestion, S3 for storage, the Glue Data Catalog plus Lake Formation for cataloging and governance, Glue/EMR for transformation, and Athena/ Redshift/QuickSight for serving. Zero-ETL integrations and open table formats like Iceberg represent the current frontier of this space — reducing custom pipeline code and blurring the historical line between a data lake and a data warehouse into the emerging lakehouse pattern. Every security, cost, and operational discipline this series has already built — IAM least privilege, KMS encryption, Serverless-by-default cost optimization, CloudTrail auditing — applies to this platform directly, not as separate new material.

Part 19, the final part of this series, covers machine learning and AI on AWS: SageMaker's full training/deployment/pipeline lifecycle, Bedrock and generative AI, and the pre-built AI services (Comprehend, Rekognition, Textract) — completing this series' certification coverage with MLA-C01's full scope. The Feature Store bridge mentioned above is the direct thread connecting this part's data platform to that next one — the curated, governed data built here is precisely what feeds a well-run ML pipeline, rather than the two being separate, disconnected disciplines.