Part 10 of 1224 min read · 5 diagramsAI-assisted

Monitoring, Logging & Tracing

A note on scope: The Three Pillars of Observability, Prometheus/Grafana, distributed tracing concepts, and alerting design already received exhaustive treatment in the Observability series. This part covers AWS's OWN native implementation of those same pillars — CloudWatch, CloudWatch Logs, and X-Ray — as concrete services with their own APIs, pricing models, and operational quirks.

Table of Contents#

CloudWatch — AWS's Native Observability Platform
CloudWatch Metrics — Namespaces, Dimensions, and Resolution
Standard vs Custom Metrics
Publishing Custom Metrics
CloudWatch Alarms
Composite Alarms
CloudWatch Dashboards
CloudWatch Logs — Core Concepts
The CloudWatch Agent
CloudWatch Logs Insights — Querying Logs at Scale
Log Aggregation Patterns — Centralizing Multi-Account Logs
Metric Filters — Turning Logs Into Metrics
X-Ray — Distributed Tracing on AWS
X-Ray Sampling
Instrumenting an Application for X-Ray
CloudWatch Synthetics — Proactive Canary Testing
CloudWatch RUM — Real User Monitoring
Alerting and Notification: SNS Integration
Third-Party Observability on AWS: The OpenTelemetry Path
Applying SLOs and Burn-Rate Alerting on AWS
A Full Worked Example: Observability for a Three-Tier Application
CloudWatch Anomaly Detection
Container and Lambda-Specific Observability
Observability Best Practices — The Consolidated Checklist
Part 10 CLI Cheat Sheet
Common Mistakes
Worked Practice Problems
Summary and What's Next

CloudWatch — AWS's Native Observability Platform#

CloudWatch is AWS's own implementation of the Three Pillars of Observability already covered in exhaustive depth in the Observability series (Part 1) — metrics, logs, and (via X-Ray) traces, all natively integrated with every other AWS service already covered throughout this entire series.

Diagram

Every AWS service already covered in this series — EC2 (Part 3), RDS (Part 6), Lambda (Part 7), ALB (Part 8) — automatically publishes metrics to CloudWatch with zero setup required, directly the same "observability should be a default, not an afterthought" philosophy already established in the Observability series.

CloudWatch Metrics — Namespaces, Dimensions, and Resolution#

# Retrieve a standard, automatically-published metric —
# EC2 CPU utilization, already referenced for Auto Scaling
# target-tracking policies in Part 3
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time 2026-08-19T00:00:00Z --end-time 2026-08-19T01:00:00Z \
  --period 300 --statistics Average Maximum

Concept	Meaning
Namespace	A top-level grouping (e.g. `AWS/EC2`, `AWS/RDS`, or a custom namespace like `MyApp/Orders`)
Dimension	A key-value pair identifying WHICH specific resource a data point belongs to (e.g. `InstanceId=i-abc123`)
Resolution	Standard (60-second) or High-Resolution (1-second) — directly a cost/granularity tradeoff

Why namespaces and dimensions together directly implement the "metric labels" concept already covered for Prometheus in the Observability series (Part 1), worth stating explicitly: a CloudWatch metric like CPUUtilization in namespace AWS/EC2 with dimension InstanceId=i-abc123 is structurally the same idea as a Prometheus metric cpu_utilization{instance="i-abc123"} — different vendor syntax, identical underlying concept of a metric name plus a set of identifying key-value labels.

Standard vs Custom Metrics#

AWS services publish standard metrics automatically (free, at standard resolution). Applications can also publish their OWN custom metrics — business-specific numbers CloudWatch has no way to know about on its own.

aws cloudwatch put-metric-data \
  --namespace "MyApp/Orders" \
  --metric-data MetricName=OrdersProcessed,Value=1,Unit=Count,Dimensions=[{Name=Environment,Value=production}]

Why custom metrics matter, worth stating explicitly, directly connecting to the Golden Signals discussion in the Monitoring Methodologies series: standard AWS metrics tell you about infrastructure health (CPU, memory, request count) — but the actual BUSINESS-level Rate/Errors/Duration signals (orders processed per minute, checkout failure rate) can only come from custom metrics your own application code explicitly publishes, since AWS has no inherent way to know what "a successful order" means to your specific business.

Publishing Custom Metrics#

# The Embedded Metric Format (EMF) — publish custom metrics
# simply by writing structured JSON to CloudWatch Logs (much
# cheaper and lower-latency than direct PutMetricData API calls
# for high-volume custom metrics, especially from Lambda, Part 7)
cat <<'EMF'
{
  "_aws": {
    "Timestamp": 1755590400000,
    "CloudWatchMetrics": [{
      "Namespace": "MyApp/Orders",
      "Dimensions": [["Environment"]],
      "Metrics": [{"Name": "OrdersProcessed", "Unit": "Count"}]
    }]
  },
  "Environment": "production",
  "OrdersProcessed": 1
}
EMF

Why EMF is worth knowing as the modern, preferred approach over direct PutMetricData API calls for high-volume custom metrics, worth stating precisely: writing structured JSON to stdout/CloudWatch Logs is essentially free and adds no additional API call latency to the application's hot path — CloudWatch automatically extracts the embedded metrics from the log data asynchronously, avoiding both the cost and the latency of a synchronous PutMetricData call on every single business event.

CloudWatch Alarms#

Directly the AWS-native implementation of the alerting concepts already covered in exhaustive depth in the Observability series (Part 3) — worth seeing the exact CLI shape.

aws cloudwatch put-metric-alarm \
  --alarm-name high-error-rate \
  --namespace "MyApp/Orders" --metric-name CheckoutErrors \
  --statistic Sum --period 300 --evaluation-periods 2 \
  --threshold 50 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:pagerduty-critical \
  --treat-missing-data notBreaching

Why --treat-missing-data notBreaching deserves an explicit callout, a genuinely common, subtle gotcha worth knowing precisely: by default, CloudWatch's handling of MISSING data points (e.g. no data published during a deploy or a metrics-pipeline hiccup) can itself trigger or clear an alarm in ways that surprise people — explicitly setting this behavior (missing data should NOT be treated as breaching the threshold, in most cases) avoids false alarms fired purely because of a metrics gap, not an actual problem.

Composite Alarms#

For genuinely reducing alert fatigue (directly the concrete AWS implementation of the alert-fatigue discussion from the Observability series, Part 3) — a Composite Alarm combines MULTIPLE underlying alarms with AND/OR logic into ONE higher-level alarm.

aws cloudwatch put-composite-alarm \
  --alarm-name real-customer-impact \
  --alarm-rule "ALARM(high-error-rate) AND ALARM(high-latency) AND NOT ALARM(scheduled-maintenance-window)" \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:pagerduty-critical

Why this directly reduces alert fatigue, worth stating explicitly: instead of paging on-call separately for "errors are up" AND "latency is up" (which are very likely the SAME underlying incident, just two different symptoms), a composite alarm fires ONE page only when BOTH conditions are true together — directly the same symptom-based-not-cause-based alerting philosophy already covered in the Observability series (Part 3), reducing duplicate, correlated pages during a single real incident.

CloudWatch Dashboards#

aws cloudwatch put-dashboard --dashboard-name production-overview --dashboard-body '{
  "widgets": [{
    "type": "metric", "x": 0, "y": 0, "width": 12, "height": 6,
    "properties": {"metrics": [["AWS/ApplicationELB", "RequestCount", "LoadBalancer", "app/app-alb/abc123"]], "period": 300, "stat": "Sum", "title": "Request Count"}
  }]
}'

Directly the AWS-native equivalent of the Grafana dashboards already covered in the Observability series (Part 1) — worth recognizing the same "layered dashboard architecture" principle (Monitoring Methodologies series, Part 3) applies here too: a high-level, business-metric dashboard for leadership, and deeper, infrastructure-metric dashboards for on-call engineers.

CloudWatch Logs — Core Concepts#

Diagram

aws logs create-log-group --log-group-name /app/production
aws logs put-retention-policy --log-group-name /app/production --retention-in-days 90

# Tail logs in real time (genuinely useful during an incident,
# directly the AWS-native equivalent of `kubectl logs -f`
# already covered in the Kubernetes Deep Dive series)
aws logs tail /app/production --follow

Why explicitly setting a retention policy matters, worth stating explicitly, directly connecting to the cost-optimization/FinOps theme carried across this series: CloudWatch Logs retains data INDEFINITELY by default unless a retention policy is explicitly set — a genuinely common, silent cost driver where old, rarely-needed logs accumulate storage charges forever, unnoticed, until a cost review surfaces it.

The CloudWatch Agent#

Standard EC2 metrics (CPU, network, disk I/O) are published automatically — but MEMORY utilization, disk SPACE usage, and custom application logs require installing the CloudWatch Agent on the instance.

# Install and configure the agent (via SSM, Part 3's
# fleet-wide command execution pattern)
aws ssm send-command \
  --targets "Key=tag:Environment,Values=production" \
  --document-name "AmazonCloudWatch-ManageAgent" \
  --parameters 'action=configure,mode=ec2,optionalConfigurationSource=ssm,optionalConfigurationLocation=AmazonCloudWatch-linux-config'

Why "memory isn't a standard EC2 metric" is worth knowing precisely, a genuinely common, real gotcha: AWS's hypervisor (Part 3's Nitro System) has no direct visibility into what's happening INSIDE the guest OS's memory — only something running WITHIN the instance itself (the CloudWatch Agent) can actually observe and report memory utilization, which is exactly why it doesn't appear as a standard, zero-setup metric the way CPU utilization does.

CloudWatch Logs Insights — Querying Logs at Scale#

Directly extending the log-querying concepts already covered generically in the Observability series (Part 1) — CloudWatch's own purpose-built query language for searching across potentially massive log volumes.

aws logs start-query \
  --log-group-name /app/production \
  --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | stats count() by bin(5m)'

aws logs get-query-results --query-id abc-123-def-456

Why Logs Insights is genuinely worth knowing as distinct from a simple grep-style search, worth stating explicitly: it's a real query language with aggregation (stats count() by ...), filtering, and time-bucketing built in, letting an on-call engineer answer questions like "how did the error rate trend over the last hour, broken down by 5-minute buckets" directly against raw log data, without needing to first extract that as a proper metric (Metric Filters, next section) ahead of time.

Log Aggregation Patterns — Centralizing Multi-Account Logs#

Directly extending the centralized-logging pattern already established for CloudTrail in Part 9, and the dedicated log-archive account from Part 1's landing zone.

# Subscribe a log group to stream its data to a CENTRALIZED
# destination (e.g. Kinesis Data Firehose, delivering to S3
# in a dedicated logging account, or directly to a third-party
# tool like Datadog/Splunk)
aws logs put-subscription-filter \
  --log-group-name /app/production \
  --filter-name centralize-logs \
  --filter-pattern "" \
  --destination-arn arn:aws:firehose:us-east-1:123456789012:deliverystream/central-logs

Why centralizing application logs (not just CloudTrail) into the same dedicated logging account matters, worth stating explicitly, directly reinforcing Part 1's tamper-evident logging discussion: a compromised workload account's local CloudWatch Logs could theoretically be tampered with or deleted by an attacker with sufficient access — streaming logs continuously to a separate, centrally-controlled destination the workload account can't delete from provides the same tamper-evidence guarantee already established for audit logs, now extended to application logs as well.

Metric Filters — Turning Logs Into Metrics#

A genuinely useful bridge between the Logs and Metrics pillars: a Metric Filter automatically extracts a metric from matching log lines, in real time, as they're ingested.

aws logs put-metric-filter \
  --log-group-name /app/production \
  --filter-name error-count \
  --filter-pattern "ERROR" \
  --metric-transformations metricName=AppErrorCount,metricNamespace=MyApp,metricValue=1

Why this is worth using specifically when an application logs errors but doesn't already publish a dedicated error-count METRIC, worth stating explicitly: it retroactively creates a genuine, alarmable CloudWatch metric from existing log data, with no application code change required — a fast, practical way to get proper alerting (this part's Alarms section) on a signal that currently only exists as unstructured log text.

X-Ray — Distributed Tracing on AWS#

Directly the AWS-native implementation of the distributed tracing concepts — traces, spans, the trace tree, context propagation — already covered in exhaustive depth in the Observability series (Part 2).

Diagram

aws xray get-trace-summaries \
  --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \
  --filter-expression 'responsetime > 1'

Why "segment" and "subsegment" are worth mapping directly onto the "span" vocabulary already established in the Observability series, worth stating explicitly: an X-Ray segment is functionally the same concept as a span — a named, timed unit of work with a parent-child relationship to other segments/spans, forming the same trace tree structure already covered generically, just with AWS-specific vocabulary and native integration with services like Lambda, API Gateway, and DynamoDB.

X-Ray Sampling#

Directly extending the sampling-strategy discussion already covered in depth in the Observability series (Part 2) — tracing EVERY single request at real production scale is both expensive and often unnecessary.

aws xray create-sampling-rule --sampling-rule '{
  "RuleName": "default-sampling",
  "Priority": 1000,
  "FixedRate": 0.05,
  "ReservoirSize": 1,
  "ServiceName": "*", "ServiceType": "*", "Host": "*", "HTTPMethod": "*", "URLPath": "*",
  "ResourceARN": "*"
}'

Why ReservoirSize combined with FixedRate is worth understanding precisely, a genuinely important nuance: the reservoir guarantees a MINIMUM number of requests are always traced per second (ensuring low-traffic services still get SOME visibility), while the fixed rate applies a PERCENTAGE on top of that for everything beyond the reservoir — directly the same "guarantee a baseline, then sample the rest" strategy already covered generically in the Observability series' sampling-strategies comparison.

Instrumenting an Application for X-Ray#

# The X-Ray SDK wraps outgoing calls (to DynamoDB, S3, HTTP
# APIs) automatically, once instrumented — directly the
# "automatic vs manual instrumentation" distinction already
# covered generically in the Observability series (Part 2)

from aws_xray_sdk.core import xray_recorder, patch_all
patch_all()  # automatically instruments boto3, requests, etc.

@xray_recorder.capture('process_order')
def process_order(order_id):
    # this function, and everything it calls via a patched
    # library, now automatically appears in the trace
    ...

Why patch_all() matters, worth stating explicitly: it's the concrete, AWS-SDK-specific realization of "automatic instrumentation" — every call the application makes through a patched library (boto3 for AWS services, requests for HTTP calls) is automatically wrapped as an X-Ray subsegment, with zero manual span-creation code needed for the vast majority of an application's actual external calls.

CloudWatch Synthetics — Proactive Canary Testing#

A genuinely distinct, worth-knowing capability: rather than waiting for REAL user traffic to reveal a problem, Synthetics runs scripted "canary" requests against your application on a schedule, from multiple locations, proactively.

aws synthetics create-canary \
  --name checkout-flow-canary \
  --code '{"Handler":"index.handler","S3Bucket":"canary-scripts","S3Key":"checkout-canary.zip"}' \
  --artifact-s3-location s3://canary-results/checkout-flow/ \
  --execution-role-arn arn:aws:iam::123456789012:role/CanaryExecutionRole \
  --schedule Expression="rate(5 minutes)" \
  --runtime-version syn-python-selenium-1.3

Why "proactive, not reactive" is the key distinguishing property, worth stating explicitly, a genuinely important complement to the Real User Monitoring approach covered next: a canary catches a broken checkout flow within 5 minutes of it breaking, EVEN IF real user traffic to that specific flow happens to be low at that exact moment — waiting for enough real users to organically encounter and report the same problem could take considerably longer, directly reducing the detection-time portion of MTTR already discussed in the Incident Management series.

CloudWatch RUM — Real User Monitoring#

The complementary, reactive counterpart to Synthetics: RUM captures actual performance and error data directly from REAL users' browsers, as they use the application.

aws rum create-app-monitor \
  --name production-frontend \
  --domain example.com \
  --app-monitor-configuration '{"AllowCookies":true,"SessionSampleRate":0.1,"Telemetries":["errors","performance","http"]}'

Why RUM and Synthetics are worth using TOGETHER, not as alternatives, worth stating explicitly: Synthetics tells you a SPECIFIC, scripted flow is (or isn't) working, proactively, regardless of real traffic volume; RUM tells you what REAL users are ACTUALLY experiencing, across every flow, but only surfaces a problem once real users encounter it — together they cover both the "did we break something before users noticed" and "what are users actually experiencing" questions, neither of which the other can fully answer alone.

CloudWatch Alarms don't directly send Slack messages or pages — they publish to SNS (Simple Notification Service), which then fans out to actual notification channels, a preview of the fuller messaging discussion in Part 11.

aws sns create-topic --name pagerduty-critical
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:pagerduty-critical \
  --protocol https --notification-endpoint https://events.pagerduty.com/integration/abc123/enqueue

Why this decoupled design matters, worth stating explicitly, directly reusing the pub/sub pattern that Part 11 covers in full depth: CloudWatch doesn't need to know anything about PagerDuty, Slack, or email specifically — it just publishes to an SNS topic, and any number of independent subscribers (a PagerDuty integration, a Lambda function, an SQS queue) can react to that same alarm, without CloudWatch or the alarm configuration itself needing to change as notification channels are added or changed.

Third-Party Observability on AWS: The OpenTelemetry Path#

Worth an honest, balanced note: many organizations use CloudWatch/X-Ray as their PRIMARY AWS-native observability stack, while others standardize on the vendor-neutral OpenTelemetry (already covered in depth in the Observability series, Part 2) and ship data to a third-party backend (Datadog, Grafana Cloud, Honeycomb) instead — or BOTH simultaneously.

# AWS Distro for OpenTelemetry (ADOT) — AWS's own supported
# OTel distribution, capable of exporting to EITHER
# CloudWatch/X-Ray OR a third-party backend, or both at once

Why this choice is a genuinely real, worth-naming-explicitly tradeoff: CloudWatch/X-Ray offers the tightest native integration and zero additional vendor relationship, while OpenTelemetry (via ADOT) offers vendor neutrality and portability if the organization might switch observability backends later, or already has multi-cloud/on-premises infrastructure that needs the SAME instrumentation approach as its AWS workloads — neither is universally correct, and many mature organizations use OpenTelemetry as the instrumentation LAYER while still exporting primarily to CloudWatch/X-Ray as the actual backend.

Applying SLOs and Burn-Rate Alerting on AWS#

Directly the AWS-concrete implementation of the SLI/SLO/SLA and multi-window burn-rate alerting concepts already covered in exhaustive depth in the SRE Fundamentals series (Parts 1-2) and the Observability series (Part 3).

# A composite alarm implementing a simplified two-window
# burn-rate check, directly reusing the SRE Fundamentals
# series' burn-rate math, now expressed as real CloudWatch alarms
aws cloudwatch put-metric-alarm \
  --alarm-name slo-burn-rate-fast \
  --namespace "MyApp/SLO" --metric-name ErrorBudgetBurnRate \
  --statistic Average --period 300 --evaluation-periods 2 \
  --threshold 14.4 --comparison-operator GreaterThanThreshold

Why publishing burn rate as its own CUSTOM metric (rather than trying to express the full burn-rate formula inside a single native CloudWatch alarm) is the practical, real-world pattern worth knowing: burn-rate calculation (good events ÷ total events, compared against the SLO target, over a specific window) is genuinely business logic — a small scheduled Lambda function (Part 7) computing it from the underlying SLI metrics and publishing the RESULT as a custom metric (this part's earlier section) is the standard way to bridge the SRE Fundamentals series' math into a real, alarmable CloudWatch signal.

A Full Worked Example: Observability for a Three-Tier Application#

Bringing this entire part together into one concrete, complete observability setup.

Diagram

Every arrow in this diagram maps to a specific section already covered — worth narrating end to end as a single, coherent answer to "design observability for a production AWS application."

CloudWatch Anomaly Detection#

Instead of a fixed, hand-picked threshold (> 500ms), CloudWatch Anomaly Detection uses machine learning to build an expected range for a metric based on its own historical pattern — including normal daily/weekly seasonality — and alarms when the metric falls genuinely OUTSIDE that learned band.

Diagram

aws cloudwatch put-anomaly-detector \
  --namespace "MyApp/Orders" --metric-name RequestLatency --stat Average

aws cloudwatch put-metric-alarm \
  --alarm-name latency-anomaly \
  --namespace "MyApp/Orders" --metric-name RequestLatency \
  --statistic Average --period 300 --evaluation-periods 3 \
  --comparison-operator GreaterThanUpperThreshold \
  --threshold-metric-id ad1 \
  --metrics '[{"Id":"m1","MetricStat":{"Metric":{"Namespace":"MyApp/Orders","MetricName":"RequestLatency","Dimensions":[]},"Period":300,"Stat":"Average"},"ReturnData":true},{"Id":"ad1","Expression":"ANOMALY_DETECTION_BAND(m1, 2)","ReturnData":true}]'

Why this directly addresses a genuinely common threshold-tuning problem worth stating explicitly, connecting to the alert-design discipline from the Observability series: a fixed threshold that's correctly tuned for quiet periods will false-alarm constantly during a known, normal daily traffic peak — and a threshold loose enough to tolerate the peak becomes too insensitive during quiet periods to catch a genuine regression. Anomaly detection sidesteps this entirely by learning the metric's OWN normal seasonal shape, rather than requiring a human to hand-tune one static number that has to work across every hour of every day.

Container and Lambda-Specific Observability#

Worth a brief, explicit tie-back to Part 7, since containers and serverless compute have their own specific observability nuances beyond standard EC2 metrics.

# Lambda automatically publishes Duration, Errors, Throttles,
# and ConcurrentExecutions — worth alarming on Throttles
# specifically, since it directly signals the reserved/account
# concurrency limits from Part 7 being hit
aws cloudwatch put-metric-alarm \
  --alarm-name lambda-throttling \
  --namespace AWS/Lambda --metric-name Throttles \
  --dimensions Name=FunctionName,Value=process-upload \
  --statistic Sum --period 60 --evaluation-periods 1 \
  --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold

# Container Insights (Part 7) automatically publishes per-task
# CPU/memory metrics for ECS, without any CloudWatch Agent
# installation needed on Fargate (which has no accessible
# host to install an agent on in the first place)
aws cloudwatch get-metric-statistics \
  --namespace ECS/ContainerInsights --metric-name CpuUtilized \
  --dimensions Name=ClusterName,Value=production-cluster Name=ServiceName,Value=my-app-service \
  --start-time $(date -d '1 hour ago' -Iseconds) --end-time $(date -Iseconds) --period 300 --statistics Average

Why alarming on Lambda Throttles specifically matters, worth stating explicitly, directly connecting to Part 7's concurrency discussion: a throttled invocation means a request was REJECTED outright because the function's concurrency limit (reserved or account-wide) was already exhausted — this is a distinct, more urgent failure mode than a slow response, and deserves its own dedicated alarm rather than being buried inside a general error-rate metric.

Observability Best Practices — The Consolidated Checklist#

Set explicit CloudWatch Logs retention on every log group — the default is indefinite, silent, growing cost.
Install the CloudWatch Agent wherever memory or disk-space visibility is needed — these are never standard, zero-setup EC2 metrics.
Use Composite Alarms to combine correlated symptoms into a single page, directly reducing alert fatigue.
Publish high-volume custom metrics via Embedded Metric Format, not synchronous PutMetricData calls, to avoid adding latency to the application's hot path.
Configure X-Ray sampling deliberately (reservoir + rate) rather than tracing every request at real production scale.
Pair Synthetics (proactive) with RUM (reactive) — neither alone answers both "did we break something" and "what are users actually experiencing."
Consider Anomaly Detection for metrics with known daily/weekly seasonality, where a single fixed threshold structurally can't work well across the full cycle.
Alarm on Lambda Throttles specifically, not just general error rate — it signals a distinct, more urgent concurrency-limit failure mode.

Part 10 CLI Cheat Sheet#

Area	Command	Purpose
Metrics	`aws cloudwatch get-metric-statistics`	Retrieve metric data
Metrics	`aws cloudwatch put-metric-data`	Publish a custom metric
Alarms	`aws cloudwatch put-metric-alarm`	Create a standard alarm
Alarms	`aws cloudwatch put-composite-alarm`	Combine multiple alarms
Dashboards	`aws cloudwatch put-dashboard`	Create a dashboard
Logs	`aws logs create-log-group` / `put-retention-policy`	Create a log group with retention
Logs	`aws logs tail --follow`	Real-time log tailing
Logs	`aws logs start-query` / `get-query-results`	Run a Logs Insights query
Logs	`aws logs put-metric-filter`	Extract a metric from log lines
Logs	`aws logs put-subscription-filter`	Stream logs to a central destination
Tracing	`aws xray get-trace-summaries`	Query recent traces
Tracing	`aws xray create-sampling-rule`	Configure trace sampling
Synthetics	`aws synthetics create-canary`	Create a proactive canary test
RUM	`aws rum create-app-monitor`	Enable real user monitoring
Notification	`aws sns create-topic` / `subscribe`	Wire alarms to notification channels

Common Mistakes#

Mistake	Why It's Wrong	Fix
Assuming EC2 memory utilization is a standard, automatic metric	AWS's hypervisor has no visibility inside the guest OS's memory	Install the CloudWatch Agent for memory/disk-space visibility
Never setting a CloudWatch Logs retention policy	Logs accumulate indefinitely by default, becoming a silent, growing cost	Set an explicit retention period matching actual operational and compliance needs
Alarming separately on every correlated symptom of the same incident	Causes duplicate, alert-fatigue-inducing pages for what's really one problem	Use Composite Alarms to combine correlated symptoms into a single, higher-confidence page
Tracing 100% of requests at real production scale	Unnecessarily expensive, and rarely adds proportional diagnostic value beyond a well-tuned sample	Configure X-Ray sampling rules with a sensible reservoir + fixed rate
Relying only on Real User Monitoring, with no proactive canary testing	A low-traffic flow can stay broken for a long time before enough real users encounter and report it	Use Synthetics canaries for critical flows, alongside RUM
Publishing high-volume custom metrics via synchronous `PutMetricData` calls on the application's hot path	Adds real API call latency and cost to every business event	Use the Embedded Metric Format (EMF) via structured log lines instead
Not setting `treat-missing-data` explicitly on alarms	A metrics-pipeline gap (e.g. during a deploy) can itself trigger a false alarm	Explicitly configure missing-data behavior appropriate to each alarm

Worked Practice Problems#

Problem 1: A team's CloudWatch alarm for CPU utilization on a critical instance fires unexpectedly during every deployment window, even though the deployment itself causes no actual CPU issue. Investigation shows the instance briefly stops publishing metrics during the deployment's instance-replacement window. What's the likely cause, and what's the fix?

Answer: The alarm's missing-data behavior is very likely defaulting to treating a data gap as breaching the alarm threshold, rather than being explicitly configured to ignore it — during the brief window where an instance is being replaced as part of a rolling deployment (Automation series), no CPU metric data is published at all, and depending on the alarm's missing-data handling, this gap itself can trigger a false alarm completely unrelated to any real CPU problem. The fix is explicitly setting --treat-missing-data notBreaching (or an appropriately chosen alternative) on the alarm, so a genuine metrics gap during expected operational events like deployments doesn't produce a false, unnecessary page.

Problem 2: A team's application logs errors to CloudWatch Logs, but has no dedicated alarm for error rate, relying instead on engineers noticing rising error counts when manually browsing logs. They want proper, automated alerting on this without any application code changes. What's the fastest path to achieve this?

Answer: A CloudWatch Logs Metric Filter, configured to match the relevant error pattern (e.g. lines containing "ERROR") and extract a count into a real CloudWatch metric, with zero application code changes required — the filter operates purely on the existing log stream. Once that metric exists, a standard CloudWatch Alarm can be configured against it exactly as with any other metric, closing the loop from "errors exist somewhere in unstructured log text" to "a proper, alarmable signal with automated paging" without touching the application itself.

Problem 3: A platform team wants to reduce the mean time to detect (a component of MTTR, per the Incident Management series) for a critical checkout flow that has relatively low, inconsistent traffic — meaning Real User Monitoring alone might take a while to accumulate enough real user sessions to reliably surface a regression. What AWS service directly addresses this gap, and how would you configure it?

Answer: CloudWatch Synthetics, configured with a scripted canary that exercises the checkout flow end to end on a fixed schedule (e.g. every 5 minutes), regardless of actual real user traffic volume. Because Synthetics runs proactively on its own schedule rather than waiting for organic user traffic, it directly closes the detection-time gap RUM alone would leave for a low-traffic flow — a regression introduced by a bad deploy would be caught within one canary interval (a few minutes) instead of however long it takes enough real users to encounter and report the same problem, which could be considerably longer for a flow with genuinely low, inconsistent traffic.

Problem 4: A team sets a fixed-threshold CloudWatch alarm on API latency (> 800ms), tuned to avoid false alarms during their known daily traffic peak (11am-1pm, when latency naturally runs higher). During quiet overnight hours, a genuine regression pushes latency to 600ms — a significant, real degradation for that time of day — but the alarm never fires, since 600ms is still under the 800ms threshold. What's the structural limitation causing this missed detection, and what AWS feature directly addresses it?

Answer: A single fixed threshold cannot simultaneously be tight enough to catch a real regression during quiet periods (where 600ms is genuinely abnormal) AND loose enough to avoid false alarms during the known daily peak (where 800ms might be entirely normal) — any one static number necessarily trades sensitivity in one period against false alarms in the other, and this team has explicitly tuned toward avoiding peak-time false alarms at the cost of missing quiet-period regressions. CloudWatch Anomaly Detection directly addresses this by learning the metric's own normal daily/weekly seasonal pattern and alarming on genuine deviation from that LEARNED band, rather than one static number — it would correctly recognize that 600ms overnight is anomalous even though the same value would be entirely normal at 12pm, catching exactly the regression this team's fixed threshold missed.

Problem 5: A team using Lambda functions notices customer complaints about failed requests during a traffic spike, but their existing CloudWatch alarm — configured on the function's general Errors metric — never fired, since the requests were being rejected before the function code ever ran. What Lambda-specific metric should have been monitored instead, and why does it represent a meaningfully different failure mode than Errors?

Answer: The Throttles metric, not Errors. A throttled invocation means the request was rejected outright because the function's concurrency limit (either its Reserved Concurrency ceiling or the account-wide concurrency limit, both covered in Part 7) was already exhausted at the moment the request arrived — critically, this happens BEFORE the function's own code ever executes, meaning it can never be reflected in the Errors metric, which only counts failures that occur DURING actual code execution. This is why Throttles deserves its own dedicated alarm: it represents a distinct, capacity-related failure mode (requests being turned away entirely) that a general error-rate alarm structurally cannot detect, since throttled requests never reach the point where an "error" in the traditional sense could even occur.

Summary and What's Next#

CloudWatch is AWS's native implementation of the Three Pillars (metrics, logs, and via X-Ray, traces) already covered in depth in the Observability series — every AWS service in this course publishes standard metrics automatically.
Custom metrics (via PutMetricData or the more efficient Embedded Metric Format) are required for business-level Golden Signals that AWS has no inherent way to know about.
Composite Alarms directly implement the symptom-based, alert-fatigue-reducing philosophy already covered in the Observability series, combining correlated conditions into a single page.
The CloudWatch Agent is required for memory/disk-space visibility, since the hypervisor has no insight into the guest OS's memory.
Metric Filters bridge the Logs and Metrics pillars, extracting alarmable signals from existing log data with zero code changes.
X-Ray directly maps segments/subsegments onto the span vocabulary from the Observability series, with sampling rules implementing the same reservoir-plus-rate strategy already covered generically.
Synthetics (proactive) and RUM (reactive) are complementary, not competing — together covering both "did we break something" and "what are real users actually experiencing."
SLOs and burn-rate alerting apply directly on AWS via a custom metric publishing the computed burn rate, bridging the SRE Fundamentals series' math into real, alarmable CloudWatch signals.

Continue to Part 11 (11-cicd-iac-and-messaging.md) to see how AWS-native CI/CD tooling, Infrastructure as Code, and messaging services tie the entire series together into an automated delivery pipeline.

← PreviousSecurity & Compliance Next →CI/CD, IaC & Messaging