Monitoring, Logging & Observability

Table of Contents#

1. Observability Across This Series — Metrics, Logs, Traces
2. Azure Monitor — Platform Metrics
3. Log Analytics Workspace — Architecture and Design
4. Log Analytics Table Plans
5. KQL — Querying Logs
6. Application Insights — Application Performance Monitoring
7. The OpenTelemetry Distro — the Current Recommendation
8. Distributed Tracing
9. Alerts and Action Groups
10. Data Collection Rules and the Azure Monitor Agent
11. Azure Monitor Workbooks and Dashboards
12. VM Insights and Container Insights
13. Bringing It Together — Correlating Across This Series
14. A Full Worked Observability Bootstrap for Meridian Freight
15. Part 13 CLI Cheat Sheet
16. Common Mistakes and Interview Traps
17. Worked Practice Problems
18. Summary and What's Next

Observability Across This Series — Metrics, Logs, Traces#

Nearly every part of this series has referenced Azure Monitor or Log Analytics in passing — NSG flow logs (Part 7), Sentinel's underlying workspace (Part 12), VM boot diagnostics (Part 3). This chapter is where that scattered thread becomes one coherent observability platform.

Azure Monitor — Platform Metrics#

az monitor metrics list --resource "<vmss-resource-id>" --metric "Percentage CPU" --interval PT5M

Metrics are lightweight, numeric time-series data collected automatically for most Azure resources with no configuration required — CPU percentage, request count, queue depth — the data source behind Part 3's autoscale rules and Part 1's cost alerts alike.

Log Analytics Workspace — Architecture and Design#

az monitor log-analytics workspace create --workspace-name log-analytics-meridian \
  --resource-group rg-shipment-api-prod --location eastus --retention-time 90

A genuinely important architectural decision worth stating explicitly: one workspace per subscription/environment is usually right, but a genuine business requirement (regulatory data residency, separate billing/cost attribution per business unit, or a hard isolation boundary between environments) can justify multiple workspaces — the same "don't over-fragment without a real reason" principle this series applied to subscriptions (Part 1) and Application Gateways (Part 6) applies here too.

Log Analytics Table Plans#

az monitor log-analytics workspace table update --workspace-name log-analytics-meridian \
  --resource-group rg-shipment-api-prod --name "ContainerLogV2" --plan Basic

Why choosing a table plan deliberately per table matters concretely for cost, worth stating the underlying reasoning: high-volume, rarely-queried verbose logs (raw container stdout, for instance) cost meaningfully more than necessary on the default Analytics plan — moving them to Basic or Auxiliary can cut ingestion cost substantially while keeping genuinely investigation-critical tables (security events, application traces) on the full Analytics plan where their query capability is actually needed.

KQL — Querying Logs#

ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(1h)
| where Log_s contains "ERROR"
| summarize ErrorCount = count() by ContainerAppName_s, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

Kusto Query Language (KQL) is the single query language spanning Log Analytics, Sentinel (Part 12), and Azure Resource Graph (Part 1) — learning it once pays off across every one of these surfaces, rather than treating each as needing its own separate query skill.

Application Insights — Application Performance Monitoring#

az monitor app-insights component create --app shipment-api-insights \
  --resource-group rg-shipment-api-prod --location eastus \
  --workspace "<log-analytics-workspace-id>"

Application Insights provides application-level telemetry — request rates, response times, dependency call durations, exceptions — automatically correlated across a distributed system, the practical tool for answering "why is shipment-api slow right now" with actual data rather than guesswork.

The OpenTelemetry Distro — the Current Recommendation#

A genuinely important, current fact worth stating explicitly: Microsoft now recommends the Azure Monitor OpenTelemetry Distro over the classic Application Insights SDK for new applications — it delivers equivalent Azure Monitor integration while being built on the vendor-neutral OpenTelemetry standard, rather than a proprietary SDK.

# Python example — the OpenTelemetry Distro, not the classic SDK
from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor(connection_string="<app-insights-connection-string>")

Why this matters concretely for a new service like a future Meridian Freight microservice, worth stating the underlying reasoning: building on OpenTelemetry avoids vendor lock-in to Application Insights' proprietary SDK specifically — the same instrumented code could, in principle, export telemetry to a different backend later, since OpenTelemetry is an open, vendor-neutral standard rather than an Azure-specific API surface.

Distributed Tracing#

Distributed tracing correlates every hop of a request across services using a shared trace ID, following the W3C Trace Context standard — genuinely essential for shipment-api's microservices architecture (Part 10, Part 11), where a single user-facing request might touch five separate services, and understanding WHERE latency or an error actually originated requires seeing the whole chain, not just one service's isolated logs.

Alerts and Action Groups#

az monitor metrics alert create --name high-cpu-alert --resource-group rg-driver-portal-prod \
  --scopes "<vmss-resource-id>" --condition "avg Percentage CPU > 80" \
  --action "<action-group-resource-id>"

az monitor action-group create --name ag-platform-oncall --resource-group rg-shipment-api-prod \
  --action email platform-team platform-oncall@meridianfreight.com \
  --action webhook pagerduty "<pagerduty-webhook-url>"

An action group decouples WHO/HOW to notify from the alert rule itself — the same action group (email + PagerDuty webhook) can be reused across dozens of alert rules, so updating the on-call escalation path means changing one action group, not hunting down every individual alert rule referencing it.

Data Collection Rules and the Azure Monitor Agent#

The Azure Monitor Agent (AMA) — the single, unified successor to several older, separate monitoring agents — collects data from VMs based on a Data Collection Rule (DCR), which declares exactly what to collect and where to send it.

az monitor data-collection rule create --name dcr-driver-portal --resource-group rg-driver-portal-prod \
  --data-flows '[{"streams": ["Microsoft-Perf"], "destinations": ["log-analytics-meridian"]}]' \
  --location eastus

az monitor data-collection rule association create --name assoc-driver-portal \
  --rule-id "<dcr-resource-id>" --resource "<vmss-resource-id>"

Why the DCR model is worth understanding as a genuine improvement over the older per-agent-configuration approach, worth stating the underlying reasoning: a single DCR can be associated with MANY VMs, declaring the collection policy once and applying it consistently — changing what's collected means updating one DCR, not reconfiguring an agent setting on every VM individually, the same "single source of truth, not per-resource configuration" pattern this series has recommended repeatedly (Firewall Manager policies in Part 7, IPAM pools in Part 4).

Azure Monitor Workbooks and Dashboards#

az monitor workbook create --name "shipment-api-health" --resource-group rg-shipment-api-prod \
  --category workbook --serialized-data @workbook-definition.json

Workbooks combine metrics, logs, and text into a single, interactive, parameterized report — genuinely more powerful than a static dashboard for an operational runbook, since a workbook's queries can accept parameters (a specific time range, a specific service name) rather than being fixed at creation time.

VM Insights and Container Insights#

az vm extension set --vm-name vm-driver-portal-01 --resource-group rg-driver-portal-prod \
  --name AzureMonitorLinuxAgent --publisher Microsoft.Azure.Monitor

az aks enable-addons --name aks-meridian --resource-group rg-shipment-api-prod --addons monitoring

VM Insights and Container Insights are pre-built, curated monitoring experiences for their respective compute types — VM Insights surfaces per-process resource consumption and dependency maps automatically; Container Insights surfaces per-pod/per-container metrics and logs for AKS (Part 10), both without hand-building the underlying KQL queries from scratch for genuinely common operational questions.

Bringing It Together — Correlating Across This Series#

This table is worth reading as the payoff of building this series' services with observability in mind from the start, rather than bolting it on afterward: every signal already exists because earlier parts already recommended enabling it — this chapter's job is showing how they combine into one coherent operational picture, not introducing yet another disconnected tool.

A Full Worked Observability Bootstrap for Meridian Freight#

# 1. One shared Log Analytics workspace for the environment
az monitor log-analytics workspace create --workspace-name log-analytics-meridian \
  --resource-group rg-shipment-api-prod --retention-time 90

# 2. Application Insights for shipment-api, using the OpenTelemetry Distro in application code
az monitor app-insights component create --app shipment-api-insights \
  --resource-group rg-shipment-api-prod --workspace "<log-analytics-workspace-id>"

# 3. A shared action group for on-call notification
az monitor action-group create --name ag-platform-oncall --resource-group rg-shipment-api-prod \
  --action email platform-team platform-oncall@meridianfreight.com

# 4. Alert rules referencing the shared action group
az monitor metrics alert create --name high-cpu-alert --resource-group rg-driver-portal-prod \
  --scopes "<vmss-resource-id>" --condition "avg Percentage CPU > 80" \
  --action "<action-group-resource-id>"

# 5. Container Insights on AKS, VM Insights on driver-portal's VMSS
az aks enable-addons --name aks-meridian --resource-group rg-shipment-api-prod --addons monitoring

Part 13 CLI Cheat Sheet#

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: Meridian Freight's Log Analytics costs grow substantially as docs-processor scales, traced to a high-volume, verbose debug-level log table that's queried maybe once a month during troubleshooting. What's the cost optimization, and what would be lost by applying it?

Answer: Moving this specific table from the default Analytics plan to the Basic (or Auxiliary, if even less frequent query need) plan reduces ingestion cost substantially, since Basic/Auxiliary plans cost meaningfully less than Analytics for the same ingested volume. What's traded away is full KQL query capability and potentially the immediate, full-retention querying Analytics provides — acceptable for this specific table given its actual usage pattern (rare, ad-hoc troubleshooting queries), but this decision should be made per-table based on actual query frequency and criticality, not applied blanket across every table in the workspace, since genuinely investigation-critical tables (security events, application traces) still need Analytics' full query capability.

Problem 2: A shipment-api request occasionally times out, and the on-call engineer spends an hour manually checking logs across shipment-api, rates-db, and the Service Bus queue independently, unable to determine which specific hop is actually slow. What observability capability would have made this investigation dramatically faster, and why?

Answer: Distributed tracing, correlating every hop of the request chain under one shared trace ID (via the W3C Trace Context standard), would let the engineer view the ENTIRE call chain — shipment-api to rates-db to Service Bus — as one connected timeline in Application Insights, immediately showing which specific hop's duration accounts for the timeout, rather than requiring separate, disconnected log searches across three different systems and manually correlating timestamps by hand. This is precisely the capability this chapter's distributed tracing section describes, and its absence is exactly why the manual, hour-long investigation was necessary in the first place.

Summary and What's Next#

• Azure Monitor Metrics, Log Analytics, and Application Insights together form one coherent observability platform — most of it already referenced piecemeal throughout this series.
• Log Analytics table plans (Analytics, Basic, Auxiliary) let cost be optimized per table based on actual query frequency, not a blanket ingestion cost applied uniformly.
• The Azure Monitor OpenTelemetry Distro is now the recommended default for new applications over the classic, proprietary Application Insights SDK.
• Distributed tracing's shared trace ID is what makes multi-service latency investigation tractable — the alternative (manually correlating separate per-service logs) doesn't scale past a couple of hops.
• Action groups decouple notification targets from alert rules — a single shared group, updated once, rather than hardcoded per-rule notification targets.
• This series' own repeated recommendations to enable NSG flow logs, Identity Protection, and cost alerts all feed directly into this chapter's unified observability picture — the payoff of building with observability in mind from Part 1 onward.
• Data Collection Rules let one declared collection policy apply consistently across many VMs — updated once, rather than reconfiguring monitoring agent settings per-resource.

Continue to Part 14 (14-business-continuity-backup-dr-and-migration.md) for the backup, disaster recovery, and migration strategy this chapter's monitoring makes possible to execute and verify.