# Azure Cloud Architecture — Part 11: Application Architecture & Messaging

> **Series:** Azure Cloud Architecture (11 of 16)
> **Part 1:** `01-fundamentals-and-governance.md` — Fundamentals & Governance
> **Part 2:** `02-identity-and-access.md` — Identity & Access
> **Part 3:** `03-compute-vms-and-scale-sets.md` — Compute: Virtual Machines & Scale Sets
> **Part 4:** `04-networking-foundations-vnets-ip-and-dns.md` — Networking Foundations: VNets, IP & DNS
> **Part 5:** `05-networking-hybrid-connectivity.md` — Networking: Hybrid Connectivity
> **Part 6:** `06-networking-application-delivery.md` — Networking: Application Delivery
> **Part 7:** `07-networking-private-access-and-security.md` — Networking: Private Access & Security
> **Part 8:** `08-storage-blob-files-and-disks.md` — Storage: Blob, Files & Disks
> **Part 9:** `09-databases-and-data-services.md` — Databases & Data Services
> **Part 10:** `10-containers-and-serverless.md` — Containers & Serverless
> **Part 11:** This file — Application Architecture & Messaging
> **Part 12:** `12-security-and-compliance.md` — Security & Compliance
> **Part 13:** `13-monitoring-logging-and-observability.md` — Monitoring, Logging & Observability
> **Part 14:** `14-business-continuity-backup-dr-and-migration.md` — Business Continuity: Backup, DR & Migration
> **Part 15:** `15-cicd-and-iac.md` — CI/CD & Infrastructure as Code
> **Part 16:** `16-multi-region-cost-optimization-and-cheatsheet.md` — Multi-Region, Cost Optimization & Cheat Sheet
> **Questions:** `questions.md`

## Table of Contents

1. [Application Architecture — Connecting What This Series Has Built](#application-architecture--connecting-what-this-series-has-built)
2. [Service Bus — Queues and Topics](#service-bus--queues-and-topics)
3. [Service Bus Advanced Features](#service-bus-advanced-features)
4. [Event Grid — Event Routing](#event-grid--event-routing)
5. [Event Hubs — High-Throughput Streaming](#event-hubs--high-throughput-streaming)
6. [Choosing Between Service Bus, Event Grid, and Event Hubs](#choosing-between-service-bus-event-grid-and-event-hubs)
7. [Combining Event Grid and Service Bus](#combining-event-grid-and-service-bus)
8. [API Management — Architecture and Tiers](#api-management--architecture-and-tiers)
9. [API Management Workspaces](#api-management-workspaces)
10. [API Management Policies](#api-management-policies)
11. [Azure Logic Apps — Low-Code Workflow Orchestration](#azure-logic-apps--low-code-workflow-orchestration)
12. [Caching Patterns With Azure Cache for Redis](#caching-patterns-with-azure-cache-for-redis)
13. [Azure App Configuration](#azure-app-configuration)
14. [Event-Driven Architecture Principles](#event-driven-architecture-principles)
15. [A Full Worked Messaging Bootstrap for Meridian Freight](#a-full-worked-messaging-bootstrap-for-meridian-freight)
16. [Part 11 CLI Cheat Sheet](#part-11-cli-cheat-sheet)
17. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
18. [Worked Practice Problems](#worked-practice-problems)
19. [Summary and What's Next](#summary-and-whats-next)

## Application Architecture — Connecting What This Series Has Built

This chapter wires together every compute and data service the series has built so far: `shipment-api` (Container Apps, Part 10) notifying `driver-portal` of a status change, `docs-processor` (Functions) picking up newly uploaded documents (Storage, Part 8), and a future analytics pipeline consuming a stream of events — all through Azure's messaging services rather than direct, tightly-coupled service-to-service calls.

```mermaid
graph LR
    ShipmentAPI["shipment-api"] -->|"Service Bus queue —\nreliable, ordered work"| Processing["Order processing"]
    Storage["Blob Storage upload"] -->|"Event Grid —\nreact to the event"| DocsProcessor["docs-processor Function"]
    Telemetry["Driver GPS telemetry"] -->|"Event Hubs —\nhigh-volume stream"| Analytics["Analytics pipeline"]
```

---

## Service Bus — Queues and Topics

```bash
az servicebus namespace create --name sb-meridian --resource-group rg-shipment-api-prod --sku Standard

az servicebus queue create --namespace-name sb-meridian --resource-group rg-shipment-api-prod \
  --name order-processing --max-delivery-count 5 --lock-duration PT30S

az servicebus topic create --namespace-name sb-meridian --resource-group rg-shipment-api-prod --name shipment-events
az servicebus topic subscription create --namespace-name sb-meridian --resource-group rg-shipment-api-prod \
  --topic-name shipment-events --name driver-portal-sub \
  --filter-sql-expression "eventType = 'StatusChanged'"
```

**Queues** implement competing-consumer semantics — each message goes to exactly ONE receiver, the right fit for `shipment-api`'s order-processing work needing to happen exactly once per order. **Topics** add publish-subscribe — one message published to a topic reaches EVERY subscription independently, each optionally filtered with a SQL-like expression, the right fit for `driver-portal` and an analytics pipeline both needing their own independent copy of the same shipment-status-changed event.

---

## Service Bus Advanced Features

| Feature | Purpose |
|---|---|
| Sessions | Groups related messages (all messages for one order) to guarantee in-order, single-consumer processing |
| Dead-letter queue | Automatically captures messages that fail delivery repeatedly (`max-delivery-count` exceeded), rather than silently dropping or endlessly retrying them |
| Duplicate detection | Rejects a message with a duplicate message ID within a configured time window — genuinely useful against at-least-once delivery producing accidental duplicates |
| Scheduled delivery | Delays a message's visibility until a specific future time |

```bash
az servicebus queue update --namespace-name sb-meridian --resource-group rg-shipment-api-prod \
  --name order-processing --enable-duplicate-detection true --duplicate-detection-history-time-window PT10M
```

**Why the dead-letter queue matters concretely, worth stating explicitly: without it, a message that repeatedly fails processing (a malformed order, a downstream dependency outage) would either loop forever consuming processing capacity, or silently vanish after the delivery count is exhausted** — the dead-letter queue preserves it for manual inspection, turning a silent failure into a visible, actionable one.

---

## Event Grid — Event Routing

```bash
az eventgrid event-subscription create --name docs-uploaded-sub \
  --source-resource-id "<storage-account-resource-id>" \
  --endpoint "<function-app-resource-id>" --endpoint-type azurefunction \
  --included-event-types Microsoft.Storage.BlobCreated
```

**Event Grid is a lightweight, low-latency FAN-OUT ROUTER, not a message queue with delivery guarantees the way Service Bus is** — it reacts to an event (a blob created, a resource provisioned) and routes it to one or more subscribers, the natural fit for `docs-processor`'s trigger: the moment a new document lands in Blob Storage, Event Grid notifies the Function directly, with no polling required.

---

## Event Hubs — High-Throughput Streaming

```bash
az eventhubs namespace create --name eh-meridian --resource-group rg-shipment-api-prod --sku Standard
az eventhubs eventhub create --name driver-telemetry --namespace-name eh-meridian \
  --resource-group rg-shipment-api-prod --partition-count 4 --message-retention 7
```

**Event Hubs is built for high-volume event STREAMING — millions of events per second, retained for a configurable window (1-90 days) allowing multiple independent consumers to replay the same stream** — the right fit for a future high-frequency driver GPS telemetry ingestion pipeline, genuinely different in both scale and semantics from Service Bus's reliable-delivery-per-message model.

---

## Choosing Between Service Bus, Event Grid, and Event Hubs

| Need | Recommendation |
|---|---|
| Reliable, ordered, exactly-once-per-consumer work processing | Service Bus |
| React to a discrete event with low latency, fan-out to multiple subscribers | Event Grid |
| Ingest and replay a high-volume continuous event stream | Event Hubs |

**Worth stating the underlying distinction precisely, since surface-level similarity ("they all move messages around") obscures real semantic differences: Service Bus optimizes for reliable DELIVERY of discrete work items, Event Grid optimizes for low-latency ROUTING of discrete events, and Event Hubs optimizes for high-throughput, replayable STREAMING** — choosing based on message volume alone, without considering these different guarantees, is a common, real design mistake.

---

## Combining Event Grid and Service Bus

```bash
az eventgrid event-subscription create --name high-priority-orders-sub \
  --source-resource-id "<eventgrid-topic-resource-id>" \
  --endpoint "<service-bus-queue-resource-id>" --endpoint-type servicebusqueue
```

**A genuinely powerful, common combined pattern worth naming explicitly: Event Grid's low-latency fan-out feeding directly INTO a Service Bus queue for reliable, ordered processing — getting Event Grid's instant reactivity for detecting that something happened, combined with Service Bus's durability and ordering guarantees for actually processing it**, rather than choosing one service and accepting whichever tradeoff it doesn't cover well.

---

## API Management — Architecture and Tiers

```bash
az apim create --name apim-meridian --resource-group rg-shipment-api-prod \
  --publisher-email platform@meridianfreight.com --publisher-name "Meridian Freight" \
  --sku-name StandardV2
```

| Tier | Fit |
|---|---|
| Consumption | Serverless, pay-per-call — lightweight or intermittent API traffic |
| Basic v2 / Standard v2 (current) | The current recommended production tiers — replacing the classic Basic/Standard |
| Premium v2 | Multi-region deployment, VNet integration, the highest-scale tier |

**API Management** fronts `shipment-api`'s public endpoints with a genuinely separate concern from Part 6's Application Gateway/Front Door: those handle network-level load balancing and WAF; APIM adds API-CONTRACT-level concerns — versioning, rate limiting per subscription key, request/response transformation, and developer-facing API documentation.

---

## API Management Workspaces

```bash
az apim workspace create --service-name apim-meridian --resource-group rg-shipment-api-prod \
  --workspace-id ws-carrier-partners --display-name "Carrier Partner APIs"
```

**A genuinely current capability worth stating explicitly, expanded in mid-2026: workspaces let a platform team delegate ownership of a specific set of APIs to an individual team (a "Carrier Partner APIs" workspace owned by the partnerships team) while keeping centralized governance at the APIM instance level** — and workspaces on the built-in gateway (rather than a separate dedicated workspace gateway) now inherit multi-region deployment, custom hostnames, and Private Link connectivity that dedicated workspace gateways don't offer, a real, current reason to prefer the built-in-gateway workspace model for new adoption.

---

## API Management Policies

```xml
<policies>
  <inbound>
    <rate-limit-by-key calls="100" renewal-period="60" counter-key="@(context.Subscription.Key)" />
    <validate-jwt header-name="Authorization" failed-validation-httpcode="401">
      <openid-config url="https://login.microsoftonline.com/<tenant-id>/.well-known/openid-configuration" />
    </validate-jwt>
  </inbound>
</policies>
```

**Policies** are XML-configured request/response pipeline logic — rate limiting per subscriber, JWT validation against Entra ID (Part 2) at the gateway itself before a request ever reaches `shipment-api`'s actual code, and request/response transformation for API versioning. **Why validating the JWT at the API gateway layer, rather than solely in application code, matters concretely: it rejects unauthenticated or malformed requests before they consume any backend compute at all**, directly extending this series' repeated "reject at the edge, not after wasting backend resources" pattern from Part 6's rate limiting discussion.

---

## Azure Logic Apps — Low-Code Workflow Orchestration

**Logic Apps** provides a visual, low-code workflow designer connecting hundreds of pre-built connectors (Office 365, Salesforce, SFTP, and Azure's own services) — genuinely distinct from Azure Functions/Durable Functions, worth stating the actual difference precisely rather than treating them as interchangeable "serverless orchestration" options.

```bash
az logic workflow create --name wf-carrier-onboarding --resource-group rg-shipment-api-prod \
  --definition @workflow-definition.json
```

| | Logic Apps | Durable Functions |
|---|---|---|
| Authoring | Visual designer, JSON workflow definition | Code (Python/C#/JavaScript/etc.) |
| Best fit | Connector-heavy integration between SaaS/enterprise systems | Complex custom logic, code-first teams |
| Audience | Often business/integration analysts, not exclusively developers | Software engineers |

**Why Logic Apps is worth reaching for specifically when a workflow is genuinely integration-heavy — connecting to a carrier partner's SFTP server, then an internal Teams notification, then a Salesforce update — rather than defaulting to a custom Durable Functions orchestrator for the same task: the pre-built connectors eliminate real, otherwise-necessary integration code for each system**, at the cost of less flexibility than a fully code-first approach for genuinely custom business logic.

---

## Caching Patterns With Azure Cache for Redis

```mermaid
sequenceDiagram
    participant App as shipment-api
    participant Cache as Redis
    participant DB as rates-db
    App->>Cache: GET rate:carrier-acme
    alt Cache hit
        Cache-->>App: Return cached rate
    else Cache miss
        App->>DB: Query rate from database
        DB-->>App: Rate data
        App->>Cache: SET rate:carrier-acme (with TTL)
    end
```

The **cache-aside pattern** shown above is the most common Redis usage: check the cache first, fall back to the database on a miss, then populate the cache for next time. **Worth stating a genuinely important operational detail: always set a TTL (time-to-live) on cached entries** — a cache with no expiration can serve stale pricing data indefinitely after `rates-db` changes, a real correctness bug, not just a performance question.

---

## Azure App Configuration

```bash
az appconfig create --name appconfig-meridian --resource-group rg-shipment-api-prod --sku Standard

az appconfig kv set --name appconfig-meridian --key "ShipmentApi:FeatureFlags:NewRatingEngine" --value "true"
```

**App Configuration** centralizes application settings and feature flags across every service in this series — distinct from Key Vault (Part 12), which holds SECRETS specifically; App Configuration holds ordinary configuration values and feature flags, with Key Vault references embeddable within it for the genuinely secret values. **Feature flags** managed here let `shipment-api` toggle its new rating engine on/off without a redeployment, directly supporting progressive rollout patterns Part 15's CI/CD chapter builds on.

---

## Event-Driven Architecture Principles

A closing conceptual frame: this chapter's services all support **loose coupling** — `docs-processor` doesn't need to know WHO consumes its processed-document events, and `shipment-api` doesn't need a direct network call to every service interested in a status change.

| Principle | What it means in practice |
|---|---|
| Producers don't know consumers | `shipment-api` publishes to a topic without knowing which subscriptions exist |
| Consumers can be added without changing producers | A new analytics subscription to `shipment-events` requires zero `shipment-api` code changes |
| Failure isolation | A downstream consumer's outage doesn't block the producer — messages queue until the consumer recovers |

**Why this matters concretely for Meridian Freight's own growth, worth stating explicitly: adding a new downstream consumer of shipment-status events (a future customer-notification service, say) requires only a new Service Bus subscription — zero changes to `shipment-api` itself**, directly the opposite of a tightly-coupled design where every new consumer requires modifying the producer's own code to add a new direct call.

---

## A Full Worked Messaging Bootstrap for Meridian Freight

```bash
# 1. Service Bus for reliable order processing and shipment events
az servicebus namespace create --name sb-meridian --resource-group rg-shipment-api-prod --sku Standard
az servicebus queue create --namespace-name sb-meridian --resource-group rg-shipment-api-prod \
  --name order-processing --enable-duplicate-detection true
az servicebus topic create --namespace-name sb-meridian --resource-group rg-shipment-api-prod --name shipment-events

# 2. Event Grid triggering docs-processor on new document uploads
az eventgrid event-subscription create --name docs-uploaded-sub \
  --source-resource-id "<storage-account-resource-id>" \
  --endpoint "<function-app-resource-id>" --endpoint-type azurefunction

# 3. API Management fronting shipment-api's public contract
az apim create --name apim-meridian --resource-group rg-shipment-api-prod \
  --publisher-email platform@meridianfreight.com --publisher-name "Meridian Freight" --sku-name StandardV2

# 4. App Configuration for centralized feature flags
az appconfig create --name appconfig-meridian --resource-group rg-shipment-api-prod --sku Standard
```

---

## Part 11 CLI Cheat Sheet

| Area | Command | Purpose |
|---|---|---|
| Service Bus | `az servicebus queue create` / `topic create` | Create a queue or topic |
| Service Bus | `az servicebus queue update --enable-duplicate-detection` | Enable duplicate detection |
| Event Grid | `az eventgrid event-subscription create` | Subscribe an endpoint to events |
| Event Hubs | `az eventhubs eventhub create` | Create an event stream |
| APIM | `az apim create` | Create an API Management instance |
| APIM | `az apim workspace create` | Create a delegated API workspace |
| Redis | `az redis create` | Create a cache (Part 9) |
| App Config | `az appconfig kv set` | Set a configuration value or feature flag |
| Logic Apps | `az logic workflow create` | Create a low-code, connector-based workflow |

---

## Common Mistakes and Interview Traps

| Mistake | Why It's Wrong | Fix |
|---|---|---|
| Using Event Hubs for reliable, ordered work item processing | Optimized for high-throughput streaming, not per-message delivery guarantees | Use Service Bus for reliable work processing |
| Using Service Bus for millions-of-events-per-second telemetry ingestion | Not built for that throughput scale the way Event Hubs is | Use Event Hubs for high-volume streaming |
| Caching data in Redis with no TTL | Serves stale data indefinitely after the source of truth changes | Always set a TTL matching how fresh the data genuinely needs to be |
| Storing secrets in App Configuration directly | App Configuration isn't a secrets store | Use Key Vault for secrets, referenced from App Configuration if needed |
| Validating JWTs only in application code, not at the API gateway | Wastes backend compute processing requests that will be rejected anyway | Validate at the APIM policy layer before the request reaches the backend |
| Ignoring dead-letter queues | Repeatedly failing messages loop or vanish silently | Monitor and act on dead-lettered messages as a standard operational practice |
| Building a custom Durable Functions orchestrator for a connector-heavy SaaS integration | Reimplements integration code Logic Apps' pre-built connectors already provide | Use Logic Apps when a workflow is primarily connecting to external systems, not custom business logic |

---

## Worked Practice Problems

**Problem 1:** Meridian Freight's platform team initially routes driver GPS telemetry (thousands of location updates per second across the fleet) through a Service Bus queue, and finds the namespace consistently hitting throughput limits and requiring constant scaling intervention. What's the architectural mismatch, and what's the fix?

*Answer:* Service Bus is optimized for reliable, ordered delivery of discrete work items, not raw high-volume streaming — its throughput ceiling per namespace is meaningfully lower than a service purpose-built for high-frequency event ingestion. Event Hubs is the correct fit for this specific workload: designed for millions of events per second, with partition-based scaling and configurable retention allowing multiple consumers to process the same stream independently. The fix is migrating the telemetry pipeline to Event Hubs, reserving Service Bus for genuinely work-item-oriented flows like order processing where its ordering and delivery guarantees matter more than raw throughput.

**Problem 2:** A team builds `shipment-api`'s rate-lookup caching layer in Redis without setting any expiration on cached entries, reasoning "rates don't change that often, so there's no rush." Months later, a rate correction in `rates-db` fails to reflect in `shipment-api`'s responses for over a week, causing real customer billing disputes. What was the actual failure, and what's the fix?

*Answer:* The failure was treating "rates don't change often" as equivalent to "stale cache entries are harmless" — without a TTL, a cached rate persists indefinitely regardless of how the underlying data changes, meaning ANY correction to `rates-db` has no path to ever reach cached responses without an explicit cache invalidation the team never built. The fix is setting a TTL appropriate to the actual acceptable staleness window (even a relatively long one, like a few hours, given rates genuinely change infrequently) — this guarantees a bounded worst case for staleness, converting an open-ended "could be stale forever" risk into a known, acceptable one, without requiring building and maintaining an explicit invalidation mechanism for every possible rate-change code path.

**Problem 3:** Meridian Freight needs to onboard a new carrier partner whose systems only support SFTP file exchange, requiring a workflow that picks up a file from the partner's SFTP server, transforms it, writes it to Blob Storage, and posts a Teams notification to the operations channel. A developer proposes writing this as a custom Durable Functions orchestrator. Evaluate this choice against the alternative.

*Answer:* Logic Apps is the better-suited tool here — the workflow is fundamentally integration-heavy (SFTP, Blob Storage, Teams), and Logic Apps provides pre-built, tested connectors for all three systems, letting the workflow be assembled largely through configuration rather than custom code for each integration point. Building this as a custom Durable Functions orchestrator would mean writing and maintaining SFTP client code, Blob Storage SDK calls, and Teams API integration by hand — all things Logic Apps' connector model already solves. Durable Functions remains the better choice when the workflow's core value is complex, custom business logic rather than connecting between systems — this specific scenario is squarely the connector-heavy case Logic Apps is designed for.

---

## Summary and What's Next

- **Service Bus, Event Grid, and Event Hubs solve genuinely distinct messaging problems** — reliable work processing, low-latency event routing, and high-throughput streaming respectively — not interchangeable options differing only by message volume.
- **Combining Event Grid's fan-out with Service Bus's reliability is a real, powerful pattern** for getting both instant reactivity and durable processing guarantees together.
- **API Management adds API-contract-level concerns** (versioning, per-subscriber rate limiting, JWT validation) distinct from Part 6's network-level load balancing and WAF.
- **API Management workspaces on the built-in gateway now inherit multi-region, custom hostname, and Private Link support** that dedicated workspace gateways don't offer — the current recommended workspace model.
- **Always set a TTL on cached Redis entries** — an unbounded cache can serve stale data indefinitely, a correctness bug, not just a performance tradeoff.
- **Event-driven architecture's loose coupling lets new consumers be added without modifying producers** — the practical payoff of this chapter's messaging services over direct, tightly-coupled service calls.
- **Logic Apps' pre-built connectors fit connector-heavy integration workflows** — Durable Functions remains the better choice for complex, custom business logic.

**Continue to Part 12** (`12-security-and-compliance.md`) for Key Vault, Microsoft Defender for Cloud, and Microsoft Sentinel — the security and compliance layer wrapping every service this series has built so far.