Routing, BGP & the Internet's Architecture

Assumes you're comfortable with basic IP addressing, subnetting, and how a single Linux host resolves a name and opens a TCP connection — see this site's Linux & Networking Fundamentals series first if any of that is shaky. This series picks up where that one stops: not what happens inside one machine, but how traffic actually finds its way across the wider network — between datacenters, between clouds, and across the public internet.

Table of Contents#

1. Why a Platform Engineer Needs to Understand Routing
2. Zooming Out From One Host to the Whole Network
3. Autonomous Systems — the Internet's Real Political Boundaries
4. IP Prefixes, CIDR, and Route Aggregation
5. What BGP Actually Is — eBGP vs. iBGP
6. The BGP Best-Path Algorithm — Attributes in Order
7. Watching a Route Propagate, Hop by Hop
8. Route Filtering — Why a Route Can Simply Vanish
9. BGP Hijacks and Route Leaks — What Actually Goes Wrong
10. RPKI — Route Origin Validation as the Real Fix
11. IPv6 Routing — the Same Rules, Different Numbers, Real Gotchas
12. ECMP — Equal-Cost Multipath Inside the Datacenter
13. Anycast — One IP Address, Many Machines
14. IXPs, Transit, and Peering — How Networks Actually Connect
15. BGP Session Mechanics and Convergence Tuning
16. BGP Communities — Tagging Routes for Downstream Policy
17. iBGP at Scale — Why Big Networks Use Route Reflectors
18. Full Worked Scenario: Taking checkout-service Multi-Region With Anycast
19. Common Mistakes and Interview Traps
20. Worked Practice Problems
21. Summary and What's Next

Why a Platform Engineer Needs to Understand Routing#

Most engineers treat the network between "my service" and "the user" as a black box that either works or doesn't — this chapter, and the rest of this series, opens that box. Throughout this series the running example is a small e-commerce platform made of three services — checkout-service, catalog-service, and inventory-service — the same throughline used across this site's Terraform, Kubernetes, Observability, and Incident Management series. Those services live behind load balancers, inside cloud VPCs, and are reached over the public internet by customers all over the world. Every one of those hops — from a customer's laptop, through their ISP, across the internet backbone, into a cloud provider's network, and finally into the pod running checkout-service — is a routing decision made by a piece of software that had no idea "checkout-service" exists. It just knows which prefix (an IP address range) is reachable through which next hop.

When a platform team is asked "why did EU customers see 3x latency for twenty minutes on Tuesday," the honest answer is frequently not in the application logs at all — it's a BGP route that changed, a load balancer that failed over to a farther region, or a route table that silently pointed traffic at the wrong place. An engineer who only understands their own service's code has no way to even start investigating that. This chapter builds the vocabulary and mental model to read a routing table, reason about why traffic took the path it took, and recognize the failure signatures of the internet's own routing layer — separate from anything your application did wrong.

Zooming Out From One Host to the Whole Network#

A useful mental model: every device that forwards IP packets — your laptop's kernel, a top-of-rack switch, a cloud provider's virtual router, an internet backbone router — keeps a routing table: a list of destination prefixes and the next hop to reach them. ip route on a single Linux box (covered in this site's Linux & Networking Fundamentals series) shows exactly this, just for one machine with a handful of entries. A backbone router on the public internet keeps essentially the same kind of table, just with roughly one million entries — one for (approximately) every routable network on the planet.

The mechanism that fills in a single host's table is usually static configuration or DHCP. The mechanism that fills in the internet's own table — a million entries, changing constantly as networks come up, go down, and renegotiate who reaches whom — is a single routing protocol: BGP, the Border Gateway Protocol. Nearly everything in this chapter is really "how does a piece of software decide, among several possible next hops toward the same destination, which one to actually use" — the same question a Linux host answers with a short static table, and the internet answers with BGP running between tens of thousands of independently operated networks.

Every one of these boxes is a separate routing decision made independently, with no single entity in charge of the whole path. The rest of this chapter explains how that works without collapsing into chaos.

Autonomous Systems — the Internet's Real Political Boundaries#

An Autonomous System (AS) is the unit the internet's routing actually organizes around — not a country, not a company name, a block of IP address space under one routing policy. Every ISP, cloud provider, university, and large enterprise that runs its own routing decisions is assigned an AS number (ASN) — a 16- or 32-bit integer, written like AS64512 — by a Regional Internet Registry (ARIN, RIPE, APNIC, LACNIC, or AFRINIC). Inside its own AS, an organization can run whatever internal routing it wants. Between ASes, BGP is the only protocol the internet actually agrees on.

This matters operationally the moment you run anything with a public IP presence: your cloud provider's network is one or more ASes (AWS advertises from several, depending on region and service), your own organization can obtain its own ASN and announce its own IP space (common for large platforms that want multi-cloud or multi-CDN failover control), and every ISP your customers use is its own AS. "The internet" is really tens of thousands of ASes, each trusting BGP announcements from their neighbors with very little built-in verification — a fact that explains most of the failure modes covered later in this chapter.

💡 A single organization can hold multiple ASNs — one per region, one per business unit, or one specifically for a CDN/edge network that peers differently than the main corporate network. Don't assume "one company = one ASN" when reading a routing table or a whois lookup.

IP Prefixes, CIDR, and Route Aggregation#

Every routing table entry, at every layer, is a prefix — an IP network expressed as address/prefix-length (CIDR notation), e.g. 203.0.113.0/24 for 256 addresses. The shorter the prefix length, the larger (and coarser) the block. This matters for routing because routers store and process a route entry per distinct prefix advertised, and the global BGP table is already at roughly one million active prefixes — every network operator has a direct incentive to advertise the fewest, largest possible blocks rather than many small ones.

Route aggregation (supernetting) is the practice of announcing one large summarized prefix instead of many small component prefixes that fall inside it — 203.0.113.0/22 instead of separately announcing four /24s. This keeps the global routing table smaller and every router's forwarding decision faster. The tradeoff: aggregation hides internal structure. If 203.0.113.0/22 is announced as one route but only 203.0.113.0/24 is actually reachable (the other three /24s inside that /22 are unused or down), the whole /22 will still attract traffic for addresses nobody is listening on — those packets are dropped once they arrive, silently, with no signal back to the sender about why.

Aggregating four /24s into one /22 cuts the number of routes the rest of the internet has to store and evaluate by 4x — multiplied across a million prefixes, this is why aggregation discipline is treated as a community responsibility, not just an internal optimization.

What BGP Actually Is — eBGP vs. iBGP#

BGP is a path-vector protocol: instead of computing a shortest path by distance or link cost the way an interior protocol like OSPF does, a BGP speaker advertises to its neighbors the full AS path — the ordered list of every AS a route has already passed through to reach it — plus a handful of other attributes covered in the next section. Every recipient can then apply its own local policy to decide which of several candidate paths to actually use, and re-advertises its own choice (with itself prepended to the AS path) to its own neighbors. No single router needs a global map of the network; the AS path accumulated along the way is enough to prevent routing loops (a router that sees its own ASN already in a received path simply discards that path).

BGP runs in two distinct modes that are easy to conflate:

A large network (a cloud provider, a big enterprise with multiple internet-facing sites) runs iBGP internally specifically so that a route learned via eBGP at one edge router becomes visible to every router in the AS — without iBGP, only the router that directly learned a route would know about it, and internal traffic destined for the internet would have no way to find its way to the right exit point.

The AS path grows by exactly one hop at every eBGP boundary crossing — iBGP never adds a hop, it only distributes the externally-learned route internally.

The BGP Best-Path Algorithm — Attributes in Order#

When a router learns multiple candidate paths to the same prefix, it runs a strict, ordered tie-breaking algorithm to pick exactly one "best path" to actually use and advertise onward. This algorithm is the single most-tested piece of BGP trivia in networking interviews, and — more importantly — it's the thing you actually reach for when a traceroute shows traffic taking a path you didn't expect. Implementations vary slightly in the exact step count, but the meaningful order (most-significant first) is consistent across Cisco, Juniper, and open-source implementations like FRRouting:

Local Preference is the attribute your own network controls to steer outbound traffic — set it high on the exit you want your own AS to prefer when multiple external paths exist. MED is the attribute a neighboring AS uses to hint which of several interconnection points it would prefer you send traffic into — but a MED is only ever a suggestion the receiving AS is free to ignore, since Local Preference and AS_PATH length are evaluated first and override it entirely.

⚙️ Worked example: checkout-service's platform team operates their own ASN with two transit providers, AS64501 (their primary, cheaper commit) and AS64509 (backup). To make all outbound traffic prefer the primary except when it's down, they set Local Preference 200 on routes learned from AS64501 and leave the default 100 on routes from AS64509 — step 2 of the algorithm above resolves in favor of the primary on every router in the AS, with zero per-router manual routing needed. If AS64501 withdraws its routes during an outage, the Local-Preference-200 paths simply disappear from every router's table, and step 2 falls through to the only remaining candidate — automatic failover, driven entirely by BGP's own convergence, not a script.

Watching a Route Propagate, Hop by Hop#

The mechanics of BGP convergence become concrete once you trace a single prefix's announcement outward. Say catalog-service's platform team originates 203.0.113.0/24 from their own AS64510, connected to two transit providers.

1. AS64510 sends an eBGP UPDATE message to each transit provider, announcing 203.0.113.0/24 with AS_PATH [64510].
2. Each transit provider evaluates it against its own routes (likely accepts it — it's a new, valid customer route) and re-advertises it to its own neighbors, prepending itself: AS_PATH becomes [64511, 64510] from one, [64512, 64510] from the other.
3. Those neighbors do the same, prepending again, until the announcement has rippled out to every AS on the internet whose policy accepts it — typically within 30-90 seconds for a healthy, well-connected network, though full global convergence after a large change can take several minutes.
4. Anywhere in the world, a router now evaluating "how do I reach 203.0.113.0/24" runs the best-path algorithm from the previous section over however many candidate AS paths it received, and picks one.

A withdrawal (route removal) propagates the same way, using a BGP WITHDRAW message — this is the exact mechanism behind anycast failover and BGP-based DDoS mitigation covered later in this chapter: stop announcing a prefix from one location, and the entire internet converges on an alternate path within the same tens-of-seconds window, with no action required from any individual client.

🔍 Real diagnostic commands (looking-glass servers — public read-only BGP query tools most large networks and IXPs operate — are the practitioner's way to see this without owning a router):

# From a Linux host near your own edge, see the AS path your own router chose
show ip bgp 203.0.113.0/24

# A public looking glass shows the SAME prefix from a completely different vantage point
# (many Tier-1/Tier-2 networks publish one, e.g. lg.he.net) — comparing your own router's
# view against a distant looking glass is the standard way to confirm global propagation
whois -h whois.radb.net 203.0.113.0/24

# traceroute reveals the ACTUAL forwarding path taken, hop by hop, which should be
# consistent with the AS path advertised for the destination prefix
traceroute -A 203.0.113.10

Route Filtering — Why a Route Can Simply Vanish#

BGP has essentially no default limit on what a neighbor can announce — every network operator applies its own inbound and outbound filters as policy, not protocol requirement. This is deliberate (BGP's design philosophy trusts operators to police their own edges) and it's also the root cause of a huge share of "unreachable from some places, fine from others" incidents:

• Prefix-length filters: most transit providers reject any announcement longer than /24 (IPv4), as covered above — a smaller block you legitimately own may simply never make it past your immediate neighbor.
• AS-path filters: a network may only accept routes whose AS path matches an expected pattern (e.g. "must originate from one of these specific customer ASNs") — protects against a downstream customer accidentally re-announcing someone else's routes.
• Prefix lists / IRR-based filtering: increasingly the industry-standard approach — a network publishes its intended announcements in the Internet Routing Registry (IRR), and its transit providers automatically generate filters (often via bgpq4, a widely used tool that converts an IRR AS-SET into a router prefix-list) that only accept exactly what's registered. If you change your announced prefixes without updating your IRR object first, the new prefix can be silently dropped by every filtering neighbor — a frequent, entirely self-inflicted outage cause.

BGP Hijacks and Route Leaks — What Actually Goes Wrong#

BGP was designed in an era when every network operator on the internet trusted every other one — there is no built-in cryptographic proof that an AS announcing a prefix is actually authorized to. This single design gap is the root cause of two related, still-common failure classes:

• A BGP hijack is an AS announcing a prefix it does not own or operate — maliciously (to intercept or black-hole traffic) or, just as often, entirely by accident (a fat-fingered router configuration). Because BGP's best-path algorithm generally prefers a shorter or more specific (longer-prefix) route, a hijacked announcement frequently wins over the legitimate one for at least some portion of the internet, until the affected operators notice and the false route is withdrawn or filtered out.
• A route leak is a network re-announcing routes it legitimately learned from one neighbor to another neighbor it should not have — most commonly a customer network accidentally announcing its transit provider's entire routing table onward to a second transit provider, turning that customer into an unintentional (and wildly under-provisioned) transit path for a chunk of the internet.

⚠️ From the trenches: on 27 June 2024, a Brazilian ISP (a small regional AS) announced the single address 1.1.1.1/32 — Cloudflare's public DNS resolver — as if it originated from their own network. The immediate cause was a misconfiguration, not malice: a router accepted and re-announced a customer route that should never have propagated past a local scope. The underlying condition that made it land globally: at least one Tier-1 transit provider accepted and re-propagated the bogus /32 without filtering it, and because a /32 is the most specific possible IPv4 route, BGP's best-path algorithm caused a meaningful share of the internet to prefer the fake route over Cloudflare's legitimate, properly-signed 1.1.1.0/24 announcement — even though Cloudflare had RPKI Route Origin Validation correctly configured (see the next section). Users worldwide saw 1.1.1.1 become unreachable or redirected for roughly two hours before the announcement was traced and filtered. The lesson that generalizes: RPKI protects you only as far as every network between you and the affected users actually enforces it — adoption gaps in the middle of the path are still a live risk in 2026, years after RPKI became widely available.

A more-specific (longer-prefix) hijack route wins over a legitimate but shorter/coarser announcement on every network that fails to filter it — regardless of how "correct" the legitimate side's own configuration is.

A second, quieter class deserves its own mention: on 1 May 2025, a single misconfigured customer network (AS22773) leaked 4,651 routes it had learned from its own transit providers back out to a peer — turning itself into an accidental transit path. APNIC's post-incident analysis found that 4,644 of those leaked routes would have been automatically rejected by any router performing RPKI-based Route Origin Validation, because the leak changed the AS path in a way that no longer matched the registered origin — concrete evidence for the next section's argument that RPKI adoption, not just BGP monitoring, is what actually prevents these incidents from spreading.

RPKI — Route Origin Validation as the Real Fix#

RPKI (Resource Public Key Infrastructure) lets the legitimate holder of an IP prefix cryptographically sign a statement — a Route Origin Authorization (ROA) — declaring exactly which ASN is authorized to originate that prefix, and up to what maximum prefix length. A router performing Route Origin Validation (ROV) checks every incoming BGP announcement's origin AS against the published ROAs and marks each route:

RPKI does not validate the full AS path — only the origin. It stops a hijack where an unauthorized AS directly originates a prefix (exactly the 1.1.1.1/32 case above, and exactly the class of route leak that made up 99.8% of the May 2025 AS22773 leak), but it does not by itself prevent a more sophisticated attack where the correct origin AS is preserved but an intermediate AS is falsely inserted into the path (a concern the newer, less widely deployed BGPsec/ASPA mechanisms address, not covered in production depth here since real-world deployment is still early as of this writing).

# Publishing a ROA (via your RIR's hosted portal, e.g. ARIN/RIPE) declares:
# "AS64510 is authorized to originate 203.0.113.0/24, max length /24"

# A router running an RPKI validator (rpki-client, Routinator, OctoRPKI are the
# common open-source choices) fetches and verifies signed ROAs, then feeds
# validity state into the router's own BGP policy:
router bgp 64502
  bgp bestpath prefix-validate  # reject ROV-invalid routes outright

IPv6 Routing — the Same Rules, Different Numbers, Real Gotchas#

Everything covered so far — the best-path algorithm, eBGP/iBGP, RPKI, route filtering — applies identically to IPv6; BGP itself is address-family agnostic, and a single BGP session can even carry both IPv4 and IPv6 routes (multiprotocol BGP, address-family ipv6 unicast). What differs in practice are the specific numbers and a handful of genuinely IPv6-only behaviors a platform team needs on their radar before their first dual-stack rollout, not after.

• Prefix filtering conventions are different. IPv4's informal /24-or-shorter filtering norm becomes /48-or-shorter for IPv6 — a /64 announced directly to the global table is as likely to be filtered as a IPv4 /28 would be. IPv6 address space is deliberately abundant specifically so operators can afford to be this conservative about table size without running out of addresses to assign internally.
• There is no NAT-driven address conservation pressure. IPv4 private address exhaustion inside a large organization routinely forces multiple layers of NAT; IPv6's address space is large enough that every device can have a real, globally-routable address, and NAT66 exists but is rarely used for conservation — when it does appear, it's almost always for a policy reason (hiding internal topology) rather than running out of addresses.
• Path MTU Discovery behaves differently. IPv4 routers can fragment an oversized packet in transit; IPv6 routers cannot — fragmentation is the source host's responsibility only. A misconfigured intermediate device that silently drops the ICMPv6 "Packet Too Big" message a sender depends on to learn the correct MTU produces a distinctive, hard-to-diagnose failure: small packets (a TLS handshake, a DNS query) succeed normally, while anything requiring a full-size packet (a large HTTP response body) hangs indefinitely with no error on either end — because the sender never receives the signal telling it to shrink its packet size, and IPv6's own routers won't quietly fragment on its behalf as an IPv4 router would.

⚠️ From the trenches: catalog-service's platform team enabled dual-stack (IPv4 + IPv6) on their public ALB ahead of a compliance deadline requiring IPv6 reachability. Internal smoke tests — small API calls — passed cleanly over IPv6. Within days, a support ticket arrived reporting that large product-image uploads over IPv6 hung indefinitely for a small subset of mobile customers, while the identical upload worked fine over IPv4 for the same customers. The immediate cause: a customer's mobile carrier network silently dropped ICMPv6 "Packet Too Big" messages at its own edge, a known, still-common carrier misconfiguration. The underlying condition that made it land specifically for this workload: uploads used a request size large enough to require Path MTU Discovery in the first place, while the earlier smoke tests' small requests never exercised that code path at all — the bug was invisible to every test the team had actually run, because none of them sent a packet close to the path's real MTU ceiling. The practical fix was defensive rather than fully corrective (the carrier's own misconfiguration was outside the team's control): set a conservative fixed MSS clamp on the ALB's IPv6 listener specifically, trading a small amount of per-packet overhead for eliminating the class of failure entirely, rather than depending on every network in the path correctly honoring PMTUD signaling.

ECMP — Equal-Cost Multipath Inside the Datacenter#

Zooming back inside a single network: once traffic has arrived at, say, a cloud region's network, there are usually multiple physically distinct, equal-cost paths to the same destination — a modern datacenter fabric (a Clos/leaf-spine topology) is deliberately built with many redundant links precisely so no single link or switch is a bottleneck or single point of failure. Equal-Cost Multipath (ECMP) is the mechanism routers use to actually spread traffic across all of those equal-cost next hops instead of picking just one and leaving the rest idle.

The naive approach — round-robin every packet across the available paths — would work for throughput but breaks TCP badly: packets from the same connection could arrive out of order via different-latency paths, forcing constant retransmission-buffer reordering. ECMP instead hashes a fixed set of packet header fields (source IP, destination IP, source port, destination port, protocol) into a small integer that selects one specific path — and every packet with the same 5-tuple always hashes to the same path, which keeps every individual TCP connection's packets flowing over one consistent link (avoiding reordering) while still distributing different connections across every available path roughly evenly.

Solid line = the path this specific connection's 5-tuple hashed to; dashed lines = paths available to other connections from the same source. One flow, one path — many flows, evenly spread.

⚠️ From the trenches: a platform team migrating inventory-service onto a new leaf-spine fabric saw aggregate bandwidth utilization graphs that looked perfectly healthy — total traffic spread evenly across all four spine switches — while one specific batch database-sync job consistently ran 4x slower than expected. The immediate cause: that one long-lived connection's 5-tuple happened to hash onto a spine switch that, unknown to the team, had a degraded optical transceiver silently running at reduced link speed instead of failing outright. Because ECMP hashing is deterministic per-flow, every retry of that same connection kept re-hashing to the same bad path — the underlying condition that made this land was that per-flow ECMP gives zero built-in mechanism to detect or route around one degraded (not fully down) member link; only an active link-health check independent of BGP/ECMP itself (in this case, the datacenter's own telemetry fabric flagging elevated CRC errors on that specific transceiver) surfaced the actual fault.

Anycast — One IP Address, Many Machines#

Anycast is the technique of announcing the exact same IP prefix from multiple, geographically distributed locations, and letting BGP's own best-path selection — specifically, AS_PATH length, which correlates loosely with network distance — route each client to whichever location is "closest" in BGP terms. There is no single "anycast machine": DNS root servers, most major CDNs, and public DNS resolvers like 1.1.1.1/8.8.8.8 all work this way — the same address genuinely terminates on different physical servers depending on where in the world a client's traffic enters the network.

The mechanic that makes anycast useful operationally is withdrawal-based failover: to take one location out of rotation (planned maintenance, or a real outage), that location simply stops announcing the anycast prefix via BGP. Every client whose traffic was entering there re-converges, within the same tens-of-seconds window covered earlier, onto the next-nearest surviving location — with zero DNS change, zero client-side reconfiguration, and no coordination with any client at all, because from the client's perspective the IP address never changed.

The client's request target never changes — only which physical location BGP steers it to.

Cloud providers increasingly sell anycast as a managed product rather than requiring you to run your own BGP sessions — AWS Global Accelerator and Cloud CDN's anycast IPs are both built on exactly this mechanism, giving a team the operational benefit of anycast (automatic, BGP-driven regional failover) without needing their own ASN, their own BGP peering relationships, or their own edge routing hardware.

IXPs, Transit, and Peering — How Networks Actually Connect#

Two ASes exchange traffic in one of two commercial/technical arrangements, and the distinction shapes both cost and latency:

An Internet Exchange Point (IXP) is the physical/logical infrastructure — a shared switching fabric, usually inside a carrier-neutral datacenter — where many networks connect once and can then establish peering sessions with many other members over that same shared fabric, dramatically cheaper than a dedicated private link to each individual peer. Large content networks (major CDNs, big cloud providers, big streaming platforms) peer aggressively at IXPs specifically because most of their traffic is to end users, not from other content sources — keeping that traffic off paid transit is a direct, material cost saving at scale, and the shorter, more direct peered path is also measurably lower latency for end users.

🔍 A Tier-1 network is, by the informal industry definition, one that reaches the entire internet through peering alone, purchasing no transit from anyone. Tier-2 networks (most regional ISPs, most large enterprises with their own ASN) peer where it's economical and buy transit for everything else. Tier-3 networks buy transit exclusively. This hierarchy is informal — there is no governing body that certifies "Tier-1" status — but it's a real, load-bearing mental model for understanding why a traceroute takes the path it does, and why some networks' peering disputes (a public one between two large networks refusing to upgrade their shared peering link capacity) can visibly degrade end-user experience for months.

BGP Session Mechanics and Convergence Tuning#

Everything above described BGP's policy layer — the previous sections skipped past how two routers actually establish and maintain the session that carries all of it. A BGP session is a plain TCP connection on port 179 between two routers' loopback or directly-connected interfaces — which has a concrete, practical consequence: anything that breaks ordinary TCP connectivity between two routers (an access-list change, an MTU mismatch, an intermediate firewall) breaks BGP the same way, and troubleshooting often starts with telnet <peer-ip> 179 or an equivalent TCP-level check before touching any BGP-specific tooling at all.

Once the TCP session is up, BGP peers exchange KEEPALIVE messages on a configurable interval (30 seconds is the common default) and track a hold timer (typically 90 seconds) — if three consecutive keepalives are missed, the session is declared down, every route learned from that peer is withdrawn, and the best-path algorithm re-runs across every affected prefix. This is the mechanism behind BGP's own convergence delay: a hard link failure is usually detected far faster by the underlying transport (a fiber cut triggers link-down at layer 1 almost instantly), but a soft failure — a peer that's still up but has stopped forwarding correctly — can take the full hold-timer duration to be noticed by BGP alone.

⚙️ Why this matters operationally: a network relying only on BGP's own keepalive/hold-timer defaults can take up to 90 seconds to detect and route around a peer that's silently failing to forward traffic correctly — an eternity for a latency-sensitive service. BFD is the standard production fix: it runs an independent, much faster liveness probe between the same two routers and, when it fails, immediately tells BGP to tear down the session rather than waiting out the full hold timer — turning a worst-case 90-second outage into a sub-second one. Any production BGP deployment fronting real user traffic should run BFD alongside it, not rely on default BGP timers alone.

Route Flapping and Dampening — When Stability Itself Becomes the Problem#

A link or session that repeatedly goes up and down — a flapping route — forces every downstream router to keep re-running the best-path algorithm and re-propagating the change, each time it flaps. A single unstable link deep in the network can generate a disproportionate amount of control-plane churn across a wide radius of routers that have nothing to do with the actual faulty link, simply because they all learned a route that depends on it. Route flap dampening is the classic mitigation: a router tracks a penalty score per prefix that increases with each flap and decays exponentially over time, and once the penalty crosses a threshold, the router suppresses (stops using and re-advertising) that route entirely until the penalty decays back below a reuse threshold — trading a temporarily unreachable prefix for protecting the rest of the network from the churn.

BGP Communities — Tagging Routes for Downstream Policy#

A BGP community is an optional, transitive tag — a plain 32-bit value, often written as two 16-bit numbers like 64510:100 — attached to a route, carrying no inherent meaning of its own beyond whatever policy the originating and receiving networks agree it signals. Communities are how operators build policy that would otherwise require constantly updating explicit prefix lists by hand: instead of a transit provider maintaining a manually updated list of "which of my customer's prefixes should never be re-announced to my other customers," the customer simply tags the relevant routes with a well-known community the transit provider's own router policy already understands, like NO_EXPORT (never re-announce this route to any external peer) or NO_ADVERTISE (don't even tell iBGP neighbors).

The single most operationally important provider-defined community in production networking today is Remote Triggered Black Hole (RTBH) — a community a transit or DDoS-scrubbing provider publishes specifically so a customer under attack can, in seconds, tag the targeted prefix and have the provider's network drop all traffic to it at the network edge, far upstream of the customer's own, much smaller-capacity link.

⚙️ Worked example — DDoS mitigation via RTBH: catalog-service's public API endpoint, 203.0.113.20/32, comes under a volumetric UDP-reflection DDoS attack that saturates the team's own 1 Gbps transit link, taking down every service behind that same link — including unrelated, healthy traffic to inventory-service's completely separate endpoint sharing the same upstream connection. The team's transit provider publishes RTBH support via the well-known community 64501:666. The on-call engineer announces a more-specific host route, 203.0.113.20/32, tagged with that community, with the BGP next-hop set to a reserved discard address (commonly 192.0.2.1, from the documentation-only TEST-NET range):

# On the customer's own edge router — announce a /32 for the attacked host,
# tagged with the provider's RTBH community, next-hop pointed at a discard address
router bgp 64510
  network 203.0.113.20/32 route-map RTBH-TRIGGER

route-map RTBH-TRIGGER permit 10
  set community 64501:666
  set ip next-hop 192.0.2.1

Within the same BGP convergence window covered earlier (tens of seconds), the transit provider's edge routers install a discard route for that one /32 and drop all traffic toward it before it ever reaches the customer's saturated link — sacrificing that one endpoint's availability entirely, deliberately, in exchange for restoring every other service sharing the link. This is a real, if blunt, tradeoff: RTBH makes the targeted service fully unreachable rather than degraded, which is why it's typically reserved for attacks large enough to threaten collateral damage to unrelated services, with cloud-native DDoS scrubbing services (AWS Shield Advanced, Cloudflare Magic Transit) increasingly preferred for smaller attacks because they can absorb and filter malicious traffic while keeping the legitimate service reachable throughout.

iBGP at Scale — Why Big Networks Use Route Reflectors#

Recall from earlier that iBGP requires every router inside an AS to learn every externally-learned route, so any internal router can find its way to an external destination. The naive way to achieve that is a full mesh — every iBGP router peers directly with every other iBGP router in the AS. This works cleanly at small scale, but the number of required sessions grows as n(n-1)/2 — 10 routers need 45 sessions, 50 routers need 1,225. Past a few dozen routers, a full iBGP mesh becomes a real operational burden: every new router added means manually configuring a session to every existing one.

Route reflectors solve this the same way a hub-and-spoke topology solves any all-to-all scaling problem: one or a small number of designated routers (the reflectors) peer with every other router (the clients), and are explicitly permitted to re-advertise (reflect) a route learned from one iBGP client to the other clients — something a plain iBGP router is normally forbidden from doing, specifically to prevent iBGP routing loops in a full-mesh design. A handful of route reflectors, each peered with every client but not with every other reflector's clients directly, replaces what would otherwise be thousands of individual sessions with a number that scales linearly instead of quadratically as the network grows.

Every client peers only with its reflector(s), not with every other client — a route learned by C1 reaches C3 by way of RR1 reflecting it to RR2, which reflects it onward to C3, with zero direct C1↔C3 session.

⚠️ From the trenches: a platform team running their own edge network for a multi-cloud interconnect deployed a single route reflector, reasoning it simplified the topology and their traffic volume didn't justify redundancy. During a routine OS patch reboot of that one reflector, every client router briefly lost its only path to learn about externally-sourced routes — not because any external BGP session actually went down, but because the mechanism distributing those already-valid routes internally had a single point of failure. The outage window was short (the reboot took under two minutes), but it took down internal reachability to every externally-learned prefix simultaneously, network-wide, for that window — a blast radius wildly disproportionate to "one router rebooted." The fix was mechanical once identified: always deploy route reflectors in redundant pairs (as shown in the diagram above), with every client peered to both, so a single reflector's maintenance or failure never removes internal route visibility entirely.

Full Worked Scenario: Taking checkout-service Multi-Region With Anycast#

checkout-service's platform team is expanding from a single AWS region (us-east-1) to a second region in Europe (eu-central-1), driven by a new regulatory requirement to keep EU customer checkout data processed within the EU, plus a genuine latency complaint — EU customers were seeing 180-220ms round-trip just crossing the Atlantic before their request even reached the load balancer.

Before the change: a single ALB in us-east-1, fronted by a standard (non-anycast) Elastic IP, advertised from AWS's own ASN. Every customer worldwide, regardless of location, connected to the same physical region.

The team's design, applying this chapter's concepts directly:

1. Deploy a second, fully independent stack (ALB, ECS/EKS workloads, RDS read replica configured for eventual promotion) in eu-central-1.
2. Front both regions with AWS Global Accelerator — a managed anycast service that advertises two fixed anycast IP addresses from AWS's own global network of edge locations, using exactly the BGP-driven routing this chapter describes, without the team needing their own ASN.
3. Configure Global Accelerator's traffic dial to route EU-originating requests to eu-central-1 and everything else to us-east-1, with automatic failover to the healthy region if either region's health checks fail — the same withdrawal-based mechanism covered above, fully managed.
4. Because the anycast IPs never change, no DNS TTL-driven propagation delay is in the failover path at all — a real operational win over the DNS-based multi-region failover pattern (covered in Chapter 5) the team had originally been evaluating, which depends on every resolver worldwide honoring a short TTL.

What went wrong the first time they tested regional failover: the team drained us-east-1 for a planned maintenance window by disabling its Global Accelerator endpoint group, expecting traffic to shift cleanly to eu-central-1. Instead, roughly 8% of North American traffic began failing with connection timeouts instead of failing over. The immediate cause: eu-central-1's security group had never been updated to accept traffic from Global Accelerator's EU-region edge IP ranges specifically — the original security group only allowlisted the US-region Global Accelerator ranges, because that was the only region in use when it was first written. The underlying condition: nobody had added "update security group allowlists in every region when adding a new Global Accelerator edge presence" to the region-expansion runbook, because the team's mental model of Global Accelerator was "one traffic-agnostic anycast frontend," not "a set of regional edge networks each with their own outbound IP ranges that must be explicitly trusted downstream." The fix took twenty minutes once diagnosed; the diagnosis itself took ninety, because every dashboard the team checked first (ALB health, ECS task health, RDS connections) looked completely healthy — the traffic was being rejected one hop before it ever reached anything the application team normally monitored.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: Your organization's edge router has two eBGP-learned paths to the same prefix 198.51.100.0/24 — one from AS64501 with Local Preference 100 and AS_PATH [64501, 64520], one from AS64509 with Local Preference 150 and AS_PATH [64509, 64521, 64522, 64520]. Which path wins, and why?

Answer: The path via AS64509 wins, even though its AS_PATH is longer (4 hops vs. 2). Local Preference is evaluated at step 2 of the best-path algorithm, strictly before AS_PATH length at step 4 — a higher Local Preference always wins regardless of how much longer that path's AS_PATH is. This is exactly why Local Preference is the tool of choice for deliberately steering outbound traffic toward a preferred exit, overriding whatever the "natural" shortest-AS-path route would otherwise be.

Problem 2: A team notices that curl from their office reaches their new service fine, but a synthetic monitoring probe running from a cloud region on the other side of the world times out entirely, with no TCP SYN-ACK ever received. ping to the same IP from the same failing region also fails. BGP session state on the team's own edge router shows "Established" with no flapping. What's the most likely root cause, and what's the fastest way to confirm it?

Answer: A route filtering issue at some point between the failing region and the team's network — possibly a prefix-length filter (if the announced block is longer than /24) or an IRR-based filter that was never updated when the prefix was announced. Since BGP session state on the team's own router is healthy, the problem isn't local — it's somewhere further out in the path. The fastest confirmation is a public looking-glass query from a network near the failing region (or from within that cloud provider's own network, if they publish one) — if the prefix is simply absent from that vantage point's routing table entirely (not present with a bad path, just missing), that confirms a filtering issue rather than a performance or hijack problem.

Problem 3: An anycast-fronted service fails over from Region A to Region B when Region A's health check fails. Customers with an already-open long-lived WebSocket connection to Region A report the connection simply drops with no graceful close, and must fully reconnect (losing any in-flight, un-acknowledged application state) rather than resuming. Is this expected anycast behavior, or a bug to fix?

Answer: Expected behavior, not a bug in the anycast mechanism itself — it's a mismatch between the traffic type and the tool. BGP withdrawal-based failover operates purely at the routing layer; it has no concept of an individual TCP connection's state and provides no mechanism to gracefully migrate or drain an existing connection to the new location. The fix belongs at the application/session layer, not the network layer: either design the client to detect a dropped connection and reconnect with idempotent resumption of in-flight state (the standard approach for anycast-fronted stateful protocols), or reconsider whether anycast is the right frontend for this specific traffic pattern versus a session-aware load balancer (covered in Chapter 2) that can drain connections before removing a backend from rotation.

Summary and What's Next#

BGP is the protocol that makes "the internet" a coherent, if occasionally fragile, single network out of tens of thousands of independently operated ones — every route your traffic takes, and every failover your service experiences at the network layer, ultimately traces back to the best-path algorithm and the announce/withdraw mechanics covered in this chapter. RPKI closes the biggest and most common hole in that trust model, but it's not complete, and route filtering — dull as it sounds — is behind a disproportionate share of "why can't network X reach us" incidents.

Part 2 moves one layer down the stack, from "how does traffic find its way to my network at all" to "once it arrives, how do I spread it across many backend instances correctly" — load balancing algorithms, health checking, connection draining, and the L4-vs-L7 tradeoff that shapes almost every production architecture decision from here on.