Federating clusters for zero-downtime Kubernetes

Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat them as one thing. Failover becomes a runbook: restore, repoint DNS, and wait for an outage that, on paper, you’d already paid to survive.

Linkerd’s multicluster extension closes that gap by letting several clusters present a service as a single, load-balanced endpoint. The part that the official tasks gloss over is that a real platform almost never picks one multicluster mode. Some services want federation (same service everywhere, one endpoint, automatic failover). While others want mirroring (reach a specific remote service by name). And you frequently want both patterns living on the same set of links. The docs walk through each mode on its own. This post wires all three together across three GKE clusters, with a full-mesh link topology, a chaos test that takes out an entire cluster, and scripts you can clone and run on a fresh GCP project.

Companion repo: Every script referenced here lives in this repository. Feel free to clone it, set your project ID, and run it.

Linkerd multicluster modes: Gateway, flat, and federated

Linkerd’s multicluster extension supports three modes. The nice thing is they’re not mutually exclusive: on the same set of linked clusters, the mode is chosen per service via a label.

Mode

Label

What happens

Network Requirement

Hierarchical (gateway)

mirror.linkerd.io/exported=true

Service mirrored as -federated` service for federated workloads, and every cluster can consume mirrored services from any other.

The wrinkle is the gateway. Only `east` needs one (it’s the only cluster exporting `analytics` hierarchically), so we enable the gateway in east’s install and leave everyone else gatewayless. One install per cluster, all flags at once, no re-running install a second time to bolt a gateway on afterward:

```bash # west: gatewayless, with one controller per cluster it consumes from linkerd --context west multicluster install --gateway=false \ --set controllers[0].link.ref.name=east \ --set controllers[1].link.ref.name=north \ --set controllers[2].link.ref.name=east-gw \ | kubectl --context west apply -f - # east: gateway enabled here, controllers for the clusters it consumes linkerd --context east multicluster install --gateway=true \ --set controllers[0].link.ref.name=west \ --set controllers[1].link.ref.name=north \ | kubectl --context east apply -f - # north: gatewayless, controllers for west, east, and east's gateway link linkerd --context north multicluster install --gateway=false \ --set controllers[0].link.ref.name=west \ --set controllers[1].link.ref.name=east \ --set controllers[2].link.ref.name=east-gw \ | kubectl --context north apply -f -

Note the controller count. The service-mirror controller runs on the consuming side, one per link. `west` and `north` each consume `analytics` via the gateway, so they get a third controller for the `east-gw` link; `east` doesn’t consume its own analytics, so it only needs two.

Then we generate the links. Flat/federated links use `–gateway=false`; the gateway-aware link to `east` (for the analytics export) is a separate link named `east-gw`:

```bash # Flat links (no gateway) — for federated + flat-mirrored services linkerd --context east multicluster link-gen --cluster-name=east --gateway=false \ | kubectl --context west apply -f - linkerd --context west multicluster link-gen --cluster-name=west --gateway=false \ | kubectl --context east apply -f - # ... (all six directions) # Gateway-aware link from east (for the analytics hierarchical export) linkerd --context east multicluster link-gen --cluster-name=east-gw \ | kubectl --context west apply -f - linkerd --context east multicluster link-gen --cluster-name=east-gw \ | kubectl --context north apply -f - ```

After this, `linkerd multicluster check` on any cluster should report every linked cluster healthy.

Step 4: Deploy the demo services

Deploy the demo workloads. The next sections label them for federation, flat mirroring, and gateway mirroring and show what each mode creates.

```bash ./scripts/04-deploy-app.sh ```

Three services, three modes, and deliberately the same `buoyantio/bb` image for all of them, a tiny HTTP server that echoes a fixed string. The application isn’t the point. The point is that one `kubectl label` changes how Linkerd treats the service across clusters, with everything else held constant.

frontend (federated)

Deploy to all three clusters with a per-cluster response string, then labeled for federation:

```bash for ctx in west east north; do kubectl --context $ctx -n mc-demo label svc/frontend mirror.linkerd.io/federated=member done ```

Within a few seconds, `frontend-federated` shows up in all three clusters:

```bash $ kubectl --context west -n mc-demo get svc NAME TYPE CLUSTER-IP PORT(S) AGE frontend ClusterIP 10.104.1.50 8080/TCP 45s frontend-federated ClusterIP 10.104.2.100 8080/TCP 10s ```

api (flat-mirrored)

Label the api service in `west` and `east` for flat export:

```bash kubectl --context west -n mc-demo label svc/api mirror.linkerd.io/exported=remote-discovery kubectl --context east -n mc-demo label svc/api mirror.linkerd.io/exported=remote-discovery ```

Now `north` can see `api-west` and `api-east` as separate services:

```bash $ kubectl --context north -n mc-demo get svc NAME TYPE CLUSTER-IP PORT(S) AGE frontend ClusterIP 10.120.1.50 8080/TCP 45s frontend-federated ClusterIP 10.120.2.100 8080/TCP 10s api-west ClusterIP 10.120.3.20 8080/TCP 5s api-east ClusterIP 10.120.3.21 8080/TCP 5s ```

The client in `north` picks `api-west` or `api-east` explicitly. Traffic will go straight to the remote pods with no gateway in the path.

analytics (gateway-mirrored)

Next, deploy only to `east`, labeled for hierarchical (gateway) export:

```bash kubectl --context east -n mc-demo label svc/analytics mirror.linkerd.io/exported=true ```

This creates `analytics-east-gw` in `west` and `north`, routed through east’s Linkerd gateway:

```bash $ kubectl --context west -n mc-demo get svc analytics-east-gw NAME TYPE CLUSTER-IP PORT(S) AGE analytics-east-gw ClusterIP 10.104.5.10 8080/TCP 5s ```

The endpoints for this service point at east’s gateway IP, not the analytics pods directly. That’s the right trade when you can’t guarantee flat-network connectivity, or when you specifically want the gateway handling load balancing and mTLS termination.

Step 5: Verify all three modes

Generate traffic against all three service patterns and verify that each resolves the way you expect.

```bash ./scripts/05-verify.sh ```

This deploys a traffic generator in `north` that hits all three service patterns in a loop and tails the logs. The response strings come straight from the deployments, so you’ll see which cluster served each request:

``` [federated] frontend from east [federated] frontend from west [federated] frontend from north [flat-west] api from west [flat-east] api from east [gateway] analytics from east ```

You can also inspect endpoints to see how differently each mode resolves:

```bash # Federated: endpoints span all three clusters $ linkerd --context west diagnostics endpoints frontend-federated.mc-demo.svc.cluster.local:8080 NAMESPACE IP PORT POD SERVICE mc-demo 10.100.1.15 8080 frontend-xxx-west frontend.mc-demo mc-demo 10.108.0.42 8080 frontend-xxx-east frontend.mc-demo mc-demo 10.116.0.33 8080 frontend-xxx-north frontend.mc-demo # Flat mirror: endpoints are the remote pod IPs $ linkerd --context north diagnostics endpoints api-west.mc-demo.svc.cluster.local:8080 NAMESPACE IP PORT POD SERVICE mc-demo 10.100.2.10 8080 api-xxx-west api.mc-demo # Gateway mirror: the endpoint is east's gateway IP on port 4143 $ kubectl --context west -n mc-demo get endpoints analytics-east-gw NAME ENDPOINTS AGE analytics-east-gw 35.186.xxx.xxx:4143 30s ```

Three modes, one mesh, one set of clusters, and the only difference between them is a label.

Step 6: The chaos test, kill a cluster

This is where federation earns its keep. We simulate a full cluster failure and watch how each service type reacts.

```bash ./scripts/06-chaos-test.sh ```

The script scales every deployment in `east` to zero replicas (standing in for a cluster outage), then samples traffic from `north` across all three patterns.

Federated service (`frontend-federated`):

``` Before: west=33% east=33% north=33% After: west=50% north=50% ← automatic rebalance, zero errors ```

Traffic redistributes immediately. No errors, no config changes. As east’s pods drop out of the endpoint list, Linkerd’s load balancer simply spreads requests across what’s left.

Flat-mirrored service (`api-east`):

``` Before: api-east responds normally After: api-east returns 503s ← expected: the remote pods are gone ```

This is the correct behavior. The client explicitly asked for `api-east`, and east is down. Handling that is the client’s job: fail over to `api-west`, retry, or front the two with a TrafficSplit. Mirroring hands you control; federation hands you automation.

Gateway-mirrored service (`analytics-east-gw`):

``` Before: analytics-east-gw responds normally After: analytics-east-gw returns 502s ← the gateway is down too ```

Same story here, the client asked for a specific remote, and that remote is gone.

Bring east back:

```bash kubectl --context east -n mc-demo scale deploy --all --replicas=1 kubectl --context east -n mc-demo scale deploy/frontend --replicas=3 ```

(The script restores `frontend` to its full `FRONTEND_REPLICAS` count rather than leaving it at 1, otherwise east would rejoin the federation under-weighted, landing around 14% instead of an even third.) Within 15–30 seconds all three patterns recover: the federated service rebalances back to 33/33/33, and the mirrored services start answering again.

The lesson worth carrying out of this: federation is the right default for anything that should simply be available everywhere. Mirroring, flat or gateway, is the right call when the client genuinely needs to know which cluster it’s talking to.

Step 7: Teardown

When you’re finished with the demo, run the teardown script to remove all the infrastructure and avoid ongoing GCP charges.

```bash ./scripts/99-teardown.sh ```

This removes all three clusters, the VPC peerings, subnets, firewall rules, and VPCs created by the earlier steps. Run it when you’re done so the meter stops.

Selecting your Linkerd multicluster architecture strategy

After running all three side by side, here’s the decision framework I’d hand a teammate:

Question

→ Mode

Should the client be cluster-agnostic?

Federated

Does the client need to pick a specific cluster?

Flat mirror

Is there no flat network between clusters?

Gateway mirror

Do you need automatic failover with no app changes?

Federated

Do you need traffic splitting with explicit weights?

Flat mirror + TrafficSplit

Is the service a singleton (only in one cluster)?

Mirror (flat or gateway)

And you can mix them freely in the same mesh. The label on each service decides its behavior independently of the others.

Linkerd multicluster gotchas and configuration lessons

The gotchas that cost us time and don’t jump out of the docs:

VPC peering route exchange. Creating the peering isn’t enough. You have to pass `–export-custom-routes` and `–import-custom-routes` on both sides, or the pod CIDRs never get advertised. The symptom is brutal to diagnose: DNS resolves fine, then connections just hang. Maddening to debug.

Regional clusters multiply your nodes. A regional cluster with `–num-nodes 1` quietly gives you three nodes (one per zone). We pin `–node-locations` to a single zone to keep it at one. Easy to miss until the bill arrives.

Overlapping CIDRs. GKE auto-allocates large ranges out of the `10.0.0.0/8` space by default, and three clusters built with defaults will overlap, at which point peering fails silently. Always set explicit, non-overlapping `–cluster-ipv4-cidr` and `–services-ipv4-cidr`.

Controller count matters. Each cluster needs one service-mirror controller per link it consumes. Miss one and the Link CR is created, but nothing mirrors, and `linkerd multicluster check` still looks green, so you’ll stare at it for a while before the penny drops.

Federated service naming is fixed. The federated service is always `-federated`; you can’t change the suffix. Clients have to target `frontend-federated`, not `frontend`. Plan your naming around it, or use a TrafficSplit to point `frontend` at `frontend-federated`.

Gateway and flat can’t share one link. A single Link CR is either gateway or flat, not both. To get both behaviors to the same cluster you create two links with different names. That’s why our setup uses `east` (flat) and `east-gw` (gateway) as separate links, with a matching controller for each on the consuming clusters.

Production checklist

Bidirectional links between all clusters (full mesh) so every cluster has the federated service

cert-manager with a shared CA instead of hand-rolled `step` certificates

Separate issuer certs per cluster (don’t skip it!)

NetworkPolicies restricting cross-cluster traffic to only the services that need it

Linkerd authorization policies for fine-grained access control

Monitoring: pipe Linkerd-Viz metrics into your Prometheus/Grafana stack, and alert on a federated service’s endpoint count dropping

GitOps: keep Link CRs and multicluster config in version control

Test failover regularly: scale a cluster to zero in staging and confirm traffic redistributes

Key takeaways: Mastering multi-region Linkerd deployments

The docs show each multicluster mode in isolation; real platforms need all three at once. Federation covers the common case: The same service everywhere, automatic failover, and nothing to change in the app. Flat mirrors give you explicit, cluster-aware routing when data locality matters. Gateway mirrors get you cross-cluster reach when a flat network isn’t on the table.

What surprised me most about building this is how little of it is genuinely complex. It’s mostly wiring. Once the trust anchor is shared and the links are up, adding a service to the federation is a single `kubectl label`, and removing a cluster is as simple as letting it go down. The mesh adjusts on its own.

For teams running across regions, that’s a real chunk of operational toil gone: your services run everywhere, traffic finds the healthy copies, and you pick the multicluster mode per service based on what that service actually needs.

References:

– [Linkerd Federated Services Task Guide](https://linkerd.io/2-edge/tasks/federated-services/): official walkthrough for federation

– [Linkerd Multicluster Reference](https://linkerd.io/2-edge/reference/multicluster/): architecture and mode details

– [Linkerd Multicluster Communication Guide](https://linkerd.io/2-edge/tasks/multicluster/): hierarchical mirroring walkthrough

– [Installing Multicluster Components](https://linkerd.io/2-edge/tasks/installing-multicluster/): installation reference

– [Linkerd 2.17 Announcement](https://linkerd.io/2024/12/05/announcing-linkerd-2.17/): federated services introduction

– [GKE VPC-native Clusters](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/alias-ips): flat networking on GCP