Zero-Downtime Deployments Are Hard - Especially When the Database Changes

Most “zero-downtime deployment” tutorials show you two identical stacks, a load balancer, and a switch you flip from blue to green. Traffic moves, nobody notices, everyone claps. That demo works because the demo has no database, no long-lived connections, and no in-flight work. Every hard part of a real deployment is exactly the part the demo removed.

The honest version: swapping stateless web servers is a solved problem. The moment your deploy also changes a database schema, drops a websocket, or invalidates a session, “zero downtime” becomes a set of design constraints on your application code, not a checkbox on your CI pipeline. This post walks the naive blue-green picture, breaks it on purpose, and maps each failure mode to a pattern that actually handles it.

The Naive Picture and Where It Breaks

Here is the blue-green story everyone starts with. You run two production environments. Blue is live. You deploy the new version to green, smoke test it, then repoint the load balancer.

                 ┌─────────────┐
   clients ────▶ │ load balancer│
                 └──────┬───────┘
                        │  (flip)
             ┌──────────┴──────────┐
             ▼                     ▼
        ┌─────────┐          ┌─────────┐
        │  BLUE   │          │  GREEN  │
        │ v1 (live)│         │ v2 (new)│
        └────┬────┘          └────┬────┘
             └───────┬────────────┘
                     ▼
             ┌───────────────┐
             │   database    │   ◀── shared, single copy
             └───────────────┘

Look at the bottom of that diagram. Blue and green are separate, but the database is not. You do not get a blue database and a green database, because the data is the state and you cannot fork it mid-flight. So the instant green needs a schema that blue cannot tolerate, your two “isolated” environments are coupled through the one thing you cannot duplicate.

That single shared box is where most zero-downtime incidents actually happen. The rest of this post is about the failure modes hiding in it and in the connections the diagram does not show.

Failure Mode 1: The Migration That Locks a Table

The classic outage is not a bad deploy. It is a migration that takes a lock the database holds while real traffic waits behind it.

-- Looks harmless. Is not.
ALTER TABLE orders ADD COLUMN region VARCHAR(32) NOT NULL DEFAULT 'IN';

On older Postgres or on MySQL, adding a NOT NULL column with a default can rewrite the entire table, holding an ACCESS EXCLUSIVE lock the whole time. On a 200 GB orders table that is minutes of every query blocking. Your app servers are healthy, green is running, the load balancer is happy, and every request 500s because they all touch orders.

The fix is to never let a schema change and the code that depends on it ship together. Split it.

-- Step 1: add the column, nullable, no default rewrite.
ALTER TABLE orders ADD COLUMN region VARCHAR(32);

-- Step 2 (separate migration, later): backfill in batches.
UPDATE orders SET region = 'IN'
WHERE region IS NULL AND id BETWEEN $1 AND $2;

-- Step 3 (after backfill completes): add the constraint.
ALTER TABLE orders ALTER COLUMN region SET DEFAULT 'IN';
ALTER TABLE orders ADD CONSTRAINT orders_region_nn
  CHECK (region IS NOT NULL) NOT VALID;
ALTER TABLE orders VALIDATE CONSTRAINT orders_region_nn;

NOT VALID lets you add the constraint without a full scan, then VALIDATE checks existing rows using a weaker lock that does not block writes. Every step is individually safe to run against live traffic. This is the core idea behind zero-downtime schema change, and it has a name.

The Pattern Underneath Everything: Expand and Contract

Expand-contract (also called parallel change) is the single most important pattern for deploying with a changing database. The rule is that the schema and the code are never allowed to require each other’s new version at the same instant. You get there in three phases that each ship independently.

The property that makes this work: at every phase boundary, the running code and the current schema are compatible in both directions. Old code can run against the new schema, and new code can run against the old schema. That two-way compatibility is what makes rollback and rolling deploys survivable.

Renaming a column is the canonical example. You never run ALTER TABLE ... RENAME. You add email_address, dual-write to both email and email_address, backfill, cut reads over, and only drop email a deploy or two later.

# During expand phase: the app writes both, reads the old one.
def save_user_email(conn, user_id, value):
    conn.execute(
        "UPDATE users SET email = %s, email_address = %s WHERE id = %s",
        (value, value, user_id),
    )

def load_user_email(conn, user_id):
    # Still reading old column until backfill is verified complete.
    row = conn.execute(
        "SELECT email FROM users WHERE id = %s", (user_id,)
    ).fetchone()
    return row["email"]

The cost is real: for the duration of the migration your code carries branches it will delete later, and you deploy three or four times to accomplish one logical rename. That is the price of not taking downtime. Teams that skip it are not faster, they are borrowing against an outage.

Failure Mode 2: In-Flight Requests During the Cutover

When you flip blue to green, some requests are already mid-execution on blue. A naive flip that kills blue immediately drops every one of them: half-written responses, orphaned transactions, a payment charged with no confirmation returned.

The pattern is connection draining, and it has to be cooperative between the load balancer and the app.

flip initiated
   │
   ├─▶ LB stops sending NEW connections to blue
   │
   ├─▶ blue keeps serving IN-FLIGHT requests (drain window, e.g. 30s)
   │
   ├─▶ app receives SIGTERM, stops accepting, finishes active work
   │
   └─▶ after drain window OR zero active requests → blue terminates

The application has to participate. On Kubernetes this means handling SIGTERM by flipping readiness to failing (so no new traffic arrives) while keeping liveness healthy (so you are not killed), then finishing open requests before exit.

import signal, threading

shutting_down = threading.Event()

def on_sigterm(signum, frame):
    # Fail readiness so the LB drains us; keep serving what we have.
    shutting_down.set()

signal.signal(signal.SIGTERM, on_sigterm)

def readiness_probe():
    return 503 if shutting_down.is_set() else 200

def liveness_probe():
    return 200  # stay alive so we are not force-killed mid-drain

The trap is terminationGracePeriodSeconds. If your drain window is 30 seconds but the orchestrator kills the pod after 10, you drop the last 20 seconds of requests every single deploy and blame it on “flakiness.” Match the grace period to your real p99.9 request duration, and put a hard cap on request time so a single slow handler cannot hold the whole rollout hostage.

Failure Mode 3: Long-Lived Websocket Connections

HTTP requests finish in milliseconds, so draining them is quick. A websocket can stay open for hours. You cannot wait for every socket to close on its own, and you cannot silently kill them without the client noticing a dead connection and a gap in messages.

Three things have to be true for websockets to survive a deploy:

• The client must reconnect automatically. Any long-lived connection needs exponential backoff reconnect logic, because sockets die for a hundred reasons that have nothing to do with deploys.
• Session state must not live in the socket’s process. If the connection carried in-memory state, reconnecting to a fresh server loses it. State belongs in Redis or the database, keyed by user, not by connection.
• The server must ask clients to leave gracefully. Instead of dropping sockets, send a “reconnect soon” signal so clients disconnect and come back over a spread-out window instead of all at once.

# On SIGTERM: tell connected clients to reconnect, staggered.
async def drain_websockets(connections):
    for i, ws in enumerate(connections):
        await ws.send_json({
            "type": "server_draining",
            # Spread reconnects over 60s to avoid a thundering herd.
            "reconnect_after_ms": (i % 60) * 1000,
        })
    # Give clients a moment to ack, then stop accepting new frames.

The thundering-herd risk is the subtle one. If 50,000 sockets all reconnect the instant green comes up, green falls over from the reconnect storm and you have caused the outage the deploy was meant to avoid. Jittered reconnect windows are not optional at scale. This is also why sticky sessions and websockets fight each other: the more state you pin to a specific server, the harder every one of these problems gets.

Failure Mode 4: Stateful Sessions and In-Memory Caches

If a user’s session lives in the memory of the blue server, flipping to green logs them out. The old answer was sticky sessions, pinning each user to one server. That directly breaks zero-downtime deployment, because draining that server has to drop that user.

The fix predates all of this: make the app tier stateless and push session state to a shared store.

# Bad: session in process memory. Dies with the server.
sessions = {}  # local dict

# Good: session in a shared store both blue and green can read.
def get_session(session_id):
    raw = redis.get(f"sess:{session_id}")
    return json.loads(raw) if raw else None

The same discipline applies to in-memory caches and feature flags. If green computes a cache differently than blue, users bounce between two behaviors during the cutover window. Either share the cache or make both versions produce compatible results for the overlap period, which is the same two-way-compatibility rule from expand-contract, applied to derived state.

Choosing a Strategy: Blue-Green vs Canary vs Rolling

These are not competitors so much as different trade-offs on the same axes: how fast you cut over, how much you can observe before committing, and how expensive rollback is.

Canary is the strongest when you genuinely do not trust a change, because you route a slice of real traffic to green, watch error rate and latency, and only widen if the metrics hold.

100% ─────────────────────▶ blue (v1)
                                       observe: error rate, p99, saturation
  5% ──▶ green (v2)  ◀── auto-rollback if metrics regress
                     ◀── ramp 5 → 25 → 50 → 100 if healthy

But notice what canary does not solve: the database is still shared. A canary running 5% of traffic on new code still writes to the same tables as the 95% on old code. If the new code writes a schema the old code cannot read, your 5% experiment corrupts data for the 95% majority. Canary reduces the blast radius of application bugs. It does nothing for schema incompatibility. Only expand-contract does that.

Putting It Together: A Deploy That Changes the Schema

Here is the end-to-end sequence for a change that renames a column, under canary, with live websockets. Each numbered step is a separate, independently revertible deploy.

1. EXPAND     migration: add new column (nullable, no rewrite)
2. DUAL-WRITE deploy code that writes old + new, reads old
              → canary 5% → 100%, socket drain with jittered reconnect
3. BACKFILL   batched UPDATE to populate new column for old rows
4. VERIFY     assert new column matches old for all rows (counts + spot checks)
5. READ-SWITCH deploy code that reads new, still writes both
              → canary, observe, ramp to 100%
6. STOP-WRITE deploy code that writes new only
7. CONTRACT   migration: drop old column   ◀── first irreversible step

The property that makes this safe: steps 1 through 6 are all reversible, because the old column stays populated and readable the whole time. You only cross the point of no return at step 7, long after the risky code has proven itself in production. If step 5 misbehaves, you route the canary back to old code in seconds and the data is still intact. The websocket drain and connection draining happen inside every code deploy (steps 2, 5, 6), independent of the schema work.

Honest Assessment

What actually works: expand-contract is non-negotiable for schema changes and it is the pattern that separates teams who deploy calmly from teams who schedule maintenance windows. Connection draining with a correctly sized grace period handles in-flight HTTP. Stateless app servers with shared session state make the whole blue-green flip trivial for everything except the database.

What does not work: pretending the database is just another stateless tier. It is not. You cannot fork it, you cannot flip it, and every “instant rollback” story quietly assumes the schema is backward compatible. If you deploy a schema and its dependent code together, no amount of load-balancer cleverness saves you.

What to actually do: separate every schema change from the code that needs it, and make each half safe to run alone. Add columns nullable, backfill in batches, add constraints with NOT VALID then VALIDATE. Size your termination grace period to real request durations, not to a default. Give every long-lived connection automatic jittered reconnect and keep its state out of the process. Use canary when you want to observe a risky change, blue-green when you want a fast reversible switch, rolling when you are stateless and cost-sensitive. And accept the real cost up front: zero downtime means more deploys, more transitional code, and more discipline, not a better load balancer. The teams that ship without outages are not using a magic tool. They are paying that tax on every change.