Subscription billing looks like a cron job and a charge() call, and that is exactly the trap. “Every month, charge each active customer their plan price” sounds like one scheduled loop. The reason interviewers like it is that almost every word in that sentence hides a correctness landmine. “Every month” means a durable scheduler that fires once per customer per period even across deploys and crashes. “Charge” means a call to an external payment gateway that is slow, can time out ambiguously, and whose webhooks arrive late, duplicated, and out of order. “Their plan price” means proration math when they upgrade on day 14, add seats on day 3, and downgrade the next cycle. And the invariant that ties it together is brutal: a customer must be billed exactly once per billing period, for exactly the right amount, no matter how many times the scheduler fires, the worker retries, or the gateway re-delivers a webhook.

Double-charging a customer is a refund, a support ticket, a chargeback, and a trust hit. Under-charging is silent revenue loss that nobody notices until the finance team reconciles. Both come from the same root cause: a charge pipeline that is not idempotent against an at-least-once world. So the real problem is not “run a loop.” It is “build an idempotent, retryable, exactly-once charge pipeline on top of a scheduler that over-fires, a gateway that is eventually consistent, and plan changes that rewrite the amount mid-flight.” Let me build it properly.

Functional Requirements (FR)

In scope:

  • Plans and subscriptions. Define plans (price, currency, billing interval - monthly/annual, metered vs flat, trial length). A customer subscribes to a plan; the subscription has a lifecycle (trialing, active, past_due, canceled).
  • Recurring charges on schedule. At the end of each billing period, generate an invoice for each active subscription and charge the customer’s saved payment method automatically.
  • Mid-cycle plan changes with proration. A customer can upgrade, downgrade, or change seat count partway through a cycle. We credit the unused portion of the old plan and charge the prorated portion of the new one, either immediately or rolled into the next invoice.
  • Idempotent, exactly-once charging. A given invoice is charged against the gateway exactly once, even if the scheduler fires twice, a worker retries, or a webhook is re-delivered. No double charges, ever.
  • Failed-payment retries (dunning). A declined charge is retried on an exponential-backoff schedule (e.g. day 1, 3, 5, 7) before the subscription is marked past_due and eventually canceled. Customers are notified between attempts.
  • Webhook ingestion. Consume asynchronous gateway webhooks (payment_succeeded, payment_failed, dispute_created, refund) that arrive late, out of order, and duplicated, and reconcile them against our invoice state correctly.
  • Invoice history and receipts. Every customer can see past invoices, line items (base, proration, credits, tax), status, and download receipts.

Explicitly out of scope (own the scope by naming it):

  • The payment gateway internals. I integrate with Stripe/Adyen/Braintree as a black box: a create_charge API with an idempotency key, plus asynchronous webhooks. I do not design the card network, the acquiring bank, or PCI card-vault storage - the gateway holds the card; I hold a token.
  • Tax calculation engine. I call a tax service (Avalara/Stripe Tax) as a dependency and store the result on the invoice line; I do not design the tax-rules engine itself.
  • Fraud and sanctions screening. Relevant but a separate system; I note the hook, not the design.
  • The pricing/catalog UI and the dunning email templates. I expose APIs and emit events; the app and email service are downstream consumers.

The decision that drives everything: the charge is a call to an irreversible, eventually-consistent external gateway, so the invoice is a persisted state machine with an idempotent charge pipeline - not a function that returns success or failure synchronously. Correctness comes from an idempotency key per charge attempt, a monotonic invoice state machine, and webhook handlers that are safe to run twice - not from trusting the gateway’s synchronous response.

Non-Functional Requirements (NFR)

  • Scale: 50M active subscriptions. Spread over a month that is only ~19 invoices/sec average, but billing is bursty - many merchants bill on the 1st of the month, and a naive design stampedes. Peak during a month-boundary run can be 10-50x the average. The hard part is correctness and the burst, not steady-state QPS.
  • Latency: billing is asynchronous; the customer is not waiting on a request. What matters is that an invoice completes (charged or entered dunning) within minutes of its due time, and that the customer-facing API (view invoices, change plan) returns p99 under ~300ms. The charge itself can take seconds (gateway round-trip) and that is fine because it is a background job.
  • Availability: 99.99% on the customer-facing API and webhook ingestion (dropping a webhook can lose a payment outcome). The billing runner degrades safely: if the gateway is down, invoices queue and retry; nothing is dropped or double-charged.
  • Consistency: strong on the money path. An invoice’s state and the charge-attempt ledger are strictly ordered and monotonic; an invoice can never be charged twice. We do not trade consistency for availability when money moves.
  • Durability: absolute, zero loss. Once an invoice is created or a payment outcome is recorded, it survives any node loss. A lost payment_succeeded means we re-charge a paying customer; a lost invoice means we never bill. Both are incidents, so every money event is synchronously replicated and never hard-deleted.
  • Correctness of money: amounts are exact integer minor units; proration arithmetic is deterministic and reproducible; the sum of invoice line items equals the charged amount to the cent. Billing is audited by finance and must reconcile.

The tension: recurring charges over an eventually-consistent, irreversible gateway want perfect ordering and exactly-once semantics, which fight the bursty, high-volume schedule. We resolve it by making the scheduler cheap and over-firing (at-least-once) but the charge pipeline idempotent, so firing twice is harmless; and by spreading the burst with jittered, per-subscription anchor dates instead of billing everyone at midnight.

Back-of-the-Envelope Estimation (BoE)

Real numbers, because they tell us this is a burst-and-correctness problem, not a steady-throughput one.

Billing volume (invoices generated + charged):

50,000,000 active subscriptions
Monthly billing => 50M invoices/month
= 50,000,000 / (30 * 86,400) sec
≈ 19 invoices/sec average

Nineteen per second is nothing. But billing is not uniform:

If a naive design bills everyone at 00:00 on the 1st:
  a large fraction of 50M lands in a few hours
  say 40% of monthly volume in a 6-hour window on day 1
  = 20M / 21,600 sec ≈ 925 invoices/sec sustained, higher at the spike
Each invoice => 1+ gateway charge call + several state transitions + webhooks

So the real number the system must survive is the burst, ~1k/sec sustained for hours if we let everyone share an anchor date, far more at the leading edge. The fix (below) is to anchor each subscription to its own signup day and jitter within the day, flattening the curve back toward the ~19/sec average.

Charge attempts including retries (dunning):

Assume ~8% of charges fail on first attempt (expired cards, insufficient funds).
First attempts:  50M/month
Retries:         8% * 50M * ~2 avg retries ≈ 8M/month
Total charge calls ≈ 58M/month ≈ 22/sec average

Webhook volume:

Each charge call yields 1-2 webhooks (succeeded/failed, sometimes intermediate).
Plus disputes, refunds, payment-method updates.
≈ 58M * 1.5 + overhead ≈ ~90M webhooks/month ≈ 35/sec average,
bursting with the billing run. Each MUST be processed idempotently.

Storage per year:

invoice row            ~600 bytes (customer, amounts, state, period, refs)
invoice line items (~3) ~3 * 200 = 600 bytes
charge attempts (~1.1)  ~1.1 * 250 ≈ 280 bytes
webhook events (~1.5)   ~1.5 * 300 = 450 bytes
------------------------------------------------
≈ 1.9 KB per invoice, round to ~2.5 KB with indexes/overhead

50M invoices/month * 2.5 KB ≈ 125 GB/month
* 12 ≈ 1.5 TB/year

Invoices are financial records retained 7+ years and never deleted, so plan ~10-12 TB over the window. Modest. The story is retention and immutability, not sharding for space.

Idempotency / dedup key store:

One idempotency key per charge attempt, kept long enough to cover retries
and late webhooks - say 30 days hot.
58M attempts/month => ~58M keys hot.
Per key: key(40) + status(1) + invoice_id(16) + response(~200) ≈ ~260 bytes
58M * 260 ≈ 15 GB

The headline: steady-state throughput is trivial (~20/sec), but a naive shared anchor date turns it into a ~1k/sec multi-hour stampede, and every one of those ~90M monthly webhooks must be exactly-once. The engineering spends itself on flattening the burst and on an idempotent charge pipeline, not on raw QPS.

High-Level Design (HLD)

The spine: a Billing Scheduler enqueues due subscriptions (at-least-once, over-firing is safe), a Billing Runner generates an Invoice (a persisted state machine) and computes line items including proration, a Charge Pipeline calls the Payment Gateway with a per-attempt idempotency key, a Webhook Consumer ingests asynchronous gateway callbacks and advances invoice state idempotently, and a Dunning Service schedules exponential-backoff retries for failures. Everything money-related is backed by an append-only ledger.

                   ┌──────────────────────────────────┐
                   │  Customer app / Admin / API caller │
                   └───────┬──────────────────┬─────────┘
          change plan /    │                  │  view invoices
          subscribe        │                  │
                   ┌───────▼──────────────────▼─────────┐
                   │        Subscription API              │
                   │  authN/Z, plan changes, proration    │
                   │  Stateless, horizontally scaled      │
                   └───────┬──────────────────────────────┘
                           │ writes subscription + next_billing_at
                   ┌───────▼──────────────────────────────┐
                   │   Subscriptions / Plans DB (SQL)       │
                   │   source of truth, strongly consistent │
                   └───────┬──────────────────────────────┘
                           │ due? (next_billing_at <= now)
                   ┌───────▼──────────────────────────────┐
                   │        Billing Scheduler               │
                   │  scans due subs in shards, enqueues    │
                   │  at-least-once (over-fire is safe)     │
                   └───────┬──────────────────────────────┘
                           │ bill(subscription_id, period)
                   ┌───────▼──────────────────────────────┐
                   │        Billing Runner (workers)        │
                   │  create/resume Invoice state machine   │
                   │  compute line items + proration + tax  │
                   └───────┬──────────────────────────────┘
                           │ invoice OPEN -> charge
                   ┌───────▼──────────────────────────────┐
                   │        Charge Pipeline                 │
                   │  per-attempt idempotency key           │
                   │  pre-write CHARGING, then call gateway  │
                   └───┬───────────────────────────┬────────┘
             charge    │                           │  record outcome
          ┌────────────▼───────────┐     ┌─────────▼──────────────┐
          │   Payment Gateway       │     │  Ledger (append-only)   │
          │   (Stripe/Adyen)        │     │  immutable money events │
          │   idempotent charge API │     └─────────────────────────┘
          └────────────┬───────────┘
                       │ async webhooks (late, dup, out-of-order)
          ┌────────────▼───────────────────────────────────┐
          │        Webhook Consumer                          │
          │  verify sig, dedup on event_id, apply idempotent │
          │  advance invoice PAID / FAILED, feed dunning      │
          └────────────┬───────────────────────┬─────────────┘
                 paid   │                       │ failed
          ┌─────────────▼──────┐        ┌───────▼───────────────┐
          │  Invoice -> PAID    │        │  Dunning Service        │
          │  period advanced    │        │  exp-backoff retry sched│
          │  receipt emitted    │        │  day 1,3,5,7 -> past_due│
          └────────────────────┘        └─────────────────────────┘

Billing run path (asynchronous, no customer waiting):

  1. The Scheduler periodically scans subscriptions whose next_billing_at <= now (in shards, so the scan is parallel and bounded) and enqueues a bill(subscription_id, period_key) message. It is allowed to over-fire; the pipeline dedupes.
  2. A Billing Runner worker consumes the message. It creates or resumes the invoice for (subscription_id, period_key) - a unique constraint on that pair means a double-enqueue maps to the same invoice, never two. It computes line items: base price, any proration adjustments accrued this period, metered usage, credits, and tax. Invoice state -> OPEN.
  3. The Charge Pipeline transitions OPEN -> CHARGING, generates a deterministic idempotency key for this attempt, and calls the gateway’s charge API with it. The gateway guarantees one charge per key even if we retry after an ambiguous timeout.
  4. The gateway returns synchronously (success/decline/pending) and later sends a webhook. We treat the webhook as the source of truth for the final outcome; the synchronous response is an optimization.

Webhook / outcome path (asynchronous, idempotent):

  1. The Webhook Consumer verifies the signature, dedupes on the gateway event_id, and applies the outcome idempotently against the invoice via an optimistic state guard. A duplicate or out-of-order webhook is a no-op or is reordered by sequence.
  2. On payment_succeeded: invoice CHARGING -> PAID, post a ledger entry, advance next_billing_at by one interval, emit a receipt event.
  3. On payment_failed: invoice CHARGING -> PAYMENT_FAILED, hand to the Dunning Service, which schedules the next retry on an exponential-backoff curve and notifies the customer. After the final retry, subscription -> past_due -> eventually canceled.

The structural insight: the scheduler is cheap and over-fires; the charge pipeline is idempotent; the webhook is the authority. Exactly-once billing is not achieved by making the scheduler perfect (impossible across crashes) - it is achieved by making every downstream step safe to run twice, keyed on (subscription, period) for the invoice and on a per-attempt idempotency key for the charge.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the idempotent, exactly-once charge pipeline against an at-least-once gateway and duplicated/out-of-order webhooks, (2) the recurring scheduler and flattening the billing burst, (3) mid-cycle plan changes and proration, (4) failed-payment retries with exponential backoff (dunning).

1. The idempotent, exactly-once charge pipeline

The job: charge the customer exactly once per invoice, even though the scheduler can enqueue twice, the worker can crash mid-charge, the gateway call can time out ambiguously, and the gateway’s webhooks arrive late, duplicated, and out of order. Unlike a wire, a card charge can be refunded - but a double charge is still a refund, a support ticket, and a chargeback you want to avoid entirely.

Naive approach: scheduler fires, worker calls charge(), marks the invoice paid

def on_billing_due(subscription):
    amount = subscription.plan.price
    result = gateway.charge(subscription.card_token, amount)   # external call
    if result.success:
        db.set_invoice_paid(subscription.id)
    else:
        db.set_invoice_failed(subscription.id)

Where it breaks:

  • Scheduler double-fire => double charge. The scheduler enqueues bill(sub) twice (redelivery, retry after a slow ack, two scheduler nodes overlapping). Two workers each call charge(). The customer is billed twice. There is no key tying both attempts to one invoice.
  • Crash between charge and record => double charge on retry. The gateway accepts the charge, then the worker dies before writing paid. The message is redelivered; the worker charges again. Real money taken twice.
  • Ambiguous timeout. gateway.charge() times out. Did it go through? Retrying may double-charge; not retrying may lose the payment. The error alone cannot tell you.
  • Out-of-order webhooks corrupt state. The gateway sends payment_failed (first attempt) and later payment_succeeded (a retry), but the network delivers succeeded first and failed second. The naive handler overwrites paid with failed and starts dunning a customer who already paid.

At-least-once delivery plus a non-idempotent external charge call plus out-of-order webhooks is the core failure mode of every billing system: duplicate charges and corrupted outcomes.

Evolved approach: invoice keyed on (subscription, period) + per-attempt idempotency key + pre-written CHARGING state + idempotent, sequenced webhook handlers

Compose exactly-once from four mechanisms, each closing one hole above.

1. One invoice per (subscription, period), enforced by a unique constraint. The unit of billing is not the message; it is the invoice. The first thing the runner does is INSERT ... ON CONFLICT DO NOTHING an invoice row keyed on (subscription_id, period_key), where period_key is the billing period (e.g. 2026-11). A double-enqueue hits the conflict and both workers converge on the same invoice row. The scheduler is now free to over-fire.

INSERT INTO invoices (invoice_id, subscription_id, period_key, state, amount, ...)
VALUES (:new_id, :sub, :period, 'DRAFT', :amount, ...)
ON CONFLICT (subscription_id, period_key) DO NOTHING;
-- then read back the (single, canonical) invoice for this period
SELECT * FROM invoices WHERE subscription_id=:sub AND period_key=:period;

2. A deterministic idempotency key per charge attempt, toward the gateway. Every real gateway (Stripe, Adyen, Braintree) accepts a client idempotency key and guarantees it charges once per key. We derive it deterministically from the invoice and attempt number: key = invoice_id + ":attempt:" + attempt_no. The same key is passed on every retry of the same attempt, so a retry after an ambiguous timeout either completes the original charge or returns its already-known result - never a second charge. A new attempt (next dunning retry) uses a new key because it is deliberately a new charge.

3. Pre-write the CHARGING state with the exact key, then call, then confirm. Before calling the gateway, the pipeline writes CHARGING plus the idempotency_key it will use, in a committed transaction. Only then does it call the gateway. After any crash, recovery sees CHARGING with a known key and knows exactly which key to re-send - it never asks “did I charge something?”

def charge_invoice(invoice):
    if invoice.state in ('PAID', 'CHARGING'):
        # PAID: done. CHARGING: an attempt is already in flight with a known key;
        # re-drive with the SAME key, which the gateway dedupes.
        return resume_or_noop(invoice)
    if invoice.state != 'OPEN':
        raise IllegalTransition

    attempt = invoice.attempt_no + 1
    key = f"{invoice.id}:attempt:{attempt}"
    db.transition(invoice.id, 'OPEN', 'CHARGING',
                  idempotency_key=key, attempt_no=attempt)   # pre-write, commit

    try:
        resp = gateway.charge(invoice.card_token, invoice.amount,
                              idempotency_key=key)            # SAME key on retry
    except TimeoutOrNetworkError:
        raise RetryableError    # safe: the key makes a re-send a no-op

    # synchronous response is an optimization; the webhook is authoritative.
    if resp.status == 'succeeded':
        apply_success(invoice, resp)      # also reachable via webhook; idempotent
    elif resp.status == 'failed':
        apply_failure(invoice, resp)      # hand to dunning
    # 'pending' => do nothing; wait for the webhook.

4. Idempotent, sequenced webhook handling. Webhooks are the authoritative outcome, so they must be bulletproof:

  • Dedup on the gateway’s event_id. Store every processed event_id; a redelivered webhook is dropped at the door.
  • Apply with an optimistic state guard, so applying the same outcome twice is a no-op: UPDATE invoices SET state='PAID' WHERE invoice_id=:id AND state='CHARGING'. Zero rows changed means it was already applied.
  • Ignore stale/out-of-order transitions. A payment_failed webhook for attempt:1 arriving after the invoice is already PAID from attempt:1 is dropped because the state guard does not match, and because each webhook carries the attempt’s idempotency key / charge id - we only act on the webhook whose charge id matches the invoice’s current attempt. Terminal PAID refuses any failure transition.
  • Reconcile the synchronous path and the webhook path to the same end state. Both call the same apply_success / apply_failure, both guarded, so whichever arrives first wins and the second is a no-op. The invoice converges regardless of ordering.
def on_webhook(evt):
    if db.seen_event(evt.id):           # dedup on gateway event_id
        return
    with db.transaction():
        db.mark_seen(evt.id)
        inv = db.get_invoice_by_charge(evt.charge_id)
        if evt.type == 'payment_succeeded':
            # guarded: no-op if already PAID or if charge_id != current attempt
            db.transition_if(inv.id, from_='CHARGING', to='PAID',
                             only_if_charge=evt.charge_id)
        elif evt.type == 'payment_failed':
            db.transition_if(inv.id, from_='CHARGING', to='PAYMENT_FAILED',
                             only_if_charge=evt.charge_id)
            dunning.schedule_retry(inv.id)

5. Recovery sweeper. A periodic job scans invoices stuck in CHARGING past the expected webhook window and re-drives them - safely, because the stored idempotency key makes a re-send a no-op if the first landed, and the gateway’s charge-status API confirms the real outcome when webhooks were lost.

Why this is right: you cannot get exactly-once from an at-least-once scheduler and an eventually-consistent gateway on trust, so you build it. The (subscription, period) unique key collapses duplicate enqueues to one invoice; the per-attempt gateway idempotency key makes each charge happen once; the pre-written CHARGING state makes recovery deterministic; and dedup-plus-guarded webhook application makes duplicated, out-of-order outcomes converge to the correct final state. At-least-once plumbing becomes exactly-once billing.

2. The recurring scheduler and flattening the billing burst

The job: fire a billing event for each of 50M subscriptions once per period, durably (surviving crashes and deploys), without a midnight stampede and without missing or duplicating a period.

Naive approach: one cron job that scans all subscriptions at midnight

# runs at 00:00 daily
for sub in db.query("SELECT * FROM subscriptions WHERE active"):
    if sub.next_billing_at <= now():
        charge(sub)

Where it breaks:

  • The midnight stampede. If most subscriptions anchor to the 1st, 40% of 50M invoices try to bill in a few hours - ~1k charges/sec sustained against a gateway with rate limits, plus a matching webhook flood. The gateway throttles, retries pile up, latency spikes, and some invoices miss their window.
  • One giant scan is a single point of failure and not resumable. A full-table scan of 50M rows in one process; if it crashes at row 30M, did rows 0-30M charge? Re-running may double-charge (the naive loop has no idempotency). It also pins one node while the rest idle.
  • No durable “fired” record per period. The loop uses next_billing_at, but if it charges and crashes before advancing it, the next run charges again; if it advances before charging, a crash loses the charge.

A single midnight cron over all rows is both a throughput disaster and a correctness hazard.

Evolved approach: per-subscription anchor dates + jitter + sharded, resumable scans + idempotent enqueue

1. Anchor each subscription to its own signup day, and jitter within the day. A subscription created on the 14th bills on the 14th of each month, not the 1st. This alone spreads monthly volume across all 28-31 days, turning a single spike into ~19/sec steady. We further jitter the exact fire time within a window (e.g. a deterministic offset derived from hash(subscription_id) across a few hours) so even same-day subscriptions do not all fire at the same second.

2. Sharded, resumable scans instead of one giant loop. The scheduler is partitioned by a shard key (hash(subscription_id) % N). Each shard is scanned independently by a worker, so the work is parallel and bounded, and a crashed shard resumes from its checkpoint without touching others. The scan is a tight indexed query, not a full table scan:

-- index on (shard, next_billing_at) makes this a range scan, not a full scan
SELECT subscription_id, next_billing_at
FROM subscriptions
WHERE shard = :shard
  AND state = 'active'
  AND next_billing_at <= :now
ORDER BY next_billing_at
LIMIT :batch;   -- process in batches, checkpoint after each

3. The enqueue is idempotent, so over-firing is harmless. The scheduler does not charge; it enqueues bill(subscription_id, period_key). Because the invoice is keyed on (subscription_id, period_key) (deep dive 1), firing the same subscription twice for the same period converges on one invoice. This decouples “fire reliably” (at-least-once, easy) from “bill exactly once” (the pipeline’s job). The scheduler can be sloppy; the pipeline cannot.

4. Advance the period only on confirmed payment, never in the scheduler. next_billing_at moves forward only when the invoice reaches PAID (via the webhook handler), inside the same transaction that marks it paid. So a crash in the scheduler or runner never advances the period without a successful charge, and never charges without eventually advancing. The period pointer and the money are coupled in one ACID step.

5. A watchdog for missed windows. A separate low-frequency sweep looks for subscriptions whose next_billing_at is well in the past with no corresponding invoice for that period - catching any shard that silently fell behind - and enqueues them. Because enqueue is idempotent, the watchdog cannot cause a double bill.

Why this is right: anchoring + jitter converts a destructive burst back into the trivial average load the BoE predicted; sharded resumable scans remove the single-giant-loop SPOF and make the scan parallel and checkpointed; and the idempotent enqueue plus period-advances-only-on-payment means the scheduler can over-fire or crash freely while billing stays exactly-once. Reliability and throughput are separated cleanly.

3. Mid-cycle plan changes and proration

The job: when a customer upgrades, downgrades, or changes seat count partway through a billing period, charge them fairly - credit the unused time on the old plan and charge the used/remaining time on the new plan - with deterministic, reproducible arithmetic that reconciles to the cent.

Naive approach: on plan change, immediately charge the full new price

def change_plan(sub, new_plan):
    gateway.charge(sub.card_token, new_plan.price)   # charge full new price now
    sub.plan = new_plan

Where it breaks:

  • Double-paying for overlapping time. The customer already paid the old plan for the whole period on day 1. Charging the full new price on day 14 means they pay for two plans for the same days. They are furious and they are right.
  • No credit for the downgrade. A customer moving to a cheaper plan gets nothing back for the premium time they already paid for.
  • Non-deterministic, non-reconcilable math. Ad-hoc charging on change scatters money events with no coherent per-period total. Finance cannot tie invoice lines to the charged amount.
  • Rounding and timezone ambiguity. “Half the month” computed in floats and wall-clock days drifts by cents and by a day across DST, and two services compute different numbers.

Charging the full new price on change is both unfair and unauditable.

Evolved approach: proration as signed line items on the next invoice, computed on exact time, charged immediately or deferred

1. Model proration as credit + charge line items, not a special charge. A plan change generates two signed line items against the current period:

  • A credit for the unused fraction of the old plan: -old_price * (remaining_seconds / period_seconds).
  • A charge for the remaining fraction of the new plan: +new_price * (remaining_seconds / period_seconds).

Their sum is the net proration adjustment (positive for an upgrade, negative for a downgrade). These are just invoice line items, so they flow through the same invoice state machine and reconcile with everything else.

Example: $10/mo plan, upgrade to $30/mo on day 15 of a 30-day period.
  remaining fraction = 15/30 = 0.5
  credit (old):  -$10 * 0.5 = -$5.00
  charge (new):  +$30 * 0.5 = +$15.00
  net proration:            = +$10.00

2. Compute on exact time in integer minor units, with a fixed proration basis. Use second-level timestamps (UTC) for remaining_seconds / period_seconds, and do the multiply in integer minor units with a single, documented rounding rule (round half to even on the final cent), so the number is deterministic and reproducible. We store the exact period_start, period_end, and effective_at on the line so anyone can recompute it. Never floats, never wall-clock day counts.

3. Immediate vs deferred proration is a policy, not an accident. Two configurable behaviors, both correct:

ModeWhen chargedUse when
Immediate (invoice now)A new prorated invoice is created and charged right away on upgradeUpgrades, so the customer gets premium features and pays for them now
Deferred (next invoice)The signed line items accrue and roll into the next scheduled invoiceDowngrades and small changes, to avoid a flurry of tiny charges

A common policy: upgrades invoice immediately (customer wants access now), downgrades defer the credit to the next period (avoids refund mechanics and tiny charges). The point is that both are expressed as the same signed line items; only the when differs.

4. Seat/quantity changes are the same math on quantity. Adding 5 seats mid-cycle is a prorated charge for 5 * per_seat_price * remaining_fraction; removing seats is a prorated credit. Metered usage is billed in arrears on the next invoice, summed from a usage ledger. All of it lands as line items that sum to the invoice total.

5. Idempotent plan changes. A plan-change request carries an idempotency key too, so a double-submit (double-click, retry) does not create two proration adjustments. The change writes the new plan and the proration lines in one transaction, guarded so a replay is a no-op.

Why this is right: expressing proration as signed credit/charge line items on an invoice keeps one coherent per-period total that reconciles to the cent, instead of scattered ad-hoc charges; computing on exact UTC seconds in integer minor units with a fixed rounding rule makes the arithmetic deterministic and reproducible by finance; and separating the amount (always prorated line items) from the timing (immediate vs deferred policy) lets product choose behavior without ever breaking the accounting.

4. Failed-payment retries with exponential backoff (dunning)

The job: when a charge is declined (expired card, insufficient funds, temporary bank decline), retry on an escalating schedule that maximizes eventual recovery without hammering the gateway or the customer, then gracefully degrade the subscription if it never succeeds.

Naive approach: retry immediately in a tight loop

while not charged:
    result = gateway.charge(sub.card_token, amount)
    if result.success:
        break
    # try again right away

Where it breaks:

  • Immediate retries almost never recover and waste everything. “Insufficient funds” at 2pm is still insufficient at 2:00:01pm. A tight loop burns gateway rate limit, triggers fraud heuristics, and recovers nothing. Recovery happens on a timescale of days (next payday, customer updates card), not milliseconds.
  • No cap => infinite retries. A permanently dead card is retried forever, consuming resources and, worse, repeatedly hitting the gateway which may flag the merchant.
  • No customer communication. The best recovery lever - emailing the customer to update their card - is never pulled.
  • Not durable. An in-memory retry loop dies with the worker; the retry schedule is lost.

A tight retry loop maximizes cost and minimizes recovery, the exact opposite of what dunning needs.

Evolved approach: durable, scheduled exponential-backoff retries + smart decline handling + a terminal dunning ladder

1. Backoff on a days-scale schedule, persisted as scheduled jobs. On a failure, the Dunning Service schedules the next attempt at an increasing interval - a standard ladder like day 1, day 3, day 5, day 7 (geometric-ish), capped at a max number of attempts (e.g. 4). Each scheduled retry is a durable row with a next_attempt_at, picked up by the same scheduler machinery, so it survives crashes and deploys. Jitter is added so retries do not stampede the gateway at the same instant.

attempt 1: at invoice due time           (initial charge)
attempt 2: due + 1 day
attempt 3: due + 3 days
attempt 4: due + 5 days
attempt 5: due + 7 days  -> if still failing: subscription past_due -> canceled

2. Distinguish retryable (soft) from permanent (hard) declines. The gateway returns a decline code. A soft decline (insufficient funds, temporary bank issue, issuer timeout) is worth retrying on the backoff ladder. A hard decline (stolen card, closed account, “do not honor”) will never recover; we stop retrying immediately, mark the payment method invalid, and move straight to customer action. Retrying hard declines is pure waste and a fraud-flag risk.

Decline typeExamplesAction
Soft / retryableinsufficient_funds, issuer_timeout, try_again_laterschedule next backoff retry
Hard / permanentstolen_card, closed_account, do_not_honor, invalid_cardstop retries, flag payment method, require customer update

3. Each retry is a new charge attempt with a new idempotency key. Deliberately - it is a new attempt on a (hopefully) now-funded card - so attempt_no increments and key = invoice_id:attempt:N is fresh. This keeps retries exactly-once per attempt while letting legitimate re-charges happen.

4. Customer communication between attempts. Each failure emits an event that triggers a dunning email/SMS (“your payment failed, update your card”), and each successful recovery emits a confirmation. A payment-method update by the customer can trigger an immediate retry out of the backoff schedule - the highest-recovery action. The notification cadence is part of the ladder, not an afterthought.

5. Terminal state and grace. After the final retry fails, the subscription moves to past_due (service may continue during a grace period) and then canceled (access revoked). The invoice stays PAYMENT_FAILED / uncollectible for finance, never silently deleted. All transitions are recorded, so the dunning history is auditable - how many attempts, when, which declines.

Why this is right: declines recover on a days-scale, so backoff spread over days is what actually collects money while an immediate loop collects nothing and burns rate limit; splitting soft from hard declines stops wasting attempts on dead cards and protects the merchant’s gateway standing; persisting each retry as a durable scheduled job makes the ladder survive crashes; and the new-idempotency-key-per-attempt keeps every retry exactly-once while allowing the legitimate re-charge. Dunning becomes a controlled, auditable, high-recovery ladder instead of a destructive loop.

API Design & Data Schema

API

All mutating calls require an Idempotency-Key.

POST /api/v1/subscriptions
Headers: Authorization: Bearer <token>
         Idempotency-Key: <client-generated UUID, reused on retries>
Body:
{
  "customer_id":     "cus_123",
  "plan_id":         "plan_pro_monthly",
  "payment_method":  "pm_tok_456",
  "quantity":        3,                 // seats
  "trial_days":      14
}
Response 201:
{
  "subscription_id": "sub_9f3a...",
  "state":           "trialing",
  "plan_id":         "plan_pro_monthly",
  "current_period_end": "2026-12-03T00:00:00Z",
  "next_billing_at":    "2026-12-03T00:00:00Z"
}
POST /api/v1/subscriptions/{id}/change-plan   (Idempotency-Key required)
Body: { "new_plan_id": "plan_enterprise", "quantity": 5,
        "proration": "immediate" | "deferred" }
Response 200: { "subscription_id", "new_plan_id",
                "proration_line_items": [ {desc, amount}, ... ],
                "proration_total": 1000 }   // minor units

POST /api/v1/subscriptions/{id}/cancel         -> at_period_end | immediate
GET  /api/v1/subscriptions/{id}                -> state, plan, period, next_billing_at
GET  /api/v1/customers/{id}/invoices           -> paginated invoice history (read model)
GET  /api/v1/invoices/{id}                     -> line items, state, amounts, charge refs
GET  /api/v1/invoices/{id}/events              -> immutable audit timeline
POST /api/v1/invoices/{id}/retry               -> manual out-of-band retry (admin)
POST /api/v1/webhooks/{gateway}                -> gateway callbacks (signed)

Subscription creation returns 201; a charge outcome is not part of this response - the first invoice is billed asynchronously (or after the trial), and its final state arrives via webhook. Invoice-affecting responses are deliberately eventual.

Errors:

400  invalid payload / unknown plan
402  payment method declined (surfaced async for recurring; sync only for immediate upgrade)
401  bad token
409  idempotency key still in progress (retry shortly)
422  plan change not allowed for current state

Data stores

Different access patterns, different stores. Be explicit.

1. Subscriptions, plans, invoices, charge attempts, ledger - SQL, ACID, synchronously replicated. This is where strong consistency, the (subscription, period) uniqueness, monotonic invoice state, the per-attempt idempotency key, and the period-advance-on-payment all live. A relational database (PostgreSQL, or distributed SQL like CockroachDB/Spanner for horizontal scale and HA) is the correct, non-negotiable choice. At ~20 writes/sec steady and ~1k/sec in a controlled burst it fits comfortably with sharding by customer_id. NoSQL eventual consistency would reintroduce the double-charge and lost-update races we designed out.

Table: plans
  plan_id       STRING  PRIMARY KEY
  price         BIGINT  -- minor units
  currency      CHAR(3)
  interval      STRING  -- month | year
  metered       BOOL
  trial_days    INT

Table: subscriptions
  subscription_id   UUID    PRIMARY KEY
  customer_id       BIGINT  INDEX
  plan_id           STRING
  quantity          INT     -- seats
  state             STRING  -- trialing|active|past_due|canceled
  payment_method    STRING  -- gateway token (NOT card data)
  current_period_start TIMESTAMP
  current_period_end   TIMESTAMP
  next_billing_at   TIMESTAMP INDEX     -- scheduler scans this
  shard             INT     INDEX       -- hash(subscription_id) % N
  version           BIGINT              -- optimistic concurrency

Table: invoices                         -- one per (subscription, period)
  invoice_id        UUID    PRIMARY KEY
  subscription_id   UUID    INDEX
  customer_id       BIGINT  INDEX
  period_key        STRING  -- e.g. '2026-11'
  state             STRING  -- DRAFT|OPEN|CHARGING|PAID|PAYMENT_FAILED|
                            --   UNCOLLECTIBLE|VOID
  amount            BIGINT  -- minor units, = SUM(line items)
  currency          CHAR(3)
  attempt_no        INT     -- current charge attempt
  idempotency_key   STRING  NULL  -- key for the in-flight attempt
  charge_id         STRING  NULL  -- gateway charge id of current attempt
  version           BIGINT
  created_at        TIMESTAMP
  UNIQUE(subscription_id, period_key)   -- collapses duplicate enqueues

Table: invoice_line_items               -- APPEND ONLY
  line_id       UUID    PRIMARY KEY
  invoice_id    UUID    INDEX
  kind          STRING  -- base | proration_credit | proration_charge |
                        --   usage | tax | discount
  description   STRING
  amount        BIGINT  -- signed minor units
  period_start  TIMESTAMP   -- for reproducible proration
  period_end    TIMESTAMP
  effective_at  TIMESTAMP

Table: charge_attempts                  -- APPEND ONLY, immutable money event
  attempt_id      UUID   PRIMARY KEY
  invoice_id      UUID   INDEX
  attempt_no      INT
  idempotency_key STRING UNIQUE          -- exactly-once toward gateway
  gateway_charge_id STRING NULL
  status          STRING -- pending|succeeded|failed
  decline_code    STRING NULL            -- soft vs hard
  amount          BIGINT
  created_at      TIMESTAMP
  UNIQUE(invoice_id, attempt_no)

Table: ledger_entries                   -- APPEND ONLY, double-entry
  entry_id     UUID    PRIMARY KEY
  invoice_id   UUID    INDEX
  account      STRING  -- revenue | receivable | cash | refund | credit
  direction    CHAR(1) -- D | C
  amount       BIGINT
  currency     CHAR(3)
  created_at   TIMESTAMP

Table: webhook_events                   -- dedup + audit
  event_id     STRING  PRIMARY KEY       -- gateway's id; dedup key
  gateway      STRING
  type         STRING
  charge_id    STRING  INDEX
  payload      JSON
  processed_at TIMESTAMP

Table: dunning_schedule
  dunning_id      UUID   PRIMARY KEY
  invoice_id      UUID   INDEX
  attempt_no      INT
  next_attempt_at TIMESTAMP INDEX         -- scheduler scans this
  state           STRING -- scheduled|done|exhausted

Table: idempotency_keys                 -- API-level dedup
  idem_key     STRING  PRIMARY KEY
  resource_id  UUID    NULL
  status       STRING  -- IN_PROGRESS | DONE
  response     JSON
  created_at   TIMESTAMP   -- 30 days hot

Table: outbox
  outbox_id    BIGINT  PRIMARY KEY
  topic        STRING
  payload      JSON
  sent         BOOL    INDEX (partial: WHERE sent=false)

2. Invoice/history read model - NoSQL wide-column (Cassandra/DynamoDB), by customer. Customer-facing invoice lists are read by customer_id in reverse-chronological pages, never joined, and tolerate eventual consistency. Built by a projector off the event bus so history reads never touch the transactional primary.

3. Idempotency and dedup hot store - Redis + the SQL unique constraint as the real guarantee. Redis fast-paths the common “already processed” check; the DB unique constraint on idempotency_keys and on charge_attempts(idempotency_key) is the authoritative backstop that survives a cache miss or flush.

4. Audit cold storage - WORM object storage. Invoices, line items, charge attempts, and events are financial records; they are continuously exported to write-once-read-many storage with a compliance lock for the 7-year retention window, never mutated or purged early.

Why SQL for the core, NoSQL for the projections: the money path needs multi-row ACID (invoice + line items + ledger + period advance in one transaction), uniqueness constraints for exactly-once ((subscription, period) and per-attempt keys), and monotonic state transitions - exactly what relational databases provide, and at this volume sharded by customer_id it scales fine. History and invoice lists are key-scoped, read-heavy, and eventual-consistency-tolerant, so they go to a wide-column store. Match the guarantee to the data: absolute for money, relaxed for its read projections.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix. Note that at ~20 writes/sec steady, most “bottlenecks” are correctness and burst-shaping failures, not throughput ceilings.

1. Double charges (breaks first, worst impact). A scheduler double-fire, a worker crash mid-charge, or an ambiguous timeout charges the customer twice. Fix: invoice keyed on (subscription_id, period_key) with a unique constraint so duplicate enqueues converge on one invoice; a deterministic per-attempt idempotency key passed to the gateway on every retry so each charge happens once; the pre-written CHARGING state so recovery is deterministic; webhook dedup on event_id. This is the single most important mechanism in the system.

2. Out-of-order / duplicated webhooks corrupting invoice state. A late payment_failed overwrites a PAID invoice; a redelivered succeeded double-advances the period. Fix: dedup on gateway event_id; apply outcomes with optimistic state guards and a terminal PAID that refuses failure transitions; act only on the webhook whose charge_id matches the invoice’s current attempt. Both the synchronous response and the webhook route to the same guarded handler, so they converge.

3. The billing stampede. A shared anchor date pushes 40% of 50M invoices into a few hours (~1k/sec), throttling the gateway and the DB. Fix: per-subscription anchor dates (bill on the signup day) plus intra-day jitter, flattening the burst toward the ~19/sec average; sharded, resumable scans so the scheduling work is parallel and checkpointed; rate-limit the charge pipeline to stay within the gateway’s limits, buffering excess in the queue.

4. The dual-write problem (DB commit vs event publish). Invoice created but the “charge it” signal lost, or paid in the DB but the receipt/period-advance event lost. Fix: the outbox pattern - domain events written in the same ACID transaction as the state change; a relay publishes at-least-once; a recovery sweeper re-drives invoices stuck in CHARGING or subscriptions past their billing date with no invoice.

5. Gateway outage or rate limiting. The gateway is down or throttling; charges cannot complete. Fix: queue and retry (invoices persist in OPEN/CHARGING), typed retries with exponential backoff and jitter, a circuit breaker, and a DLQ for attempts that exhaust retries, worked by ops. Billing degrades to “will charge shortly,” never to “lost” or “doubled.”

6. Hot customers / large accounts. An enterprise customer with 100k seats and heavy metered usage makes one invoice expensive and one customer_id shard hot. Fix: shard by customer_id so load spreads across customers; for a single huge account, batch usage aggregation asynchronously into a usage ledger and compute the invoice from the pre-aggregated total, not by scanning raw events at bill time; cap line-item cardinality by rolling up usage.

7. Proration and money correctness bugs. Float math or wall-clock day counts drift by cents and across DST; two services disagree. Fix: integer minor units, exact UTC second-level time fractions, a single documented rounding rule, and storing period_start/end/effective_at on each line so proration is reproducible. A per-invoice invariant check (SUM(line_items) == amount) and a global double-entry check (SUM(debits) == SUM(credits)) catch drift.

8. Retry storms from dunning. Thousands of failed invoices all retry at the same backoff tick, re-stampeding the gateway. Fix: jittered next_attempt_at, soft-vs-hard decline classification so dead cards stop immediately, and the same rate-limited pipeline. Dunning retries flow through the identical throttle as fresh charges.

9. Single points of failure / durability. Any node loss must lose no invoice or payment outcome. Fix: SQL with synchronous replication and automatic failover; stateless API, scheduler shards, and runners horizontally scaled; the outbox, DLQ, dedup store, and sweeper guarantee nothing in flight vanishes silently; webhook ingestion is HA because a dropped webhook can lose a payment outcome.

Shard keys / partitioning, stated plainly: shard subscriptions, invoices, charge_attempts, and ledger_entries by customer_id so all of a customer’s billing data co-locates (single-partition reads and transactions) while writes spread across customers. The scheduler partitions by hash(subscription_id) % N for parallel, resumable scans. The read model shards by customer_id for single-partition reverse-chronological invoice lists. Never shard by billing date - it concentrates every month-boundary write on one partition, recreating the stampede at the storage layer. Queue topics partition by invoice_id for per-invoice ordering.

Wrap-Up

The trade-offs that define this design:

  • Cheap, over-firing scheduler + idempotent charge pipeline. We do not try to make the scheduler fire exactly once (impossible across crashes and multiple nodes). We make it at-least-once and cheap, and make every downstream step safe to run twice - the invoice keyed on (subscription, period), the per-attempt gateway idempotency key. Reliability (fire) and correctness (bill once) are separated.
  • The webhook is the authority, not the synchronous response. The gateway is eventually consistent, so the real outcome arrives asynchronously, late, duplicated, and out of order. We dedup on event_id, apply with optimistic state guards and a terminal PAID, and route the synchronous and webhook paths to the same handler so they converge no matter which lands first.
  • Flatten the burst at the source. Per-subscription anchor dates plus jitter turn a destructive ~1k/sec month-boundary stampede back into the ~19/sec average, and sharded resumable scans remove the single-giant-loop SPOF. We never shard by date.
  • Proration as signed line items, computed on exact time in integer units. Credits and charges for plan changes are just invoice lines that sum to the period total and reconcile to the cent, computed on UTC seconds with a fixed rounding rule, so finance can reproduce every number. Amount (always prorated) is separated from timing (immediate vs deferred policy).
  • Dunning is a days-scale, auditable ladder, not a loop. Declines recover over days, so retries back off (day 1, 3, 5, 7) with jitter, split soft from hard declines to stop wasting attempts on dead cards, use a fresh idempotency key per attempt, and drive customer communication - then degrade the subscription gracefully through past_due to canceled.

One-line summary: a subscription billing system anchors each subscription to its own jittered billing date, has a cheap over-firing sharded scheduler enqueue work, and bills through an idempotent charge pipeline - one invoice per (subscription, period), a deterministic per-attempt gateway idempotency key, a pre-written CHARGING state, and dedup-plus-guarded webhook handling - so each customer is charged exactly once per period for exactly the prorated amount, failures retry on a days-scale backoff ladder, and every money event is recorded immutably even when the gateway’s webhooks arrive duplicated and out of order.