“Design insurance claims” sounds like a form with a Submit button. A customer uploads a photo of their dented car, someone looks at it, money gets paid. The interviewer lets that stand until they ask the question that breaks it: this is not a request-response system, it is a long-running, human-in-the-loop workflow that must be fully auditable years later. A single claim lives for days or weeks. It is touched by a customer, an ML model, a fraud engine, one or more human adjusters, maybe a surveyor and a hospital, and finally a payment system. Any of them can send it backward - ask for more documents, flag fraud, dispute the estimate. And every single one of those touches is a regulated record: when a claim is denied, the regulator (and possibly a court) can demand the exact inputs, model version, and human decision that produced that outcome, reproducible on demand.

That is the spine of this system: orchestrating a multi-week, multi-actor decision workflow over large unstructured documents, where an ML layer pre-screens, humans decide the hard cases, and every state transition and input is preserved immutably for audit. Everything hard hangs off it - how you ingest and store gigabytes of photos and PDFs reliably, how you run ML pre-screening without blocking the claim, how you route the right claim to the right adjuster without one sitting idle while another drowns, how you model a claim that moves through fifteen states and can always go backward, and how you build an audit trail a regulator will accept. The scale (100K claims/day) is modest in raw QPS terms; the difficulty is the workflow, the unstructured data, and the audit guarantee, not the request rate. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Claim intake (FNOL - First Notice of Loss). A customer files a claim against a policy: incident type (motor accident, health, property), description, date, location, and a set of attachments - photos of damage, PDFs of bills, police reports, prescriptions.
  • Document and photo ingestion. Attachments (often tens of MB each, dozens per claim) are uploaded reliably, virus-scanned, stored durably, and made available to downstream ML and humans.
  • ML pre-screening. Before any human touches it, a pipeline runs: OCR/extraction on documents, damage/severity assessment on photos, fraud scoring, and a completeness check (are all required documents present?). It produces a recommendation - auto-approve (simple, low-value, clean), auto-deny (policy clearly excludes), or route-to-human (everything else).
  • Adjuster routing and workqueue. Claims that need a human are routed to an adjuster with the right skill (motor vs health), license region, authority limit (an adjuster can approve up to X rupees), and current load. The adjuster reviews evidence, requests more documents if needed, and approves or denies with a reason.
  • Claim lifecycle / state machine. A claim moves through well-defined states (submitted -> screening -> pending-docs -> under-review -> approved/denied -> paid/closed) and can move backward (request more info, escalate, reopen on appeal).
  • Payout handoff. An approved claim hands an approved amount to a payment/disbursement system and tracks settlement.
  • Regulatory audit trail. Every state change, every input (which documents, which model version, which score), every human decision and its reason, is recorded immutably with who/when/why, reproducible for years.
  • Customer status and notifications. The customer sees their claim status and is notified on every meaningful transition (“documents received”, “more info needed”, “claim approved”).

Explicitly out of scope (say this out loud so you own the scope):

  • Policy administration and underwriting. Whether the policy exists, what it covers, premium status - that is a separate Policy system. The claims system calls it to validate coverage; it does not own policies.
  • The actual money movement / ledger. Disbursing funds, bank rails, reconciliation, and the financial ledger are a Payments system. We hand it an approved amount and a payee and track the result; we do not design the ledger.
  • Training the ML models. We consume model inference as a versioned service. Data labeling, training pipelines, and model dev are a separate ML platform. We own serving, versioning, and recording which version scored each claim, because that is an audit requirement.
  • Full fraud investigation / SIU. Deep special-investigations workflows are their own system; we flag, score, and hand suspected fraud off.

The one decision that drives everything: this is a workflow-and-evidence problem, not a high-QPS problem. 100K claims/day is ~1-2 writes/sec of new claims - trivial. What is hard is that each claim is a weeks-long saga over heavy unstructured data, with an ML layer in the middle and a hard, non-negotiable requirement that the entire decision be reproducible and immutable for regulators. So the workflow engine, the document pipeline, and the audit log, not the database throughput, are the heart of the design.

Non-Functional Requirements (NFR)

  • Scale: ~100K claims/day. Average ~15 attachments per claim, averaging ~3 MB each (photos, scanned PDFs). ~1M active (open) claims at any time - claims live for days to weeks, so the in-flight set is large even though intake is slow. ~10K adjusters across skills and regions.
  • Latency: intake (FNOL submit) must feel instant, p99 under ~1s to accept the claim and return a claim ID - but the processing is deliberately asynchronous and can take seconds (ML) to days (human). Document upload must be reliable on flaky mobile networks (resumable). The adjuster workqueue and claim detail views must load under ~500ms - adjusters work these all day.
  • Availability: intake and the adjuster console need 99.95%+ - a customer who cannot file after an accident, or 10K adjusters idled, is a serious failure. Async ML processing can tolerate brief backlog (a claim screened 10 minutes late is fine); intake and the audit write cannot be lost.
  • Consistency: the claim state machine must be strongly consistent - a claim is in exactly one state, transitions are atomic and guarded, and two adjusters must never both “own” and decide the same claim. The audit log must be write-once and never lost - a transition that is not durably audited did not happen. Document metadata and ML scores are attached to the claim transactionally. Workqueue counts and dashboards can be eventually consistent.
  • Durability: this is the strictest NFR. Claim records, every document, and the full audit trail are legal records retained for 7-10 years (varies by jurisdiction and line of business). Documents must never be lost or silently altered; the audit log must be tamper-evident. Nothing here is “ephemeral.”
  • Reproducibility: given a closed claim, we must reconstruct exactly what the system saw and did - which document versions, which model version and its output, which adjuster decided and why. This is not a nice-to-have; it is the core regulatory requirement.
  • Security / privacy: claims carry PII and sensitive data (medical records, financials). Encryption at rest and in transit, access control on documents, and audit of who viewed what are mandatory.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they tell you where the difficulty actually is.

Claim intake (the transactional write path):

100,000 claims/day
= 100,000 / 86,400
≈ 1.2 claims/sec average
Peak (business hours, ~70% of claims in ~8 hours):
  0.7 * 100,000 / (8 * 3600) ≈ 2.4 claims/sec
Momentary spikes (a storm, a regional accident surge) ≈ 20-50 claims/sec

So new-claim writes peak in the low tens per second. This is nothing - a single relational node handles it. Intake QPS is not a scaling driver. Lock this in early; it tells the interviewer you know where the real work is.

Document ingestion (the real data firehose):

100,000 claims/day * 15 attachments * 3 MB
= 100,000 * 45 MB
= 4.5 TB/day of uploads
= 4.5 TB / 86,400 s ≈ 52 MB/sec average ingest
Peak (~3x) ≈ 150 MB/sec
Upload request rate: 100,000 * 15 / 86,400 ≈ 17 uploads/sec avg, ~50/sec peak

So the data plane moves ~4.5 TB/day, not the control plane’s trivial QPS. This is the number that dictates object storage + CDN upload, not a database holding blobs.

ML pre-screening (the compute path):

Each claim runs: OCR on ~10 document pages, damage CV on ~5 photos,
                 a fraud model, a completeness check.
~1.2 claims/sec avg, ~2.4/sec peak -> each spawning ~16 inference tasks
= ~20 inference tasks/sec avg, ~40/sec peak.
OCR/CV are the heavy ones (hundreds of ms to seconds each on GPU).

Modest task rate but each task is heavy and bursty. This is a queue + autoscaled worker pool shape, not a synchronous call - a claim with 20 photos should not block anything.

Storage growth (the durability driver):

Documents: 4.5 TB/day * 365 ≈ 1.64 PB/year.
Retain 7-10 years -> ~12-16 PB total at steady state.

Petabytes, and it must live for a decade. That is object storage with lifecycle tiering (hot -> cold -> archive), never a database.

Claim + audit records:
  Per claim ~20 KB (claim fields, ~30 audit events, ML score snapshots, refs).
  100,000 * 20 KB = 2 GB/day ≈ 730 GB/year -> ~5-7 TB over retention.

Tiny and relational. The metadata is small and belongs in SQL; the documents are petabytes and belong in object storage. That split is the whole storage story.

Adjuster workload:

Say ~30% of claims route to a human = 30,000/day.
10,000 adjusters -> 3 claims/adjuster/day of new routing (plus re-reviews).
Workqueue reads: 10K adjusters refreshing queues ~ every 30s during an 8h day
  ≈ 10,000 / 30 ≈ 330 queue reads/sec -> cacheable, modest.

Comfortable. The routing problem is about correctness and fairness (right skill, right authority, balanced load), not read volume.

Bandwidth: ingest ~150 MB/sec peak (uploads, absorbed by object-store direct upload + CDN). Adjuster document viewing is served from object storage/CDN, point-to-point, tens of MB/sec. Nothing here stresses a well-built data plane.

The takeaways: claim/control-plane QPS is tiny (tens/sec) and belongs in a strongly-consistent relational store; the document plane moves petabytes over years and belongs in object storage with direct upload; ML is a heavy, bursty, asynchronous compute pool fed by queues; and the genuine difficulty is the long-running workflow, the immutable audit trail, and matching the store to the data shape - not absorbing raw scale. The workflow engine and the audit log, not the database throughput, are the design drivers.

High-Level Design (HLD)

The architecture separates four planes: the ingestion plane (intake API + direct-to-object-store document upload), the processing plane (an async ML pre-screening pipeline over queues), the workflow plane (a strongly-consistent claim state machine + adjuster routing), and the audit plane (an immutable, append-only event log underpinning everything). A claim is created in the workflow plane, its documents land in the ingestion plane, the processing plane scores it asynchronously, humans act via the workflow plane, and every step writes to the audit plane.

   ┌──────────────┐        ┌──────────────┐        ┌────────────────┐
   │ Customer App  │        │Adjuster Conso-│        │ Partner/Hospital│
   │ (file claim,  │        │ le (review,   │        │ (bills, reports,│
   │  upload docs) │        │  decide)      │        │  surveys)       │
   └──────┬────────┘        └──────┬────────┘        └────────┬───────┘
          │ submit + presigned     │ queue/review             │ upload
   ┌──────▼────────────────────────▼──────────────────────────▼───────┐
   │                       API Gateway  /  LB  (authN/Z)               │
   └───┬───────────────┬───────────────────┬───────────────────┬──────┘
       │ FNOL          │ presigned upload   │ adjuster actions  │
 ┌─────▼──────┐  ┌──────▼─────────┐   ┌──────▼────────┐   ┌──────▼──────┐
 │ Intake Svc  │  │ Document Svc   │   │ Workflow Svc  │   │ Routing Svc  │
 │ (validate   │  │ (presign, meta,│   │ (STATE MACHINE│   │ (assign to   │
 │  policy,    │  │  scan trigger) │   │  + orchestr.) │   │  adjuster)   │
 │  create)    │  └───┬────────┬───┘   └───┬───┬───┬───┘   └──────┬──────┘
 └─────┬──────┘       │ direct │           │   │   │              │
       │              │ upload │     emit  │   │   │ transition   │ workqueue
       │        ┌─────▼────┐   │   event   │   │   │              │
       │        │  Object  │   │   ┌───────▼─┐ │ ┌─▼──────────┐ ┌─▼────────┐
       │        │  Store   │◀──┘   │ Kafka   │ │ │ Claim Store │ │ Workqueue│
       │        │ (S3, 16PB│       │ (claim  │ │ │ (sharded SQL│ │ (per adj,│
       │        │  tiered) │       │  events)│ │ │  by claim)  │ │  indexed)│
       │        └────┬─────┘       └────┬────┘ │ └─────────────┘ └──────────┘
       │             │ scan/OCR         │consume
 ┌─────▼─────────────▼──────┐    ┌──────▼───────────────────┐
 │  Virus Scan + ML Workers  │    │  Audit Log Service        │
 │ (OCR, damage CV, fraud,   │    │ (append-only, WORM,       │
 │  completeness) autoscaled │    │  hash-chained, immutable) │
 │  GPU pool, queue-fed      │    │  -> cold archive          │
 └───────────┬──────────────┘    └──────────────────────────┘
             │ writes scores + recommendation back as events
             └──────────────────────────────────────────────▶ Workflow Svc

Intake / document flow:

  1. Customer files an FNOL. The Intake Service validates the policy against the Policy system (coverage active, incident type covered), creates a claim row in state SUBMITTED, and returns a claim_id. This is the only synchronous, fast path.
  2. For each attachment, the client asks the Document Service for a presigned URL and uploads the file directly to object storage - bytes never flow through our app servers. The Document Service records document metadata (type, size, checksum, claim ref) and triggers a virus scan.
  3. Each uploaded, scanned document emits an event. When required documents are present, the claim transitions SUBMITTED -> SCREENING.

ML pre-screening flow (asynchronous):

  1. On SCREENING, the Workflow Service enqueues pre-screening tasks on Kafka. ML Workers (autoscaled GPU pool) consume them: OCR/extraction on documents, damage/severity CV on photos, a fraud score, and a completeness check.
  2. Each model writes its output - and critically, its version and inputs - back as events. An aggregation step combines them into a recommendation: AUTO_APPROVE, AUTO_DENY, or ROUTE_TO_HUMAN (with a reason and the fraud/severity scores).
  3. Clean, low-value, low-risk claims may auto-approve (-> APPROVED) and hand off to payout. Clearly excluded claims auto-deny. Everything else transitions to PENDING_REVIEW and goes to routing.

Adjuster / human flow:

  1. The Routing Service picks an adjuster matching skill, region/license, and authority limit, and places the claim in that adjuster’s workqueue (and atomically claims ownership so no one else gets it). State -> UNDER_REVIEW when the adjuster opens it.
  2. The adjuster reviews documents (served from object storage), the ML scores, and the policy. They can request more documents (-> PENDING_DOCS, back to the customer), escalate (to a higher authority adjuster), or decide - approve (with amount, up to their authority limit) or deny (with a coded reason).
  3. Approved -> payout handoff -> PAID -> CLOSED. Denied -> DENIED (appealable, which can reopen).

Audit flow (cross-cutting): every transition and input - who, what, when, why, which document versions, which model version and score - is appended to the immutable audit log (write-once, hash-chained, archived). The claim’s current status is a materialized convenience; the audit log is the source of truth and the thing a regulator reads.

The key insight: the claim is a strongly-consistent state machine orchestrated by a workflow engine, the heavy data lives in object storage reached by presigned upload, ML runs asynchronously off queues, and an append-only audit log makes every decision reproducible. Scale is easy here; correctness, asynchrony, and auditability are the design.

Component Deep Dive

The hard parts, naive-first then evolved: (1) document/photo ingestion and storage, (2) the asynchronous ML pre-screening pipeline, (3) adjuster routing and the claim state machine, (4) the immutable regulatory audit trail.

1. Document and photo ingestion

Naive approach: upload files through the app server into the database. The client POSTs each photo/PDF to the Intake Service, which writes the bytes into a documents table (a BLOB column) alongside the claim.

Where it breaks:

  • App servers become a data pipe. 150 MB/sec peak of uploads flowing through your application tier means every intake node is saturated moving bytes it should never touch. Your control-plane servers fall over doing data-plane work.
  • Databases are terrible blob stores. Petabytes of 3 MB blobs in a relational DB bloats it, wrecks backups, blows the buffer cache, and makes every query slower. A 16 PB database of photos is an operational nightmare; object storage exists precisely for this.
  • Flaky mobile uploads. A customer on 3G at an accident site uploading a 30 MB photo set will have connections drop mid-upload. A single POST through the app server has no resumability - a dropped byte means re-uploading the whole file, repeatedly, and timing out the request.
  • No scan before use. Taking an untrusted file and handing it straight to ML/humans is a malware vector.

First evolution: object storage, but still proxied. Put blobs in S3-style object storage (good), but have the app server receive the upload and forward it to object storage. This fixes the database problem but the app tier is still a saturated data pipe, and uploads still are not resumable.

The answer: presigned direct-to-object-store upload, with a metadata record, resumable multipart, and an async scan gate.

  • Presigned URLs, bytes bypass the app. The client asks the Document Service for a short-lived presigned upload URL per file. The client uploads the bytes directly to object storage; they never pass through our servers. Our app tier handles only tiny metadata requests (tens/sec), while object storage absorbs the 150 MB/sec. The data plane and control plane are fully separated.
client -> Document Svc: POST /claims/{id}/documents  {filename, type, size, sha256}
Document Svc -> client: { document_id, upload_url (presigned, 15min), parts? }
client -> Object Store: PUT bytes directly (multipart if large)
client -> Document Svc: POST .../documents/{document_id}/complete
Document Svc: verify checksum, mark UPLOADED, trigger scan, emit event
  • Resumable multipart for flaky networks. Large files upload as parts (e.g. 5 MB chunks); a dropped connection resumes from the last acknowledged part instead of restarting. This is the difference between “works at an accident site on 3G” and “customer gives up.”
  • Checksum + idempotency. The client sends a sha256; the Document Service verifies the stored object matches, so a corrupted upload is caught. The document_id is assigned up front, so a retried complete is idempotent - no duplicate documents.
  • Scan gate before processing. On complete, the document is virus/malware-scanned (async) and only then marked AVAILABLE and exposed to ML and adjusters. An infected file is quarantined and the customer is asked to re-upload. Documents are immutable once stored - a correction is a new version, never an overwrite, because the original is a legal record (see the audit section).
  • Completeness drives the state machine. The Document Service knows which document types a given incident type requires (motor: photos + police report + estimate; health: bills + prescription + discharge summary). When the required set is AVAILABLE, it emits “documents complete,” which lets the claim move SUBMITTED -> SCREENING. Missing a required type keeps the claim in PENDING_DOCS.
  • Storage tiering for a decade of retention. Hot documents (active claims) sit in standard object storage; once a claim closes, lifecycle rules move its documents to infrequent-access, then to cold archive (Glacier-class) for the 7-10 year retention tail. Encryption at rest with per-claim access control throughout. This keeps 16 PB affordable - most of it is cold.

The elegant part: the app tier never touches a byte of claim data. Presigned direct upload separates the trivial control plane from the petabyte data plane; resumable multipart makes real-world mobile uploads reliable; immutability + versioning + tiering satisfy the legal-retention NFR without bloating anything hot.

2. The asynchronous ML pre-screening pipeline

Naive approach: run all the models synchronously at intake. When the claim is submitted, call OCR, then the damage CV model, then fraud scoring, inline, and return the recommendation in the submit response.

Where it breaks:

  • Submit blocks on seconds of GPU work. A claim with 20 photos and 10 document pages is dozens of heavy inferences - several seconds to tens of seconds. Holding the FNOL request open that long destroys the “instant submit” NFR and ties up app threads.
  • Documents arrive over time. The customer uploads photos over several minutes on a flaky connection; a hospital sends the bill an hour later. There is no single moment when “all inputs” exist to score synchronously. Processing is inherently triggered by events, not a one-shot call.
  • Bursty, heavy, failure-prone. A storm spikes claims 30x; a GPU inference can OOM or time out. A synchronous chain has no buffering and no retry - one slow model stalls the whole intake path.
  • No partial progress / reproducibility. If fraud scoring fails after OCR succeeded, a synchronous call loses the OCR work and has nothing recorded about what version ran.

The fix: an event-driven pipeline of queued, idempotent inference tasks over an autoscaled worker pool, each recording its version and output, aggregated into a recommendation.

  • Queue-fed workers, decoupled from intake. When the claim enters SCREENING, the Workflow Service publishes pre-screening tasks to Kafka topics (one per model type: ocr, damage-cv, fraud, completeness). ML Workers - an autoscaled GPU pool - consume and process. Intake returns instantly; the burst is absorbed by queue depth, and workers scale with backlog.
on SCREENING:
  for doc in claim.documents(type=document): enqueue(ocr,    claim, doc, model_ver)
  for img in claim.documents(type=photo):    enqueue(damage, claim, img, model_ver)
  enqueue(fraud, claim, features, model_ver)
  enqueue(completeness, claim, required_set)
workers: consume -> run inference -> emit ScoreEvent{claim, model, version, output, input_refs}
aggregator: when all expected scores present -> compute recommendation -> emit
  • Each task is idempotent and versioned. A task carries the exact input object refs (document versions) and the pinned model version. The worker writes back a ScoreEvent recording model id, version, inputs, and output. Reprocessing the same task (after a worker crash / at-least-once delivery) is a no-op by (claim_id, task_id, model_version) dedup. Recording the version is not optional - it is what makes the decision reproducible for audit.
  • The aggregation / recommendation step. Once all expected scores for a claim are in, an aggregator combines them into a recommendation using a deterministic, versioned rules policy (not a black box): e.g. “low value AND fraud_score < 0.1 AND completeness=OK AND damage within policy -> AUTO_APPROVE”; “policy excludes incident -> AUTO_DENY”; “fraud_score > 0.6 OR high value OR ambiguous damage -> ROUTE_TO_HUMAN.” The policy version is recorded too. Keeping the final gate as explicit, versioned rules (fed by ML scores) rather than one end-to-end model is deliberate: a regulator can read why a claim was auto-denied, and you can change thresholds without retraining.
  • Human-in-the-loop by design. Most claims route to a human. Auto-approve is reserved for small, clean, low-risk claims where the expected cost of a bad auto-approval is low; auto-deny only for unambiguous policy exclusions (and even those are appealable). The ML layer’s job is to triage and pre-fill, not to replace the adjuster on anything consequential - both for fairness/regulatory reasons and because a wrong auto-deny is a reputational and legal event.
  • Graceful failure. A model that fails is retried with backoff; after N failures the task is dead-lettered and the claim defaults to ROUTE_TO_HUMAN (fail safe toward a human, never silently auto-deny on a model error). A slow model never blocks intake because intake already returned.
  • Re-screening on new documents. When a customer uploads more documents later (from PENDING_DOCS), only the affected tasks re-run; the pipeline is incremental, not all-or-nothing.

The elegant part: ML is a buffered, versioned, asynchronous triage layer, not a synchronous gate. Queues absorb bursts and decouple heavy GPU work from instant intake; recording model/policy versions per task makes every recommendation reproducible; and the explicit final rules policy keeps the auto-decision explainable - which is a regulatory requirement, not a nicety.

3. Adjuster routing and the claim state machine

Two tightly linked hard parts: getting the right claim to the right human exactly once, and modeling a claim that lives for weeks and moves backward.

Naive approach: a shared queue and a status column. All human-review claims go into one list; adjusters pick from the top. The claim has a status string each service overwrites with UPDATE claims SET status=? WHERE id=?.

Where it breaks:

  • Double-ownership (the correctness bug). Two adjusters open the top claim at the same instant; both “own” it; both decide. One claim, two conflicting approvals - a direct consistency and money violation.
  • Wrong adjuster. A shared queue ignores skill (a health adjuster should not assess a car crash), license region (regulated per state), and authority limit (a junior adjuster must not approve a 50 lakh claim). A free-for-all queue routes the wrong human to the wrong claim.
  • Load imbalance. Cherry-picking means easy claims get grabbed and hard/old ones rot; some adjusters drown while others idle. There is no fairness or aging.
  • Illegal transitions / lost history. A free-text status with no guard lets a retried message move DENIED -> UNDER_REVIEW, or two updates race and corrupt state. Overwriting the column loses the history - fatal when every transition is a legal record.
  • No backward movement. Real claims go backward constantly (request docs, escalate, appeal). A naive forward-only flow cannot model it.

The fix: an explicit guarded state machine, plus a routing service that matches skill/region/authority/load and atomically claims ownership.

Explicit states and legal transitions. Define the states and exactly which transitions are legal and who may trigger them:

  SUBMITTED ─(docs complete)─▶ SCREENING ─(reco: human)─▶ PENDING_REVIEW
     │                            │                            │ (routed)
 (invalid policy)           (reco: auto)                       ▼
     ▼                        │      │                    UNDER_REVIEW
  REJECTED              AUTO_APPROVE  AUTO_DENY            │   │   │
                           │            │        (need docs)│  │(escalate)
                           ▼            ▼         ◀─────────┘  │   ▼
                        APPROVED     DENIED ◀──┐   PENDING_DOCS │ ESCALATED
                           │           │(appeal)│      │        │   │
                        (payout)       └────────┘  (docs back)  │(decide)
                           ▼                            └───────┴──▶ APPROVED/DENIED
                         PAID ───▶ CLOSED

Each transition is a guarded compare-and-swap: UPDATE claims SET status=?, owner=?, version=version+1 WHERE claim_id=? AND status=<expected> AND version=<expected_version>. The AND status=expected makes transitions atomic and idempotent - a retried event when already in the target state is a no-op, and an illegal jump (DENIED -> UNDER_REVIEW without an appeal) fails the guard. This satisfies the strong-consistency NFR for the claim.

Backward transitions are first-class. UNDER_REVIEW -> PENDING_DOCS (need more info), -> ESCALATED (beyond authority), DENIED -> UNDER_REVIEW (appeal reopens) are legal, defined edges - not hacks. The state machine explicitly models the loops a real claim takes.

Routing: match then atomically claim. The Routing Service, on PENDING_REVIEW, selects a candidate adjuster:

route(claim):
  eligible = adjusters WHERE skill == claim.line
                         AND region covers claim.jurisdiction
                         AND authority_limit >= claim.estimated_value
                         AND status == ACTIVE
  pick = least_loaded(eligible, tie_break=oldest_claim_first)  # fairness + aging
  # atomically claim ownership so no one else gets it:
  if CAS(claim, owner=NULL -> owner=pick, status: PENDING_REVIEW -> ROUTED):
      enqueue(pick.workqueue, claim); notify(pick)
  else: retry  # lost the race, re-route
  • Skill / region / authority filter guarantees the right human (regulatory and competence constraints).
  • Least-loaded with oldest-first tie-break balances load and ages old claims so nothing rots - fairness the shared queue lacked.
  • Atomic ownership claim (CAS setting owner from NULL) is the single strong-consistency point that kills double-ownership: exactly one routing attempt wins; the claim is pinned to one adjuster. Opening it moves ROUTED -> UNDER_REVIEW. If an adjuster goes idle/sick, a supervisor or a timeout releases ownership (owner -> NULL, back to PENDING_REVIEW) for re-routing - with that release itself audited.

Authority and escalation. An adjuster can only approve up to their authority_limit. Approving above it, or a judgment call they want reviewed, transitions -> ESCALATED and re-routes to a higher-authority adjuster. This enforces the regulatory control that large payouts need senior sign-off.

Every transition is an event, not just a column write. Alongside the guarded status update, append an immutable event to the audit log (next section). The status column is a materialized convenience for fast queries; the event log is the truth.

The elegant part: one guarded state machine handles the whole weeks-long, back-and-forth lifecycle, and one atomic ownership claim is the only strong lock needed for human routing. Skill/region/authority matching puts the right human on each claim; least-loaded-with-aging keeps it fair; backward edges are modeled, not bolted on; and the CAS makes double-decisioning impossible.

4. The immutable regulatory audit trail

This is the requirement that makes insurance claims different from a generic workflow system, so it gets its own deep dive.

Naive approach: an updated_at column and some application logs. Record the latest status and timestamp on the claim row; rely on scattered app logs for “who did what.”

Where it breaks:

  • Mutable state has no history. An updated_at column tells you the current state, not the sequence of decisions. When a regulator asks “on what basis was this claim denied on 3 March, and what did the system see,” a current-state row cannot answer.
  • App logs are not an audit trail. They are unstructured, rotated away, not tamper-evident, incomplete, and not tied transactionally to the state change. You cannot hand a court a Kibana dashboard.
  • No reproducibility. Without recording which model version scored the claim and which document versions it saw, you cannot reconstruct the decision. If the model was retrained since, you have lost the ability to explain the original outcome.
  • Not tamper-evident. A plain table can be UPDATE-ed or DELETE-ed. For a legal record you must be able to prove nothing was altered after the fact.

The fix: an append-only, hash-chained, write-once event log, with full input capture, versioned document and model references, and WORM archival.

  • Append-only event sourcing for the claim. Every meaningful thing is an immutable event: ClaimSubmitted, DocumentUploaded(version), ScoreProduced(model, version, inputs, output), RecommendationMade(policy_version), Routed(adjuster), MoreDocsRequested, Decided(adjuster, outcome, reason_code, amount), PaidOut, Closed. Events are never updated or deleted. The claim’s current status is a projection of this log. The log is the audit trail - it is not a separate thing bolted on.
Event: {
  event_id, claim_id, seq,              # seq = per-claim monotonic order
  type: "Decided",
  actor: {id: "adj_88", role: "adjuster"},
  ts, 
  payload: {outcome: "APPROVED", amount: 45000, reason_code: "COVERED_MOTOR_OD"},
  input_refs: [doc_v, doc_v, score_event_ids...],  # exactly what was seen
  prev_hash,                            # hash of previous event
  hash = H(prev_hash || canonical(this_event))     # tamper-evident chain
}
  • Hash chaining for tamper-evidence. Each event stores the hash of the previous event and its own hash over (prev_hash + its content). Altering any past event breaks the chain for every subsequent event, which is detectable. Periodically the chain head is anchored (signed, or written to a WORM store / external notarization) so the whole history is provably intact. This is what lets you assert to a regulator that the record was not altered.
  • Capture inputs by reference, immutably. An event records which document versions and which score events fed a decision. Because documents are immutable-and-versioned (section 1) and scores record their model version (section 2), replaying a closed claim reconstructs exactly what the system saw and did - the reproducibility NFR. “Show me why claim X was denied” returns the inputs, the model version and its output, the recommendation policy version, and the adjuster’s coded reason.
  • Transactional with the state transition. The audit event is written in the same transaction (or atomically linked) as the guarded status CAS. A transition that is not durably audited must not be considered to have happened - durability of the audit is as strong as durability of the state. In practice: write the event and flip the projection together; on the event store, the append is the commit.
  • Access auditing too. Because claims carry medical/financial PII, reads are audited: DocumentViewed(adjuster, doc, ts). Who looked at a sensitive record is itself a regulated question.
  • WORM archival and retention. The audit log is written to write-once-read-many storage and retained for the 7-10 year window, tiered to cold archive as claims close, encrypted throughout. It is never purged before its retention expiry, and even expiry is a governed, logged process.

The elegant part: the audit trail is not a feature added on top - it is the primary data model. The claim state is a projection of an append-only, hash-chained event log that captures every input by immutable version and every decision by actor and reason. That single design choice delivers reproducibility, tamper-evidence, and legal retention at once, and it is why the whole system is modeled as events rather than mutable rows.

API Design & Data Schema

Intake and adjuster actions are REST; document upload is presigned direct-to-object-store; processing is event-driven internally.

REST (customer-facing)

POST /api/v1/claims
  body: { policy_id, incident_type, incident_date, location, description }
  -> 201 { claim_id: "clm_7f3", status: "submitted", required_documents: [...] }

POST /api/v1/claims/{claim_id}/documents            # get a presigned upload URL
  body: { filename, doc_type, size, sha256 }
  -> { document_id, upload_url, method:"PUT", expires_in: 900, multipart?:{...} }

POST /api/v1/claims/{claim_id}/documents/{document_id}/complete
  -> { document_id, status: "scanning" }            # -> available after scan

GET  /api/v1/claims/{claim_id}
  -> { claim_id, status: "under_review", incident_type, submitted_at,
       required_documents:[{type, status}], missing:[...],
       decision?:{outcome, amount, reason}, history:[{state, ts}] }

REST (adjuster-facing)

GET  /api/v1/adjusters/me/workqueue?limit=25
  -> { claims:[ {claim_id, line, value_est, fraud_score, age_hours,
                 ml_recommendation, priority} ] }   # cached projection

POST /api/v1/claims/{claim_id}/open   -> { lock:"owned", documents:[...], scores:[...] }
POST /api/v1/claims/{claim_id}/request-docs  body:{ doc_types:[...], note }
                                      -> { status:"pending_docs" }
POST /api/v1/claims/{claim_id}/escalate      body:{ reason }
                                      -> { status:"escalated" }
POST /api/v1/claims/{claim_id}/decide
  body: { outcome:"approved"|"denied", amount?, reason_code, notes }
  -> { status:"approved", amount:45000 }            # guarded: must own + be UNDER_REVIEW

Internal / audit

GET /internal/claims/{claim_id}/audit               # the regulator-facing export
  -> { events:[ {seq, type, actor, ts, payload, input_refs, hash, prev_hash} ],
       chain_verified: true }

Data stores - the right tool per plane

1. Claim + workflow store - relational (PostgreSQL, sharded by claim_id). The claim state machine demands SQL: strict guarded transitions, relationships (policy, adjuster, documents, payout), and small volume (tens of writes/sec). ACID gives us the atomic CAS transitions and ownership claims.

Table: claims
  claim_id       BIGINT  PRIMARY KEY            -- Snowflake, time-sortable
  policy_id      BIGINT  INDEX
  customer_id    BIGINT  INDEX
  incident_type  ENUM(motor, health, property, ...)
  jurisdiction   VARCHAR                        -- region/state for licensing
  status         ENUM(submitted, screening, pending_docs, pending_review,
                      routed, under_review, escalated, approved, denied,
                      paid, closed, rejected)
  owner_adjuster BIGINT  NULL INDEX             -- CAS target for ownership
  estimated_value NUMERIC
  fraud_score    NUMERIC NULL                   -- latest, for triage/priority
  recommendation ENUM(auto_approve, auto_deny, route_to_human) NULL
  version        BIGINT                         -- optimistic-lock for CAS
  shard_key      = claim_id                     -- intake volume is tiny; shard for size/retention
  submitted_at, decided_at, closed_at  TIMESTAMP
  Indexes: (status), (owner_adjuster, status), (policy_id), (submitted_at)

Table: claim_events                             -- APPEND-ONLY, the audit source of truth
  event_id   BIGINT PK
  claim_id   BIGINT INDEX
  seq        INT                                -- per-claim monotonic (UNIQUE claim_id,seq)
  type       VARCHAR
  actor_id   BIGINT, actor_role ENUM
  ts         TIMESTAMP
  payload    JSONB                              -- outcome, amount, reason_code, scores
  input_refs JSONB                              -- doc versions + score event ids
  prev_hash  BYTEA, hash BYTEA                  -- tamper-evident chain
  -- INSERT only; no UPDATE/DELETE grant; archived to WORM

Table: documents
  document_id   BIGINT PK
  claim_id      BIGINT INDEX
  doc_type      ENUM(photo, police_report, bill, prescription, estimate, ...)
  version       INT                             -- immutable; corrections add versions
  object_key    VARCHAR                         -- pointer into object storage
  sha256        BYTEA, size BIGINT
  scan_status   ENUM(scanning, clean, quarantined)
  status        ENUM(uploaded, available)
  uploaded_at   TIMESTAMP

2. Documents / photos - object storage (S3-class), NOT the database. Petabytes, immutable, versioned, tiered hot -> IA -> cold archive over the retention window, encrypted, per-claim access control. The DB holds only the small metadata + object key.

3. ML scores - recorded as claim_events (model, version, inputs, output). The score is an audit event; there is no separate mutable scores table that could drift. A read-model/materialized view surfaces the latest scores for the adjuster UI.

4. Workqueue / dashboards - a cached read-model (Redis + a search index). The adjuster workqueue is a projection built from claim events, kept in Redis/Elasticsearch for fast, filterable, eventually-consistent queue reads (330/sec, non-critical). Ownership truth stays in SQL; the queue is just a fast view.

5. Audit archive - WORM / cold object storage. The claim_events chain is periodically sealed and written to write-once storage for the 7-10 year legal tail.

Why SQL for claims/workflow, object storage for documents, event log for scores and audit, and a cached read-model for queues: the control plane is small, relational, and needs ACID transitions; the documents are petabytes of immutable blobs; the audit/scores must be append-only and reproducible; and the queues are high-read, staleness-tolerant views. A durable blob store for photos keeps the DB lean; putting the audit as the primary event log (not an afterthought column) is what makes the whole thing reproducible and tamper-evident.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Document ingestion through the app tier (breaks first). 150 MB/sec of uploads flowing through application servers saturates them and bloats the DB. Fix: presigned direct-to-object-store upload - bytes bypass the app entirely; the app handles only tiny metadata requests. Resumable multipart for flaky mobile networks; checksum + idempotent complete. The control plane never touches a byte of claim data.

2. Synchronous ML on the submit path. Running OCR/CV/fraud inline blocks intake for seconds and cannot handle documents arriving over time. Fix: event-driven, queue-fed ML workers (autoscaled GPU pool) decoupled from intake. Intake returns instantly; bursts (a storm, 30x) buffer in Kafka; workers scale with backlog; a slow or failing model never blocks intake and fails safe to a human.

3. The claim state machine under concurrent transitions. Two adjusters deciding the same claim; retried events causing illegal jumps. Fix: guarded CAS transitions (WHERE status=expected AND version=expected) make every transition atomic and idempotent, and the atomic ownership claim (owner NULL -> adjuster) makes double-ownership impossible. Exactly one of a race wins; illegal jumps fail the guard.

4. Wrong/unfair routing and rotting claims. A shared queue routes the wrong skill/region/authority and lets hard claims age. Fix: skill + region + authority filter then least-loaded with oldest-first tie-break, with ownership released on adjuster absence for re-routing. Right human, balanced load, nothing rots.

5. Audit integrity and reproducibility. A plain status column has no history, is mutable, and loses which model/document version produced a decision. Fix: append-only, hash-chained event log as the primary model, capturing inputs by immutable version and decisions by actor/reason, written transactionally with the state change, sealed to WORM. Reconstructs any closed claim and proves it was not altered.

6. ML model drift breaking reproducibility. Retraining a model would make old decisions unexplainable. Fix: pin and record model version per score event; keep old model versions deployable/replayable; the final auto-decision is an explicit, versioned rules policy over scores, so “why was this auto-denied” is always answerable regardless of later retraining.

7. Storage cost over a decade. 16 PB of documents retained 7-10 years. Fix: lifecycle tiering - active-claim documents hot, closed-claim documents to infrequent-access then cold archive; audit chain sealed to WORM cold storage. Most of the 16 PB is cheap cold data; only active-claim documents are hot.

8. Hot partitions / surge events. A regional storm or a big accident spikes claims and documents for one area at once. Fix: intake and workers are stateless and autoscale; sharding by claim_id spreads load evenly (no geographic hot shard); the queue absorbs the processing burst; routing naturally spreads to the eligible adjuster pool. Surge is a backlog that drains, not an outage.

9. Adjuster workqueue read load / staleness. 10K adjusters polling queues. Fix: serve the queue from a cached read-model (Redis/Elasticsearch projection of claim events), eventually consistent and filterable. Ownership truth stays in SQL; the queue is a fast, disposable view rebuilt from the event log.

10. PII exposure. Medical/financial documents and who views them. Fix: encryption at rest and in transit, per-claim document access control, presigned URLs that are short-lived and scoped, and read-auditing (DocumentViewed events) so access itself is part of the regulated trail.

11. Single points of failure. Intake, Workflow, Routing, Document services are stateless behind the LB and replicated; Kafka, the claim DB, object storage, and the audit store are all replicated with failover; ML workers are a fungible pool. The one strongly-consistent component (the transition/ownership CAS) runs on the replicated relational store. No single box whose loss stops claim intake or loses an audit record - at worst processing backlogs and drains.

12. Payout handoff reliability. An approved claim must result in exactly one disbursement. Fix: the payout handoff carries an idempotency key (the claim_id + decision seq); the Payments system dedups, so a retried handoff after a network blip never double-pays. The claim only moves to PAID on a confirmed settlement callback, recorded as an audit event.

Wrap-Up

The trade-offs that define this design:

  • Control plane and data plane fully separated. New-claim QPS is trivial (tens/sec) and lives in a strongly-consistent relational store; the petabytes of documents live in object storage reached by presigned direct upload. We trade a slightly more complex upload handshake for an app tier that never moves a byte of claim data and a database that stays lean.
  • ML as asynchronous, versioned triage - never a synchronous gate. Pre-screening runs on queue-fed autoscaled workers, decoupled from instant intake, with every score pinned to a model version and the final auto-decision an explicit rules policy. We trade “the recommendation is ready at submit” for burst tolerance, document-arrives-over-time handling, and - crucially - explainability and reproducibility.
  • Human-in-the-loop by default. Auto-approve only for small, clean, low-risk claims; auto-deny only for unambiguous exclusions; everything consequential goes to the right human, failing safe to a human on any model error. We trade some automation rate for correctness and regulatory defensibility.
  • The claim as a guarded state machine with first-class backward edges. One state machine models the weeks-long, back-and-forth lifecycle (request docs, escalate, appeal); guarded CAS transitions and one atomic ownership claim make illegal jumps and double-decisioning impossible. We pay for strong consistency exactly at the transition and the ownership claim, nowhere else.
  • The audit trail is the primary data model, not an add-on. The claim state is a projection of an append-only, hash-chained event log that captures every input by immutable version and every decision by actor and reason, sealed to WORM for a decade. That one choice delivers tamper-evidence, reproducibility, and legal retention together - and it is why the whole system is modeled as events rather than mutable rows.

One-line summary: an insurance-claims platform where a trivial relational control plane runs claims as strongly-consistent, backward-capable state machines; documents flow petabyte-scale into tiered object storage via presigned direct upload; an asynchronous, version-pinned ML pipeline pre-screens off queues and triages to the right adjuster by skill/region/authority/load with an atomic ownership claim; and every input and decision is recorded in an append-only, hash-chained, WORM-archived audit log that makes any closed claim reproducible and tamper-evident years later.