Content moderation looks like a machine learning problem and ends up being a systems problem. The model is the easy part - a toxicity classifier is a solved research problem you can download. The hard part is running it on 1 billion comments a day, in the write path, inside a 100ms budget, while attackers mutate their phrasing faster than you can retrain. If scoring a comment takes 300ms, every comment post gets 300ms slower and your throughput falls off a cliff. If you put the model behind the write and score async, abusive comments go live before you catch them. And if your filter is static, the spammers route around it by lunchtime: f r e e m0n3y, zero-width characters between letters, homoglyphs, image-text, the same slur spelled nine ways.

So the real question is not “how do I classify toxicity.” It is: how do I score a billion comments a day in real time, cheaply enough to afford, fast enough to block inline, and adaptable enough to survive adversaries who treat your filter as a puzzle to solve. That means a tiered pipeline where 95% of comments never touch the expensive model, a decision policy that separates block from flag from allow, and a feedback loop that turns human reviewers and user reports into new training data on a short cycle. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Score every comment for toxicity, hate speech, and spam. On submission, a comment gets one or more risk scores (toxicity, harassment, hate, spam, sexual, self-harm, violence) before it becomes visible.
  • Real-time block / flag / allow decision. Based on scores and policy, a comment is either allowed through, soft-blocked (hidden pending review), hard-blocked (rejected outright), or flagged (published but queued for human review).
  • Sub-100ms inline scoring. The decision returns inside the comment-post request so clearly abusive content never goes live, and the author sees a rejection immediately.
  • Human review queue. Flagged and soft-blocked comments land in a prioritized queue for human moderators, whose decisions are authoritative and feed back into training.
  • User reports. Readers can report a published comment; a report re-scores it and can escalate it into the review queue.
  • Evolving attack adaptation. New obfuscation patterns and slur variants get caught quickly - a short loop from “humans label it bad” to “the filter catches it automatically.”
  • Policy configurability. Per-surface thresholds (a kids’ forum is stricter than a gaming chat), per-language models, and the ability to change thresholds without redeploying.
  • Appeals. An author whose comment was blocked can appeal; an overturned decision is also training signal (a false positive).

Explicitly out of scope (so I own the scope):

  • Image / video / audio moderation. This is a separate (heavier) pipeline. I handle text, and note where a multimodal hook sits for image-text (memes).
  • Training the models themselves. I design the serving and feedback systems and treat the classifier as a versioned artifact produced offline. I will not derive a transformer architecture.
  • Account-level actions (bans, strikes, shadow-bans). The output is a per-comment decision plus signals; the trust-and-safety account system consumes those downstream.
  • Legal / CSAM reporting workflows. Mandatory-reporting pipelines are their own regulated system; I note the handoff but do not design it.
  • The comment storage / display system. Covered elsewhere; here the moderation service is a dependency the comment-write path calls.

The one decision that drives everything: this is a latency-bounded, cost-bounded ML inference pipeline in the write path, not an offline batch classifier. The whole design is about doing as little expensive work as possible per comment while still catching the bad ones, and about closing the loop fast enough that a static model never becomes the weak point.

Non-Functional Requirements (NFR)

  • Scale: 1 billion comments/day. Average ~11.6K comments/sec, peak ~3-4x at ~40K/sec. Hundreds of millions of these are trivially clean (short, benign) and must be cheap.
  • Latency: p99 scoring under 100ms inline. The common case (clean comment) should be far under that (single-digit ms) so the pipeline cost is dominated by the rare expensive path. The author-facing post request adds this budget on top of the normal write.
  • Availability: 99.9%+ on the scoring path. But availability here has a policy dimension: when the scorer is down or over budget, do we fail-open (allow, risk missing abuse) or fail-closed (block, risk blocking good content)? The answer is per-surface and part of the design, not an afterthought.
  • Consistency: the decision for a given comment must be deterministic and reproducible - same comment, same model version, same policy version yields the same decision, because we have to audit and explain blocks. Model/policy versions are pinned per decision and logged.
  • Durability: every decision (scores, model version, policy version, outcome) is durably logged for audit, appeals, metrics, and as training data. No decision is silently lost. The comment content for flagged items is retained for the review window.
  • Freshness / adaptability: the time from “a new attack pattern is observed and labeled” to “the filter catches it” must be hours, not weeks. Fast-path rules (blocklists, regex, hashes) update in minutes; model updates ship on a daily-ish cadence; thresholds change in seconds.
  • Cost: the GPU/accelerator bill for running a transformer on 1B comments/day is the dominant cost. The architecture must drive the fraction of comments that hit the big model as low as possible (ideally single-digit percent) without letting abuse slip.

That cost line is the unlock. Because the expensive model is the budget killer, the entire pipeline is organized to avoid invoking it for the overwhelming majority of comments.

Back-of-the-Envelope Estimation (BoE)

Real numbers, because they dictate the architecture.

Throughput:

1,000,000,000 comments/day
/ 86,400 sec/day         = ~11,574 comments/sec average
Peak (3-4x diurnal)      = ~40,000 comments/sec

The naive cost (why this is hard): run the big model on everything.

A quality toxicity transformer (BERT-ish, ~100M params) on a GPU:
  ~5-15ms per short comment batched, say effective 10ms of accelerator time.
1B comments/day * 10ms = 10,000,000 accelerator-seconds/day
  = ~116 accelerator-seconds per wall-clock second
  -> need ~116 GPUs saturated 24/7 JUST for steady state,
     and ~400 for peak, with zero headroom.
At cloud GPU ~$2/hr: 400 GPUs * 24 * $2 = ~$19,200/day = ~$7M/year
  just for inference, before redundancy, before multi-region.

Seven million dollars a year to run a model on comments that are 95% “nice recipe, thanks!” is absurd. Every design decision below exists to shrink that number.

The tiered reality: most comments are trivially clean.

Assume realistic mix of 1B/day:
  ~85% resolved by cheap tier (length/lang/cache/rules/tiny model)  -> ~850M
  ~13% need the medium model                                        -> ~130M
  ~2%  need the big model + maybe human review                      -> ~20M

Big-model load: 20M/day * 10ms = 200,000 acc-sec/day
  = ~2.3 acc-sec/sec  -> ~3 GPUs steady, ~10 at peak.
From ~400 GPUs to ~10. That is the entire point of tiering.

Human review capacity:

~2% flagged/soft-blocked = 20M/day reaching review in the worst framing,
but policy only sends the UNCERTAIN band to humans, say ~0.2% = 2M/day.
A moderator handles ~1 item / 10-20 sec sustained -> ~2,000/hour.
2M/day / 2,000 per moderator-hour = ~1,000 moderator-hours/day
  = ~125 moderators on 8h shifts (before timezones/redundancy).
This is why precision matters: every false positive is human cost.

Storage (decision log + training data):

Decision record ~ 500 bytes (ids, scores, versions, outcome, features).
1B/day * 500B = ~500 GB/day of decision logs
  -> ~180 TB/year.  Hot for 90 days, then cold/archive.
Retained comment text for flagged items (~2%): 20M * ~300B = ~6 GB/day.

Cache sizing (dedup of repeated content):

Spam and copy-paste abuse repeat heavily. Hash(normalized text) -> decision.
Assume ~30% of comments are near-duplicates of something seen in 24h.
Cache ~50M distinct normalized hashes * (16B hash + 8B decision) = ~1.2 GB.
Trivially fits in Redis; a 30% hit rate removes 300M/day from the model path.

Bandwidth / payload: comments are tiny (~300 bytes). The data plane is cheap; the compute plane (model inference) is the whole cost. This is the inverse of the fan-out problems - here bytes are free and FLOPs are expensive.

Takeaways: naive “big model on everything” is ~400 GPUs and ~$7M/year. Tiering (cheap filters + cache + cascade) cuts the big-model path ~20x to ~10 GPUs. The dominant costs become inference compute and human review, so the architecture optimizes both: do less model work, and send fewer items to humans.

High-Level Design (HLD)

Three planes. A synchronous scoring plane sits in the comment-write path and returns a decision inside 100ms using a cascade of increasingly expensive scorers, short-circuiting as early as it can. An asynchronous review + feedback plane handles human moderation, user reports, and appeals, and turns their verdicts into labeled data. An offline training + deployment plane retrains models and compiles rule/blocklist updates, shipping them back to the scoring plane on their own cadences.

   Author posts comment
        │ POST /comments  (text)
        ▼
  ┌──────────────┐     calls (sync, <100ms budget)      ┌───────────────────────┐
  │ Comment-Write │ ───────────────────────────────────▶│  Moderation Gateway    │
  │   Service     │ ◀─────────────────────────────────── │  (orchestrates cascade)│
  └──────┬───────┘        {decision, scores, ids}        └───────────┬───────────┘
         │ allow -> publish                                          │
         │ block -> reject to author                                 ▼
         │ flag  -> publish + enqueue           ┌────────────────────────────────────┐
         ▼                                       │  TIERED SCORING CASCADE (sync)      │
  (comment store)                                │  T0 normalize + dedup cache (Redis) │
                                                 │  T1 rules: blocklist/regex/hash     │
                                                 │  T2 tiny model (logreg/CNN, CPU)    │
                                                 │  T3 medium transformer (GPU batch)  │
                                                 │  T4 big ensemble (GPU, rare)        │
                                                 └───────┬──────────────┬─────────────┘
                                                         │ scores       │ features
                                                         ▼              ▼
                                       ┌──────────────────────┐  ┌──────────────────┐
                                       │ Decision Policy Engine │  │ Decision Log     │
                                       │ thresholds per surface │  │ (Kafka -> store) │
                                       └──────────┬────────────┘  └────────┬─────────┘
            uncertain band / flagged ────────────▼                         │
                                       ┌──────────────────────┐            │
                                       │ Review Queue (priority)│           │
                                       │  + user reports/appeals│           │
                                       └──────────┬────────────┘            │
                                         human verdicts                     │
                                                  ▼                         ▼
                                       ┌─────────────────────────────────────────────┐
                                       │ OFFLINE: label store -> train models ->      │
                                       │ eval/shadow -> deploy; compile blocklists/   │
                                       │ regex/hashes -> push to T1 (minutes)         │
                                       └─────────────────────────────────────────────┘

Synchronous scoring flow (author posts a comment):

  1. Author sends POST /posts/{id}/comments. The Comment-Write Service calls the Moderation Gateway with the text, author context (account age, prior strikes), and surface id - synchronously, with a hard ~80ms deadline so the total post stays under 100ms.
  2. The gateway runs the cascade. T0 normalizes the text (unicode NFKC, lowercase, strip zero-width, collapse look-alikes) and checks the dedup cache: have we scored this exact normalized text recently? If yes, return the cached decision - done in ~1ms.
  3. On a cache miss, T1 rules run: exact blocklist hits, high-precision regex (known spam URLs, slur variants), and known-bad content hashes. A confident hit short-circuits to a block without any model call.
  4. If still uncertain, T2 runs a tiny CPU model (logistic regression / small CNN on char n-grams) - sub-millisecond, cheap. If it is confidently clean or confidently toxic, decide now.
  5. Only the uncertain middle escalates to T3, the medium transformer on a GPU (micro-batched). The rare, high-stakes ambiguous cases (or a sample for monitoring) go to T4, a bigger ensemble.
  6. The Decision Policy Engine maps the final scores + author context + surface policy to one of allow / flag / soft-block / hard-block, pinning the model and policy versions. The gateway returns it; the Comment-Write Service publishes, rejects, or publishes-and-enqueues accordingly.
  7. Every decision is written to the Decision Log (async, fire-and-forget to Kafka) for audit, metrics, and training.

Asynchronous review + feedback flow:

  1. Flagged, soft-blocked, and reported comments land in the Review Queue, prioritized by score, surface sensitivity, virality, and reporter trust.
  2. Human moderators resolve them; each verdict is an authoritative label written to the label store. Appeals and false-positive overturns are labels too.
  3. Offline, the label store feeds (a) model retraining and (b) rule/blocklist compilation. New blocklist/regex/hash entries push to T1 within minutes; retrained models ship daily after shadow evaluation.

The key insight: the cascade is a funnel, and the policy engine is separate from the scorers. Scorers produce probabilities; policy turns probabilities into actions and owns the thresholds, the fail-open/closed choice, and the per-surface strictness - so you can tune behavior without touching models, and audit exactly why any comment was blocked.

Component Deep Dive

The hard parts, naive-first then evolved: (1) hitting sub-100ms at 1B/day without a GPU bill that bankrupts you, (2) the block-vs-flag decision policy and failure modes, (3) adversarial obfuscation and the normalization arms race, (4) the feedback loop that keeps the filter current.

1. Sub-100ms scoring at 1B/day without going broke

Naive approach: one model service, call it synchronously for every comment. The write path does score = model.predict(text) on a transformer and blocks on the result.

on new comment c:
    scores = big_model.infer(c.text)   # ~10ms accelerator + queue + network
    decision = scores.toxicity > 0.8 ? BLOCK : ALLOW

Where it breaks:

  • Cost. As computed, running the big model on all 1B/day is ~400 peak GPUs and ~$7M/year. You are paying transformer prices to classify “lol same” a billion times.
  • Latency and queueing. At 40K/sec peak, a single model tier must batch to be efficient, but batching adds queue wait. Under load the inference queue grows and p99 blows past 100ms - the classic throughput-latency collision. One stuck batch stalls everyone behind it.
  • Tail amplification. A GPU node hiccups (model reload, memory pressure) and every comment waiting on it misses the 100ms deadline. The whole write path inherits the model’s worst tail.

First evolution: a cache for repeated content. Spam and copy-paste abuse repeat heavily. Normalize the text and key a cache hash(normalized) -> decision. A ~30% hit rate removes ~300M/day from the model entirely at ~1ms each. This is the cheapest possible win and it disproportionately catches spam, which is the most repetitive abuse.

Second evolution: a cascade of cheap-to-expensive scorers that short-circuit. Order scorers by cost and only escalate the uncertain ones. The trick is that cheap scorers are confident on most inputs - a comment with no risky tokens is obviously clean, a comment that is a known slur is obviously toxic. You only pay for the big model on the genuinely ambiguous middle.

score(c):
    t = normalize(c.text)                       # T0
    if hit = cache.get(hash(t)): return hit     # ~30% of traffic, ~1ms
    if r = rules.match(t):                        # T1 blocklist/regex/hash
        return cache.put(hash(t), r)             # confident block, no model
    s2 = tiny_model.predict(t)                    # T2 CPU, <1ms
    if s2 < LOW or s2 > HIGH:                      # confidently clean / toxic
        return decide(s2)                          # most of the rest
    s3 = medium_model.predict(t)                  # T3 GPU, only the uncertain
    if s3 in GREY_BAND and c.high_stakes:
        s4 = big_ensemble.predict(t)              # T4 GPU, rare
        return decide(s4)
    return decide(s3)

Tier    Fraction of 1B/day   Cost/item        Catches
T0 cache     ~30%            ~1ms CPU/Redis    repeats, known spam
T1 rules     ~25%            ~1ms CPU          blocklist, regex, hashes
T2 tiny      ~30%            <1ms CPU          obvious clean / obvious toxic
T3 medium    ~13%            ~10ms GPU         the ambiguous middle
T4 big       ~2%            ~20ms GPU          high-stakes grey band

Now the GPU path is ~15% of traffic, and the big model is ~2%. The ~400-GPU fleet becomes ~10-20 GPUs. Latency improves too: ~85% of comments finish in single-digit milliseconds on CPU and never queue behind a GPU batch at all.

Third evolution: isolate the GPU tier behind its own micro-batching service with a hard deadline and a fallback. The GPU tiers run as a separate service with dynamic batching (collect requests for ~5ms or until batch size N, whichever first) to keep accelerators busy without unbounded queue wait. The gateway calls it with a deadline; if the GPU tier is slow or unhealthy, the gateway does not hang - it applies the fallback policy (below) using whatever scores it already has from T1/T2. This decouples the write path’s tail from the model’s tail: the CPU tiers always answer fast, and the GPU tier is best-effort within budget.

gateway, GPU call with deadline:
    try: s3 = medium.infer(t, deadline=40ms)
    except Timeout | Unhealthy:
        return fallback_decision(t, s2, surface)   # never block the author on a slow GPU

The whole thing is a funnel whose job is to make the expensive tier rare. Cost and latency both fall out of the same idea: do the least work that confidently decides each comment.

2. The block / flag / allow policy and failure modes

Naive approach: one threshold on one score. if toxicity > 0.8: block else allow.

Where it breaks:

  • A single threshold can’t express the real actions. There is a difference between “obviously a slur, reject it outright,” “probably fine but let a human glance,” and “borderline, publish but watch.” One number and one cutoff collapse these into a false binary, so you either over-block (angry users, false positives are human-review cost) or under-block (abuse goes live).
  • Context-blind. The same text from a brand-new account on a kids’ forum is riskier than from a 10-year account in a gaming chat. A flat threshold ignores author trust and surface sensitivity.
  • No story for failure. When the scorer times out, a single if has no defined behavior - it just errors, and the write path either crashes or silently allows everything.

The fix: a separate Decision Policy Engine with bands, context, and an explicit failure policy.

  • Multiple scores, multiple bands. The models emit per-category scores (toxicity, hate, harassment, spam, sexual, self-harm, violence). Policy defines, per category and per surface, thresholds that split into bands:
score < t_allow            -> ALLOW
t_allow <= score < t_flag  -> FLAG      (publish, queue for review, watch)
t_flag  <= score < t_block -> SOFT_BLOCK(hide pending human review)
score >= t_block           -> HARD_BLOCK(reject to author inline)
  • Context adjusts thresholds, it doesn’t replace them. Author trust (account age, prior strikes, verified), surface sensitivity (kids’ forum vs adult gaming chat), and virality raise or lower the effective thresholds. A low-trust author on a sensitive surface gets stricter cutoffs. This is config, versioned and hot-reloadable - no model redeploy to make a surface stricter for a week.
  • Thresholds live in config, pinned per decision. Every decision logs {model_version, policy_version, scores, band, action}. That makes decisions reproducible and auditable - when an author appeals, you can replay exactly why it was blocked, and when you change a threshold you know which decisions it affects.
  • Explicit fail-open vs fail-closed, per surface. When the scorer is down or over budget, policy decides: a high-sensitivity surface fails closed (soft-block and queue, better to delay a good comment than leak abuse to minors); a low-stakes surface fails open (allow, better to keep posting flowing, catch it async). This is a product decision encoded in config, not an accident of a timeout.
policy.decide(scores, author, surface, scorer_health):
    if scorer_health == DOWN:
        return surface.fail_mode    # SOFT_BLOCK or ALLOW, per surface
    t = surface.thresholds.adjust(author.trust)
    cat, s = scores.max_category()
    return band(s, t)               # ALLOW / FLAG / SOFT_BLOCK / HARD_BLOCK

The principle: scorers estimate probability; policy owns action. Keeping them separate means you tune behavior, strictness, and failure handling without retraining anything, and every block is explainable from the pinned versions.

3. Adversarial obfuscation and the normalization arms race

Naive approach: match on the raw text. Blocklist and model see exactly what the user typed.

Where it breaks:

  • Trivial evasion. idiot becomes id!ot, 1d10t, i d i o t, i‌d‌i‌o‌t (zero-width joiners between letters), ⓘⓓⓘⓞⓣ, or Cyrillic homoglyphs that render identically to Latin. The raw blocklist misses all of them; the model, trained on clean text, sees out-of-distribution garbage and under-scores it.
  • Leetspeak and spacing defeat token models. Spreading a word with spaces or punctuation breaks tokenization so the toxic token never forms.
  • It is a moving target. Every normalization you add, attackers probe around the next day. A static matcher is permanently behind.

The fix: aggressive normalization before scoring, layered defenses, and fast rule updates.

  • A normalization pipeline, applied before every tier. Unicode NFKC normalization (collapses compatibility forms), lowercase, strip zero-width and combining characters, map homoglyphs to a canonical script (Cyrillic/Greek look-alikes to Latin via a confusables table), collapse repeated characters (heeeellooo to hello), de-leet (0->o, 1->i/l, 3->e, @->a, $->s), and remove inter-character separators so i d i o t and i.d.i.o.t reduce to idiot. Both the rule matcher and the models see the normalized form, so one canonicalization defends every tier at once.
  • Score multiple views, not just the aggressive one. Over-normalizing can create false positives (class contains a substring issue, assassin). So score both the normalized and a lightly-cleaned view and let policy consider both - aggressive normalization raises recall, the lighter view guards precision.
  • Character-level and subword models, not just word tokens. The T2/T3 models operate on character n-grams / subwords so that spacing and leet mutations still carry signal even if a clean word token never forms. Character models are inherently more robust to this class of obfuscation than word-token models.
  • Embedding / near-duplicate matching for variants. Beyond exact hashes, keep a vector index (or SimHash/MinHash) of known-bad content so a lightly-mutated repeat of a known abusive comment matches by similarity even when the exact hash differs. This catches “same spam, one word changed” without a model call.
  • The real defense is the loop speed, not any single rule. You will never pre-enumerate every obfuscation. What keeps you ahead is that when a human labels a new variant, a new regex / confusable mapping / blocklist entry / bad-hash compiles and pushes to T1 within minutes, and the variant becomes training data for the next daily model. The arms race is won on cycle time, which is why the feedback loop (next) is a first-class system, not a batch job.
normalize(text):
    t = nfkc(text); t = lower(t)
    t = strip_zero_width_and_combining(t)
    t = map_homoglyphs_to_latin(t)       # confusables table
    t = deleet(t)                         # 0->o 1->i 3->e @->a $->s
    t = collapse_repeats(t)               # heeello -> hello
    t = remove_inter_char_separators(t)   # "i d i o t" -> "idiot"
    return t
# score(normalize(t)) AND score(light_clean(t)); policy sees both.

The mental shift: you are not matching strings, you are canonicalizing an adversary’s infinite spellings into a finite space, then racing them on how fast you add the next canonical form. Normalization narrows the attack surface; the feedback loop closes the gaps that remain.

4. The feedback loop that keeps the filter current

Naive approach: train a model once, deploy it, done. The classifier is a fixed artifact.

Where it breaks:

  • Drift. Language, slang, memes, and attack patterns change weekly. A model frozen at training time steadily loses recall as attackers adapt and new slurs/spam campaigns appear.
  • No signal capture. Human moderator verdicts, user reports, and appeal overturns are the highest-quality labels you have, and a one-shot model throws them away.
  • No measurement. Without a loop you cannot even tell the model is degrading until users complain.

The fix: a closed loop from decisions and human verdicts back into rules and models, on three cadences.

  • Capture every label. Moderator verdicts (authoritative), user reports (weak/noisy signal, weighted by reporter trust), and appeal overturns (high-value false-positive labels) all write to a label store keyed to the original decision (with its pinned model/policy versions). Decision logs provide the features; labels provide the ground truth.
  • Three update cadences, matched to risk:
    • Seconds - thresholds. Policy config (per-surface thresholds, fail mode) changes live. A spam surge on one forum tightens its spam threshold immediately, no deploy.
    • Minutes - rules/blocklists/hashes. New exact-match entries, regexes, confusable mappings, and bad-content hashes compile from recent labels and push to all T1 nodes within minutes. This is the fast adversary counter - a new obfuscation seen at 10am is blocked by 10:15.
    • Daily-ish - models. Retrain T2/T3 on the accumulated labels, shadow-evaluate against a held-out set and a canary of live traffic (score but do not act, compare to current), and promote only if precision/recall clear the bar. Model versions are pinned per decision so you can attribute any regression.
  • Active learning on the grey band. The comments most worth a human label are the ones the model is least sure about (the T3 grey band) and the ones near decision boundaries. Prioritize those into the review queue, not random samples - each human minute buys the most model improvement, which is what keeps the ~125-moderator budget viable.
  • Guard the loop against poisoning. User reports are an attack vector (mass-report a rival to get them auto-blocked). So reports are signal, not verdict - they re-prioritize review and weight by reporter trust, but only a human verdict (or a very high model score) blocks. This stops coordinated reporting from weaponizing the filter.
  • Measure continuously. Track precision/recall on a sampled, human-audited golden set, false-positive rate (appeal overturns), and recall proxies (abuse caught post-publish via reports). When metrics drift, that triggers a rules update now and a model refresh next cycle.
label_store <- {moderator verdicts (authoritative),
                reports (weighted by trust),
                appeal overturns (false-positive labels)}
  -> thresholds   (seconds, config push)
  -> rules/hashes (minutes, compile + push to T1)
  -> models       (daily, train -> shadow eval -> canary -> promote, versioned)
active learning: prioritize T3 grey-band + boundary items into review queue.

The principle: a moderation system is only as current as its slowest useful update path. By splitting updates into seconds/minutes/daily lanes, the parts that must move fast (thresholds, rules) move fast, and the part that is expensive to change safely (models) moves deliberately with shadow evaluation - and the adversary is fought primarily on the minutes lane.

API Design & Data Schema

The scoring call is internal and synchronous; review, reports, and appeals are REST; decisions stream to a log.

Internal scoring + public endpoints

POST /internal/v1/moderate            (called by Comment-Write Service, sync)
  body: { "text": "...", "author_id": 123, "surface_id": "forum:kids",
          "context": { "account_age_days": 2, "prior_strikes": 1 },
          "client_dedup_id": "c-abc" }
  -> runs cascade + policy within deadline
  Response 200: {
    "decision_id": "d-9f3a", "action": "SOFT_BLOCK",
    "scores": { "toxicity": 0.72, "hate": 0.41, "spam": 0.05, ... },
    "model_version": "tox-2026-11-09", "policy_version": "p-337",
    "reason": "toxicity in soft-block band", "fail_mode": false
  }

POST /api/v1/comments/{comment_id}/report   { "reason": "harassment" }
  -> re-score + enqueue to review, weighted by reporter trust
  Response 202: { "report_id": "r-...", "status": "queued" }

GET  /api/v1/review/queue?surface=forum:kids&limit=50     (moderator UI)
  -> priority-ordered items: { comment, scores, context, report_count }

POST /api/v1/review/{decision_id}/verdict
  body: { "label": "toxic" | "clean", "categories": ["hate"] }
  -> authoritative label -> label store; updates comment visibility

POST /api/v1/appeals                 { "comment_id": "...", "note": "..." }
  -> enqueues for human review; overturn = false-positive label

The scoring endpoint is idempotent on client_dedup_id (a retry returns the same decision_id, no double-processing) and always returns within the deadline, falling back to the surface fail mode if the GPU tier times out.

Data stores

1. Dedup / decision cache - Redis (in-memory). modcache:{hash(normalized_text)} -> {action, scores, version} with a short TTL (hours) so stale decisions expire as models/rules change. In-memory, hot, rebuildable, point-lookup - the perfect fit, and the single highest-leverage component for cost.

Key:   modcache:{sha1(normalized_text)[:16]}
Value: {action, top_scores, model_version, ts}   TTL ~6h

2. Rules / blocklist / bad-hash store - Redis + in-process copy. Compiled rule sets (exact terms, regex set, confusable table, bad-content hashes, SimHash index) are published to Redis and pulled into each scoring node’s memory on a minutes-level refresh, so T1 matching is local and sub-millisecond with no per-comment network hop.

rules:version            -> "2026-11-10T10:15"
rules:blocklist          -> SET{normalized terms}
rules:regex              -> [compiled patterns]
rules:badhash            -> SET{content hashes}  + SimHash LSH buckets
confusables              -> MAP{codepoint -> latin}
# nodes poll rules:version; on change, pull and hot-swap the in-memory set.

3. Decision Log - Kafka -> columnar store (e.g. BigQuery / Parquet on object storage). Every decision is appended (fire-and-forget from the gateway so logging never blocks the write path). Append-only, write-heavy, analytical reads (metrics, training feature extraction, audits) - a log into a columnar warehouse, not a transactional DB.

Table: decisions  (columnar, partitioned by date)
  decision_id   STRING   PK
  comment_id    STRING
  author_id     BIGINT
  surface_id    STRING
  action        STRING                 -- allow/flag/soft_block/hard_block
  scores        MAP<STRING,FLOAT>      -- per category
  model_version STRING   INDEX         -- pinned, for reproducibility
  policy_version STRING  INDEX
  features      BYTES                   -- normalized text hash, context snapshot
  created_at    TIMESTAMP INDEX
  Partition by: date(created_at);  cluster by surface_id, action

4. Label / training store - object storage + a relational metadata DB. Human verdicts, weighted reports, and appeal overturns, joined to their decision records, form the training set. The metadata (who labeled, when, authority weight, which model consumed it) is relational (SQL) because it is modest-volume, relational, and audited; the bulk text/features sit in object storage.

Table: labels  (SQL - Postgres)
  label_id    BIGSERIAL PK
  decision_id STRING  FK -> decisions
  label       STRING                 -- toxic/clean + categories
  source      STRING                 -- moderator/report/appeal
  weight      FLOAT                  -- authority (moderator=1.0, report=0.1*trust)
  created_at  TIMESTAMP
  INDEX(decision_id), INDEX(created_at)

5. Review Queue - a priority queue (Redis sorted set / managed queue). Score = f(toxicity, surface sensitivity, virality, report count, reporter trust, model uncertainty). Highest-risk and most-uncertain first.

Key: review:{surface}  ZSET  member=decision_id  score=priority

Why NoSQL/Redis + a log + object storage, with SQL only for labels/metadata: the scoring path is point-lookup and in-memory (cache, rules) for latency; the decision firehose is append-only analytics (a log into columnar storage), not transactional; and only the label/audit metadata - modest volume, relational, needing integrity for training provenance and appeals - justifies a SQL store. Using one relational DB for the 1B/day decision firehose would collapse under write volume and buy nothing, since there are no cross-row transactions on the hot path. Determinism (reproducible decisions) comes from pinning versions in the log, not from strong DB consistency.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. GPU inference cost and the model tier (breaks first). Running a transformer on 1B/day is ~400 peak GPUs and ~$7M/year. Fix: the cascade funnel - T0 cache (~30%), T1 rules (~25%), T2 tiny CPU model (~30%) resolve ~85% before any GPU, leaving ~15% for T3 and ~2% for T4. GPU fleet drops to ~10-20. Scale GPU nodes horizontally behind the batching service, and autoscale on queue depth. The cache is the single biggest lever - protect its hit rate.

2. Latency tail from the GPU tier. Batching for GPU efficiency fights the 100ms deadline; one slow batch stalls everyone. Fix: isolate GPU tiers behind a micro-batching service with a bounded collect window (~5ms), call it with a hard deadline, and fall back to a T1/T2-only decision (plus the surface fail mode) on timeout so the author is never blocked on a slow GPU. CPU tiers answer ~85% of traffic in single-digit ms, decoupling the common-case latency from the model’s tail entirely.

3. Hot keys in the dedup cache. A viral spam campaign hammers one normalized hash. Fix: a hot key is good here - it is a cache hit, served from memory, and it keeps that spam out of the model. Replicate hot Redis shards for read scale; local in-process LRU in front of Redis absorbs the very hottest keys with zero network hop.

4. Rule/blocklist propagation lag. A new obfuscation is labeled but nodes still run the old rule set. Fix: compile rules centrally, publish a rules:version, and have scoring nodes poll and hot-swap their in-memory copy within minutes. Pull-based, so adding nodes doesn’t fan out writes; the version gate makes propagation observable (alert if a node lags).

5. Model staleness / drift. A frozen model loses recall as attackers adapt. Fix: the three-cadence loop - thresholds (seconds), rules (minutes), models (daily with shadow eval + canary). Active-learning the grey band into review maximizes label value per human-minute. Fight the adversary mainly on the minutes lane; move models deliberately.

6. Human review as a scaling wall. Too many flags overwhelm ~125 moderators; precision is a capacity problem, not just a quality one. Fix: only the uncertain band and high-signal reports go to humans (not everything flagged); prioritize by risk and uncertainty; every false positive (appeal overturn) is a training label to shrink future false positives. Raise precision to spend fewer human-minutes.

7. Feedback-loop poisoning. Coordinated mass-reporting to weaponize auto-blocking. Fix: reports are signal, not verdict - they re-prioritize review and are weighted by reporter trust; only human verdicts or very high model scores block. Rate-limit and trust-score reporters; detect report brigading as its own anomaly.

8. Sharding and routing, stated plainly. Scoring nodes are stateless and interchangeable - route by nothing but load; any node can score any comment because rules/models are replicated in-memory. The Decision Log shards by date + surface (time-series analytics). The dedup cache shards by hash(normalized_text) (even spread, hot keys are hits). The review queue shards by surface (moderators work a surface). There is no per-user or per-post hot partition on the hot path because scoring is content-local, not entity-local.

9. Single points of failure and the fail mode. The scorer being down is a policy event, not an outage. Fix: stateless scoring nodes behind a load balancer (lose one, lose nothing); Redis and the log replicated; and crucially, an explicit per-surface fail-open/fail-closed policy so a scorer outage degrades to a defined behavior (sensitive surfaces soft-block and queue; low-stakes surfaces allow and catch async) rather than crashing the write path or silently allowing all abuse.

10. Multi-region and multi-language. A global audience in many languages. Fix: scoring nodes per region (low latency, data residency) with rules/models replicated regionally; per-language model variants routed by detected language, with a language-agnostic character-level fallback for mixed/unknown scripts. The decision log and label store replicate asynchronously to a central training pipeline. Because decisions are content-local and reproducible from pinned versions, regional independence needs no cross-region consensus on the hot path.

Wrap-Up

The trade-offs that define this design:

  • A cheap-to-expensive cascade over one model for everything. T0 cache + T1 rules + T2 tiny model resolve ~85% of 1B/day before any GPU, cutting the big-model path ~20x (from ~400 GPUs to ~10-20). The entire architecture exists to make the expensive model rare, because inference compute is the dominant cost.
  • Policy separated from scoring. Scorers emit probabilities; a versioned Decision Policy Engine owns the block/flag/allow bands, per-surface strictness, context adjustment, and the explicit fail-open/closed choice - so behavior is tunable and every block is auditable without retraining.
  • Canonicalize the adversary, then out-cycle them. Aggressive normalization (unicode, homoglyphs, de-leet, strip separators) collapses infinite spellings into a finite space, character-level models resist what slips through, and the real defense is a minutes-level rules update loop that stays ahead of evolving attacks.
  • A three-cadence feedback loop. Thresholds change in seconds, rules/blocklists/hashes in minutes, models daily with shadow evaluation and canary - matched to how fast each can safely move and how fast the adversary forces it. Reports are signal, not verdict, to prevent poisoning.
  • Fail behavior is a product decision, not an accident. When the scorer is over budget or down, each surface degrades to a defined fail-open or fail-closed mode, so the write path never hangs and abuse never silently floods a sensitive surface.

One-line summary: a latency- and cost-bounded inference pipeline that scores 1B comments/day in sub-100ms by funneling ~85% through a cache-plus-rules-plus-tiny-model cascade and reserving GPUs for the ~15% that is genuinely ambiguous, with a separate versioned policy engine deciding block/flag/allow (and the fail mode), aggressive normalization plus a minutes-fast rules loop to survive evolving obfuscation, and a three-cadence feedback loop from human verdicts that keeps the filter current without ever letting the expensive model, the human reviewers, or a scorer outage become the thing that breaks.