LLM Batch Inference Cuts API Costs in Half - When the Latency Trade-off Is Worth It

Most teams look at the 24-hour SLA on batch APIs and stop reading. That is a mistake. The word “batch” makes people picture overnight mainframe jobs and 1990s ETL, so they never do the arithmetic. Here is the arithmetic: Anthropic and OpenAI both give you a flat 50 percent discount on input and output tokens if you are willing to accept an asynchronous completion window instead of a synchronous response. That is not a routing trick or a caching hack. It is the same model, the same quality, half the bill, in exchange for latency you were probably not using anyway.

The reflex “we need it real-time” is usually wrong. A large share of what production LLM systems do is not sitting in front of a waiting user. It is enrichment, classification, summarization, and evaluation that happens on a schedule or in the background. Those workloads do not care whether the answer arrives in 800 milliseconds or 20 minutes. They only feel expensive because you are paying real-time prices for work that has no real-time requirement.

What the batch APIs actually promise

Both major providers ship an asynchronous batch endpoint. The mechanics differ slightly but the deal is the same.

Two details matter more than the discount for capacity planning. First, the 24-hour window is a ceiling, not a target. In practice most batches finish in minutes to a couple of hours when the provider has spare capacity. You should design for 24 hours and be pleasantly surprised, but do not tell your stakeholders “overnight” if a two-hour turnaround changes the product.

Second, batch requests draw from a separate rate limit pool. If your real-time traffic is already bumping into tokens-per-minute limits, moving background work to batch does not just save money, it frees synchronous headroom. That second effect is often worth more than the 50 percent.

The naive setup and where it breaks

Here is what almost everyone builds first. A nightly job that classifies the day’s support tickets by sentiment and topic, one synchronous API call per ticket, in a loop.

import anthropic

client = anthropic.Anthropic()

def classify_tickets(tickets):
    results = {}
    for ticket in tickets:
        resp = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=256,
            messages=[{
                "role": "user",
                "content": f"Classify sentiment and topic:\n{ticket.body}"
            }],
        )
        results[ticket.id] = resp.content[0].text
    return results

This works at 500 tickets. It falls apart at 200,000, in three separate ways.

It is slow. Serial synchronous calls at ~1.5 seconds each means 200,000 tickets take over three days of wall-clock time. So you reach for concurrency.

It hits rate limits. Fire 200 concurrent requests and you start eating 429 responses, so you bolt on retries with backoff, a semaphore, and a queue. Now you have written a small distributed system to do something the provider already offers as a primitive.

It costs full price. Every one of those 200,000 classifications is billed at the synchronous rate, for work no human is waiting on. This is the part that never shows up in the design review and always shows up in the invoice.

None of these tickets need a real-time answer. The dashboard that reads the results refreshes once a day. You are paying a latency premium and an engineering premium for latency you do not use.

The batch rewrite

The same job as a batch is less code, not more, because the provider owns concurrency, retries, and rate limiting.

import anthropic
from anthropic.types.messages.batch_create_params import Request
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming

client = anthropic.Anthropic()

def submit_classification_batch(tickets):
    requests = [
        Request(
            custom_id=f"ticket-{ticket.id}",
            params=MessageCreateParamsNonStreaming(
                model="claude-sonnet-4-6",
                max_tokens=256,
                messages=[{
                    "role": "user",
                    "content": f"Classify sentiment and topic:\n{ticket.body}"
                }],
            ),
        )
        for ticket in tickets
    ]
    batch = client.messages.batches.create(requests=requests)
    return batch.id  # persist this

The custom_id is the load-bearing field. Results come back unordered and possibly at different times, so custom_id is how you rejoin a result to its source row. Make it something you can map back to your database, not a random UUID you forget to store.

Retrieving results is a poll, not a webhook, on both providers today. Store the batch id, check it on a schedule, stream results when it reports ended.

def collect_batch(batch_id):
    batch = client.messages.batches.retrieve(batch_id)
    if batch.processing_status != "ended":
        return None  # not done, check again later

    results = {}
    for entry in client.messages.batches.results(batch_id):
        ticket_id = entry.custom_id.removeprefix("ticket-")
        if entry.result.type == "succeeded":
            results[ticket_id] = entry.result.message.content[0].text
        else:
            # per-request failure: errored, canceled, or expired
            results[ticket_id] = None
    return results

The critical thing the naive loop got wrong: failures are per request, not per batch. A batch can report ended while individual entries are errored or expired. You must iterate every result and handle the non-succeeded cases explicitly, or you will silently drop rows and never notice until someone asks why 3 percent of tickets have no sentiment.

The OpenAI shape is the same idea with a file instead of an array. You upload a JSONL where each line has a custom_id, a method, a url, and a body, create a batch against /v1/chat/completions, poll the batch object, then download the output file.

from openai import OpenAI
import json

client = OpenAI()

def submit_openai_batch(tickets, path="batch.jsonl"):
    with open(path, "w") as f:
        for ticket in tickets:
            f.write(json.dumps({
                "custom_id": f"ticket-{ticket.id}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": {
                    "model": "gpt-4o-mini",
                    "max_tokens": 256,
                    "messages": [{
                        "role": "user",
                        "content": f"Classify sentiment and topic:\n{ticket.body}"
                    }],
                },
            }) + "\n")

    file = client.files.create(file=open(path, "rb"), purpose="batch")
    batch = client.batches.create(
        input_file_id=file.id,
        endpoint="/v1/chat/completions",
        completion_window="24h",
    )
    return batch.id

Which workloads actually qualify

The decision is not “important vs unimportant.” It is “is a human blocked on this specific response.” Run every candidate through that single question.

That last row is the one people miss. “Personalized” and “user-facing” feel like they demand real-time, but if the email goes out on a morning schedule, you have hours of slack. Generate the drafts in an overnight batch, store them, send on cron. The user never knows the difference and you paid half.

The honest disqualifiers: anything interactive, anything where the output feeds the next input in a tight loop (multi-turn agents), and anything with a hard deadline shorter than a few hours. Do not batch a workload just because it is large. Batch it because nothing downstream is blocked waiting.

Mixed workloads: batch and real-time together

Most real systems are not purely one or the other. The clean pattern is a router in front of a job store that decides synchronous vs batch per request, based on a deadline the caller declares.

                         +------------------+
   incoming request ---> |  inference router|
                         +------------------+
                          /                 \
             deadline < 5s?              deadline hours+?
                /                              \
     +------------------+              +------------------+
     |  real-time call  |              |  batch queue     |
     |  (full price)    |              |  (accumulate)    |
     +------------------+              +------------------+
              |                                 |
        immediate response            flush on size OR timer
                                                |
                                       +------------------+
                                       | provider batch   |
                                       | API (50% off)    |
                                       +------------------+
                                                |
                                       results -> job store
                                                |
                                       callback / poll consumer

The router does not guess. The caller passes an SLA hint. A chat endpoint declares “interactive”; a nightly report declares “async, due 6am.” Everything tagged async lands in an accumulator that flushes on either of two conditions: it reaches a size threshold, or a timer fires. This gives you the discount without letting a low-volume async queue sit forever.

class InferenceRouter:
    def __init__(self, max_batch=10_000, flush_seconds=900):
        self.pending = []
        self.max_batch = max_batch
        self.flush_seconds = flush_seconds

    def submit(self, request, interactive: bool):
        if interactive:
            return self._realtime(request)      # full price, instant
        self.pending.append(request)
        if len(self.pending) >= self.max_batch:
            return self._flush()                 # size trigger
        return {"status": "queued"}

    def on_timer(self):
        # called on a schedule so a small queue still flushes
        if self.pending:
            return self._flush()

Two rules keep this honest. Never let queue depth alone gate a flush, or a quiet day strands requests past their deadline; the timer is the safety net. And size your max_batch to the provider limits, not to your convenience, so you are not silently splitting or truncating.

One more lever that stacks cleanly: prompt caching works inside batch requests. If every request in a batch shares a large system prompt or a fixed reference document, mark the cache breakpoint and the shared prefix is discounted on top of the 50 percent batch discount. A support-classification batch with an 800-token instruction prefix can compound both savings.

Failure modes nobody plans for

Batch is simpler to operate than a hand-rolled concurrency layer, but it has its own sharp edges.

Partial completion. A batch reports ended with a mix of succeeded, errored, and expired entries. Treat the batch as a set of independent outcomes. Always reconcile by custom_id against your source set and re-queue the gaps. Do not assume ended means “all done.”

Expiry. If the provider cannot finish within the window, unprocessed requests come back expired, not silently retried. Your consumer must detect expired entries and resubmit them, ideally into the next batch rather than as panicked real-time calls at full price.

Non-determinism in timing. You cannot predict whether a batch finishes in 10 minutes or 10 hours. Build the product around “results appear sometime before the deadline,” never around a specific arrival time. A dashboard that assumes the 2am batch is ready by 2:30 will occasionally show stale data.

Idempotency. Because you re-queue failures, the same logical row can be submitted twice. Make your result-writing idempotent, keyed on custom_id, so a duplicate result overwrites rather than double-inserts.

Cost blindness stays possible. Batch is cheaper per token, not free. A runaway job that submits 100,000 requests with a 4,000-token context is still a real bill, just half of a bigger one. Meter batch spend the same way you meter synchronous spend.

What actually works

Batch inference is one of the highest-leverage cost moves available in 2026 and one of the least used, purely because “24-hour SLA” reads as scarier than it is. Here is the honest assessment.

What works: taking any background workload where no human is blocked and moving it to the batch endpoint. Classification, summarization, embeddings, evals, moderation backfills, and scheduled generation are all near-automatic wins. The discount is flat 50 percent, the quality is identical, and you delete concurrency and retry code instead of writing more. The freed-up synchronous rate limit is a real bonus.

What does not work: forcing interactive or tightly-coupled agent workloads into batch to chase the discount. If a user or a next-step prompt is waiting, batch is the wrong tool and you will pay for it in product quality. Batch also does not fix a fundamentally wasteful prompt; halving a bloated bill still leaves half a bloated bill.

What to actually do: audit your current LLM spend and tag every workload with one boolean, “is a human or a live loop blocked on this response.” Move everything tagged false to batch behind a simple SLA-aware router, keep the timer-based flush so nothing strands, reconcile results by custom_id, and re-queue failures instead of falling back to real-time. Then stack prompt caching on the shared prefixes. Most teams doing background LLM work at any scale will cut total spend by 30 to 45 percent with a week of plumbing and zero quality loss. That is a better return than almost any model-routing or self-hosting project you could run instead.