Pydantic AI Makes LLM Output Validation a First-Class Engineering Problem

If you still have a re.search(r'\{.*\}', response) somewhere in your codebase to pull JSON out of an LLM reply, that line is a liability. Everyone knows this by now. The interesting question in 2026 is not “regex versus a parser.” It is “once the raw provider APIs give you JSON mode and schema enforcement, who owns validation, retries, and the fact that you want the same code to run against Claude, GPT-4o, and Gemini?”

The provider SDKs answer: you do. Every one of those concerns ends up as hand-rolled glue in your application. Pydantic AI’s answer is that output validation is a real engineering problem deserving a real abstraction, and it hands you one that looks like ordinary typed Python. I have replaced a pile of provider-specific extraction code with it on two services this year. Here is what it actually does and where it stops.

The Naive Approach, and the Less-Naive One That Still Hurts

Start with the thing everyone writes first.

import re, json

resp = client.messages.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": f"Extract the invoice as JSON: {text}"}],
    max_tokens=1024,
)
raw = resp.content[0].text
data = json.loads(re.search(r'\{.*\}', raw, re.DOTALL).group(0))

This breaks on prose preambles, inline backticks, nested braces, string-typed numbers, and model version bumps. Granted. So you graduate to the provider’s structured-output API, which is genuinely better:

from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_id: str
    total: float
    currency: str

resp = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[{"role": "user", "content": f"Extract invoice: {text}"}],
    response_format=Invoice,
)
invoice = resp.choices[0].message.parsed

The syntax problem is gone and the shape is guaranteed. But look at everything this snippet does not handle, which you will write by hand:

• The model filled total with a plausible but wrong number. Schema compliance is not semantic correctness. You want a validator that rejects a total that does not match the line items, and you want the model to try again with the error as feedback.
• You want to run the exact same extraction against Claude for one tenant and Gemini for another. This code is welded to OpenAI’s SDK surface.
• When the model does not have enough information, constrained decoding forces it to fabricate a value rather than fail. You need an explicit failure path.
• The call has no retry policy, no usage accounting, no place to inject per-request dependencies like a database handle the model’s tools can reach.

Each of these is a small amount of code. Multiplied across every call site and every provider, it is a framework you are writing by accident, badly, with no tests. That is the gap Pydantic AI fills.

The Agent as the Unit of Work

Pydantic AI’s core object is an Agent. It is not “a chatbot.” It is a typed binding of a model, a system prompt, an output type, and a set of tools. You declare what you want out and you get that type back or an exception, never a dict you have to interrogate.

from pydantic import BaseModel
from pydantic_ai import Agent

class Invoice(BaseModel):
    invoice_id: str
    total: float
    currency: str
    line_items: list[str]

agent = Agent(
    "anthropic:claude-sonnet-4-6",
    output_type=Invoice,
    system_prompt="Extract invoice fields from the text. Be precise about numbers.",
)

result = agent.run_sync("Invoice INV-204, two items, total 142.50 USD ...")
invoice: Invoice = result.output   # a real Invoice instance, validated
print(result.usage())              # token accounting, for free

result.output is an Invoice, statically typed, so your IDE and mypy know invoice.total is a float before the program ever runs. That is the “first-class” part: the LLM boundary is now a typed function call, and the type system follows the data through the rest of your program.

Validation Retries: the Feature That Earns Its Keep

Structured-output APIs guarantee the shape. They cannot guarantee the content makes sense. Pydantic AI lets you raise ModelRetry from inside a validator, which sends the error text back to the model as a correction turn and asks it to fix its own output. This is the single feature that moves validation from “defensive parsing” to “enforced invariant.”

from pydantic_ai import Agent, ModelRetry
from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_id: str
    total: float
    line_item_total: float

agent = Agent(
    "anthropic:claude-sonnet-4-6",
    output_type=Invoice,
    retries=2,
)

@agent.output_validator
def totals_must_reconcile(inv: Invoice) -> Invoice:
    if abs(inv.total - inv.line_item_total) > 0.01:
        raise ModelRetry(
            f"total {inv.total} does not match line_item_total "
            f"{inv.line_item_total}. Recompute from the line items."
        )
    return inv

When the model returns an invoice whose totals do not reconcile, the validator raises, Pydantic AI feeds the message back, and the model tries again, up to retries times. If it still fails, you get a UnexpectedModelBehavior exception, not a silently wrong invoice. Compare that to the raw API version, where reconciling totals means an if block, a manually assembled follow-up message, a loop counter, and a hope that you handled the loop termination correctly.

For the “model does not have enough information” case, you do not fake success. You make the output type a union and let the model pick the honest branch:

from typing import Union

class ExtractionFailed(BaseModel):
    reason: str

agent = Agent(
    "anthropic:claude-sonnet-4-6",
    output_type=Union[Invoice, ExtractionFailed],
)

Now the model has a legitimate way to say “I cannot find a total,” and your downstream code pattern-matches on the type instead of inspecting a boolean flag you hoped the model set correctly.

Dependency Injection for Tools

The part that genuinely separates Pydantic AI from “a nicer response_format” is how tools get their dependencies. In most frameworks, a tool function is a closure over globals or a bag of partially applied arguments. Pydantic AI uses a typed dependency container, RunContext[DepsType], that you pass in per run. Tools receive it as their first argument.

from dataclasses import dataclass
from pydantic_ai import Agent, RunContext

@dataclass
class Deps:
    db: "DatabasePool"
    tenant_id: str

support_agent = Agent(
    "anthropic:claude-sonnet-4-6",
    deps_type=Deps,
    system_prompt="Answer customer questions using the account tools.",
)

@support_agent.tool
async def get_balance(ctx: RunContext[Deps]) -> float:
    """Return the current account balance for this tenant."""
    row = await ctx.deps.db.fetchrow(
        "SELECT balance FROM accounts WHERE tenant = $1",
        ctx.deps.tenant_id,
    )
    return row["balance"]

# per request, you inject the real dependencies
result = await support_agent.run(
    "How much do I owe?",
    deps=Deps(db=pool, tenant_id="acme"),
)

Three things fall out of this that matter in production. The tool’s db and tenant_id are scoped to one request, so there is no global state and concurrent requests cannot leak each other’s tenants. The tool signature is typed, so Pydantic AI derives the JSON schema the model sees directly from your Python types and the docstring, keeping the model’s view and your code in sync automatically. And testing becomes ordinary: you pass a fake Deps with an in-memory database and the agent never knows the difference. No monkeypatching a module global.

async def test_balance_tool():
    deps = Deps(db=FakeDb({"acme": 42.0}), tenant_id="acme")
    with support_agent.override(model=TestModel()):
        result = await support_agent.run("balance?", deps=deps)
    assert "42" in result.output

TestModel and override let you exercise tool wiring without spending a token, which is the difference between tests you run on every commit and tests you run never.

Model-Agnostic by Construction

The model is a string argument, or an object, and nothing else in your agent changes when you swap it. The same Agent, the same tools, the same validators run against any supported provider.

agent = Agent(output_type=Invoice)

agent.run_sync(text, model="anthropic:claude-sonnet-4-6")
agent.run_sync(text, model="openai:gpt-4o")
agent.run_sync(text, model="google-gla:gemini-2.0-flash")

Under the hood Pydantic AI maps your output type and tools onto each provider’s native mechanism: Anthropic tool use with forced tool_choice, OpenAI structured outputs, Gemini’s schema-constrained generation. You write the schema once. The framework translates. This is what lets you route cheap classification to a small model and hard extraction to a frontier model without maintaining three copies of the parsing code, and it is what lets you survive a provider outage by flipping a string.

How It Compares

          raw regex          provider structured API        Pydantic AI
          ---------          ---------------------          -----------
shape     you parse          guaranteed by API              guaranteed, typed
validate  manual             manual                         validators + ModelRetry
retries   your loop          your loop                      built in, bounded
tools     n/a                per-provider wiring            typed DI via RunContext
provider  locked             locked to one SDK              one line to swap
testing   mock HTTP          mock the SDK                   TestModel + override

Against the two frameworks people actually weigh it against:

LangChain gives you more prebuilt pieces and more integrations. It also gives you more abstraction to learn and debug, and its typing was never the center of the design. Pydantic AI is deliberately small: if you are comfortable writing Pydantic models and type hints, there is almost no new vocabulary. That narrowness is the point. It does one layer, the output and tool boundary, and it does it as typed Python rather than as a graph DSL.

Where This Sits in a Real System

HTTP request
     |
     v
[Route handler] -- builds Deps(db, tenant_id, caches)
     |
     v
[Pydantic AI Agent] --- model = string, swappable per tenant/task
     |                     |
     |                     +--> [tool: db lookup]  (typed RunContext)
     |                     +--> [tool: pricing API] (typed RunContext)
     |
     v
[Output type]  Invoice | ExtractionFailed
     |
     |-- output_validator raises ModelRetry --> model corrects, bounded by retries
     |
     v
[Typed result] -> business logic -> persistence

The agent is a boundary, not the whole application. Everything above it is ordinary web code and everything below it is ordinary typed Python. The framework’s job is to make the crossing between stochastic text and deterministic types boring, which is exactly what you want from infrastructure.

The Honest Assessment

What works: for extraction, classification, and tool-using agents where you know the output shape in advance, Pydantic AI removes an entire category of glue code and makes the LLM boundary type-checked. Validation retries turn “parse and pray” into an enforced contract. Dependency injection makes tools testable without mocking the world. Model-agnosticism is real and it is one line, which pays for itself the first time a provider has a bad day.

What does not: it is a thin layer, not a platform. It does not give you a vector store, a document loader zoo, or a prebuilt RAG pipeline, and if you want those you are assembling them yourself or reaching for a heavier framework. Validation retries cost extra model round trips, so a validator that fires often on expensive models shows up on your bill and your latency graphs. And none of this touches semantic correctness: a reconciling total can still be the wrong total if the source was ambiguous, so confidence fields and human review queues remain your job, not the framework’s.

What to actually do: if your LLM calls have a known output shape, adopt it for those call sites and leave your free-form generation alone. Define a failure branch in every output union from day one. Put real invariants in output_validator rather than in downstream if checks, so a violation retries instead of propagating. Keep the model as a string you can configure per tenant. Treat the agent as a typed function at the edge of your system, and let the rest of your code go back to being ordinary Python that happens to know its types.