CI/CD for AI Applications Is Different - What Changes When Your Logic Lives in a Prompt

A git commit does not capture what your AI application actually does. You can freeze every line of Python, tag the release, and deploy the exact same container to two regions - and get materially different behavior because one region is pinned to a model snapshot that got deprecated and silently rerouted, and the other is reading a retrieval index that was rebuilt yesterday. Your test suite passed. Your code did not change. Your product got worse anyway.

This is the core problem with CI/CD for LLM applications: the thing that determines behavior is not just the code. It is the code, plus a prompt, plus a model version, plus a retrieval index, plus a set of tool definitions. Traditional CI/CD assumes the build artifact is deterministic and self-contained. For an AI app, the artifact is a composite of things that live in different places, version on different schedules, and fail in ways a unit test never sees.

Most teams paste their LLM deploy process on top of their normal one and wonder why regressions keep slipping through. The fix is to treat the full behavior surface as the artifact, version every piece of it, and gate promotion on evals instead of green checkmarks. This post walks from the naive setup most teams have to the pipeline you actually want.

What Is Actually in Your Build Artifact

Start by being honest about the inputs to a single LLM response. For a typical RAG or agent app, the behavior is a function of at least five things:

Response = f(
    application_code,     # parsing, orchestration, post-processing
    prompt_template,      # system prompt, few-shot examples, output schema
    model_version,        # e.g. claude-sonnet-5, or a pinned snapshot
    retrieval_index,      # embeddings + the documents they point at
    tool_definitions      # function schemas the model can call
)

Traditional CI/CD versions exactly one of these: application_code. The other four are usually a config string, an environment variable, a table in a database, and a folder in object storage. None of them are tied to the git SHA you deployed. So when behavior changes, you have no single version number to roll back to.

The mental model that fixes this: a release is a pinned tuple of all five, not a git commit. Everything downstream - versioning, eval gates, canaries, rollback - follows from taking that seriously.

The dangerous rows are the ones that change without a code deploy. A product manager editing the system prompt in an admin panel, or a nightly job re-embedding your knowledge base, both change production behavior with zero CI involvement. That is the gap.

The Naive Pipeline and Where It Breaks

Here is what most teams ship first. It is not wrong, it is just incomplete.

# .github/workflows/deploy.yml - the naive version
name: Deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/            # unit tests on parsing logic
      - run: docker build -t app .
      - run: ./deploy.sh production

This pipeline has three blind spots, and each one is a real outage waiting to happen.

Blind spot 1: the prompt is not in the artifact. If the prompt lives in a database or an admin UI, this deploy does not touch it and cannot roll it back. Someone changes the system prompt at 3pm, output quality drops, and the “last known good deploy” you roll back to still points at the bad prompt. Rollback does nothing.

Blind spot 2: the model version is a moving target. If your code calls a model alias rather than a pinned snapshot, the provider can change the underlying weights under you. Your identical container now behaves differently. There is no commit, no PR, no alert - the same input just started returning different output.

Blind spot 3: pytest cannot see quality regressions. Unit tests check that your JSON parser works and your retry logic fires. They cannot tell you the new prompt made summaries 20% more likely to hallucinate a date. The tests are green and the product is worse.

The naive pipeline optimizes for “the code runs.” What you need is a pipeline that optimizes for “the behavior is at least as good as production.”

Step 1 - Version the Prompt Like Code

The first fix is the cheapest and highest leverage: get the prompt out of the database and into git. A prompt is logic. It deserves the same review, history, and rollback story as any other logic.

# prompts/summarize/v7.py
PROMPT = {
    "id": "summarize",
    "version": 7,
    "model": "claude-sonnet-5-20260115",   # pinned snapshot, never an alias
    "template": """You are summarizing a support ticket for an engineer.
Output a JSON object with keys: summary (<= 40 words), severity, next_action.
Do not invent facts not present in the ticket.

Ticket:
{ticket}""",
    "params": {"temperature": 0.2, "max_tokens": 300},
    "created": "2026-07-20",
    "changelog": "Tightened summary length, added severity field.",
}

Three rules make this work in practice.

Pin the model snapshot inside the prompt version. The prompt and the model it was tuned against are not separable. A prompt that works on claude-sonnet-5-20260115 may drift on a later snapshot. Bind them together so a “prompt version” is really a prompt-plus-model version. Never call a floating alias in production - aliases are for experimentation, not releases.

Make the version immutable and additive. Do not edit v7 in place. Ship v8. Every version that ever ran in production stays in the repo so you can diff behavior and roll back to an exact known state. This is the single feature that makes rollback actually work.

Load prompts by explicit version at deploy time. The running service should never “grab the latest prompt.” It should load the version baked into this release.

class PromptRegistry:
    def __init__(self, prompts: dict[tuple[str, int], dict]):
        self._prompts = prompts  # keyed by (id, version)

    def get(self, prompt_id: str, version: int) -> dict:
        key = (prompt_id, version)
        if key not in self._prompts:
            raise ValueError(f"Unknown prompt {prompt_id} v{version}")
        return self._prompts[key]

# Release config pins the exact version - this is part of the artifact
RELEASE = {
    "summarize": 7,
    "classify_intent": 3,
    "extract_entities": 5,
}

Now “what prompt was running when this ticket came in” has an exact answer, and rolling the prompt back is a one-line config change plus a redeploy, not an archaeology project.

Step 2 - Make Eval Gates Block Promotion

Once the prompt is versioned, CI can actually reason about it. The gate you want is not “do the tests pass” but “is this release at least as good as what is in production.” That is an eval gate, and it belongs in the pipeline the same way a unit test does.

I will keep this short because evals deserve their own treatment, but the CI shape is what matters here:

# eval gate as a required check
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run eval suite against candidate release
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python evals/run.py \
            --release candidate \
            --dataset golden_v4.jsonl \
            --baseline production \
            --fail-on-regression 0.03

The two decisions that make an eval gate useful:

• Compare to production, not to an absolute number. A fixed “must score 85%” threshold either blocks everything when your baseline is 82%, or waves through a 3-point regression when your baseline is 90%. The meaningful question is the delta against what is live right now.
• Gate on the paths that change behavior. Run the expensive eval suite when prompts/, the model pin, or the retrieval config change - not on every commit to a doc file. Evals cost real money and minutes; spend them where behavior can actually move.

The important structural point: because the model version and prompt version are now part of the artifact, the eval gate can evaluate the exact tuple that will ship. A model upgrade is no longer invisible to CI - it is a diff to the pinned snapshot, and it triggers the same eval gate as a prompt change. That is the whole payoff of Step 1.

Step 3 - The Retrieval Index Is Also a Release

The piece teams forget entirely is the retrieval index. If you run RAG, the documents and their embeddings are as much a part of behavior as the prompt. Two failure modes bite here.

The first is silent index drift. A nightly job re-embeds your knowledge base and rebuilds the index in place. Nobody deployed anything, but retrieval results shifted, and the model now grounds answers on different chunks. The second is the embedding model mismatch: you upgrade the embedding model for new documents but do not re-embed the old ones, so half your index lives in a different vector space and similarity search quietly returns garbage.

The fix is to treat the index as an immutable, versioned artifact and pin it in the release, exactly like the prompt.

INDEX_MANIFEST = {
    "index_id": "kb-2026-07-28",
    "embedding_model": "voyage-3.5",     # pinned, same for every doc in this index
    "chunk_strategy": "recursive-512-overlap-64",
    "doc_snapshot": "s3://kb/snapshots/2026-07-28/",
    "doc_count": 48213,
    "built": "2026-07-28T02:00:00Z",
}

# The release pins the index by id, never "latest"
RELEASE_INDEX = "kb-2026-07-28"

Build indexes with a date-stamped id, never overwrite the previous one, and keep the last few around so you can roll back retrieval independently of the prompt and code. When you change the embedding model or chunking, you build a brand new index and re-embed everything against the new model - a partial re-embed is the mismatch bug, not an optimization.

Here is the full artifact, pinned end to end:

              Release v312 (a pinned tuple, not a git SHA)
              +---------------------------------------------+
   git SHA -> |  application_code   9f3a1c                  |
              |  prompts            summarize v7, intent v3 |
              |  model              claude-sonnet-5-20260115|
              |  index              kb-2026-07-28           |
              |  tools              lookup v2, action v2    |
              +---------------------------------------------+
                                  |
                          eval gate (vs production)
                                  |
                          canary  ->  full rollout

Every arrow into the box is a version you can roll back independently. That is the property the naive pipeline lacked.

Step 4 - Canary Rollouts for Model and Prompt Changes

Even a green eval gate is not proof. Your golden dataset is a sample; production is the full distribution, and it contains inputs you never thought to put in the dataset. So you do not flip the whole fleet to a new release at once. You canary it: route a small slice of live traffic to the new tuple, watch the metrics that matter, and promote only if it holds.

The mechanism is a weighted router keyed on the release, not the code deploy.

class ReleaseRouter:
    def __init__(self, stable: str, canary: str | None, canary_pct: float):
        self.stable = stable          # e.g. "v311"
        self.canary = canary          # e.g. "v312", or None
        self.canary_pct = canary_pct  # 0.0 - 1.0

    def pick(self, request_id: str) -> str:
        if not self.canary:
            return self.stable
        # Deterministic per request so a user has a consistent experience
        bucket = stable_hash(request_id) % 100
        return self.canary if bucket < self.canary_pct * 100 else self.stable

What makes an AI canary different from a normal one is what you measure. A code canary watches error rate and latency. Those still matter, but they will stay flat while quality quietly craters - a worse prompt returns HTTP 200 with a bad answer. You have to watch behavioral signals too.

A practical rollout ladder: 5% for an hour, 25% for a few hours, 50% overnight, then 100%. At each step compare the canary’s behavioral metrics against the stable release on the same traffic window, so you are controlling for time-of-day and input mix. If any tracked signal degrades past its threshold, the router flips canary_pct to zero. No redeploy, no rebuild - rollback is a config write, which is the whole reason to route on release instead of on deploy.

def evaluate_canary(stable_metrics, canary_metrics) -> str:
    if canary_metrics["schema_error_rate"] > stable_metrics["schema_error_rate"] + 0.01:
        return "rollback"
    if canary_metrics["judge_score"] < stable_metrics["judge_score"] - 0.03:
        return "rollback"
    if canary_metrics["p95_cost"] > stable_metrics["p95_cost"] * 1.5:
        return "hold"   # cost regression - pause, do not promote
    return "promote"

Note that cost is a first-class rollout signal here. A prompt or model change that improves quality but doubles token usage is not automatically a win, and the router should be able to hold on economics alone.

Handling Provider-Forced Model Changes

One scenario deserves its own callout because it is unique to AI apps and it will happen to you: the provider deprecates the snapshot you pinned. You do not get to keep claude-sonnet-5-20260115 forever. When a deprecation notice lands, treat the successor snapshot as a new release candidate and run it through the exact same machinery - eval gate against production, then canary. Do not let the provider’s timeline force you to skip the gate.

The reason pinning still wins, even though snapshots expire, is that pinning converts an invisible, un-dated behavior change into a scheduled, testable migration. Without a pin, the change arrives silently and you debug it as a mystery regression. With a pin, it arrives as a deprecation date on your calendar and a candidate release you can evaluate before it touches users. You trade “surprise outage” for “planned upgrade,” which is the entire point.

The Honest Assessment

What works today: versioning the prompt in git with a pinned model snapshot, an eval gate that compares candidate to production, and a release router that lets you canary and instant-rollback by config. None of this needs an exotic platform - it is a prompt registry, a golden dataset, a weighted router, and the discipline to treat the five-part tuple as the artifact. A small team can build the whole thing in a week and it will pay for itself the first time a bad prompt does not reach 100% of users.

What does not work: relying on your git SHA to describe behavior, calling model aliases in production, rebuilding retrieval indexes in place, and trusting unit tests to catch quality regressions. Each of these is a silent-failure generator. They feel fine right up until the incident.

What to actually do, in order: pin the model snapshot today (it is one line and removes a whole class of mystery regressions), move prompts into version control this week, add an eval gate that blocks on regression versus production, and only then build the canary router. Do not start with the fancy rollout system. Start by making sure a rollback actually rolls back the thing that changed - which, for an AI app, is almost never just the code.