They Could Swap AI Models in a Day. That Was the Problem.

If you had to leave your AI provider tomorrow, could your business still operate?

Responsible companies have had an answer to that for databases, cloud infrastructure, networking, and backups for decades. Almost nobody has one for AI.

I spent this week running that exercise with a product team. Their tool is an internal reviewer: an LLM reads a piece of work, scores it against a quality rubric, and flags problems for a person to approve or dismiss. Not customer-facing, with a human in the loop above every result.

They came in expecting a bad grade. Their architecture is better than most teams I see. And the review still found something that should worry them, in the exact place they felt safest.

This team did not arrive at portability as a theory. They had been burned.

The platform their whole product sits on top of is run by a vendor who declined to provide an API. No SLA either. So they work at risk, permanently, with no credible exit and no leverage in any conversation about price or roadmap.

That experience is why the LLM layer got built the way it did. They had already paid for that mistake once and were not going to pay for it twice.

So on paper they were in good shape. Which is where it gets interesting.

Four things, and any one of them puts you ahead of the average team.

Every LLM call goes through an abstraction layer. No vendor SDK called directly from application code.

The model is configuration, not code. Swapping means inserting a row with the model string, its sampling settings, and its price. No deploy.

Structured output is schema-enforced at runtime. Every response validated against a JSON Schema, with retry logic and an error ledger that tracks provider failures separately from malformed responses.

Half the checks don't touch the model at all. A large share of what the tool flags is caught by deterministic rules, not by the LLM. That's plain code. A model swap cannot affect it, by construction, and it is the reason the model-facing surface is narrow enough for portability to be tractable here.

They also had a gold-eval harness, meaning a test rig that scores any prompt against expert-labeled examples, and they had already used it to compare three vendors.

And they had swapped models twice already. Both times, no incident.

Every one of those is a genuine engineering achievement. Now here's the part that turned the review.

Nothing gated either of those swaps.

No quality bar to clear. No written record of what changed or why. No defined rollback. Both swaps went fine, and nobody could tell you whether that was because the new model was good or because they got lucky.

Read the two uneventful swaps again with that in mind. "It has worked so far" is not evidence. It's untested luck wearing evidence's clothes, and it reads as proof right up until the swap where it isn't.

Which is why the six questions matter more than the architecture diagram does.

Here is where they actually landed. I'd encourage you to answer these for your own system before reading their verdicts, because the pattern in the answers is the whole point.

Could we switch models in a day? YES. Mechanically. A config row, not a code change.

Would our prompts still work? PARTIAL. The prompts were tuned against one vendor's models. Whether they port is an empirical question nobody has run. Cost to find out: an afternoon.

Would our tools still work? YES. The deterministic checks are model-independent.

Would our structured outputs still work? STRONG YES. Schema validation is enforced on every call and the pass rate is directly measurable.

Have we measured quality? PARTIAL. The instrument exists. The gold set holds one labeled example, and no pass or fail thresholds are written down anywhere.

Have we tested real production traffic? NO. No shadow mode running a candidate silently against live requests.

Look at the shape of that. The three yes answers are all architecture. The three weak answers are all measurement.

Question five is the one everything else hangs on. Without it, the other five still leave a swap as a leap of faith. With it, a swap becomes a scored decision. That is the entire difference between "we could switch" and "we can switch safely, provably, and again next quarter."

The fix is not more architecture. They have enough architecture. The fix is a gate and a record.

Feed the gold set. Ten to twenty expert-labeled examples covering the error categories that matter, with clean examples labeled too so false positives get measured instead of guessed at. This needs QA's time, not an engineer's, and it is the single highest-value action on the list.

Write the thresholds down before you need them. Theirs: no error category worse than five points of recall against the incumbent, false-positive rate no worse, overall F1 equal or better, schema-pass rate at or above, cost within the budget envelope, latency inside the response time users already expect. The specific numbers matter less than the fact that they exist on paper before a swap instead of getting invented during one.

Pin the model identity. This one surprised them. Cloud inference profiles are routing aliases, and an alias can roll to a new snapshot silently. If you only log the alias, you cannot answer "which model actually produced this result" six months later. So log the resolved model ID the provider returns, on every run.

Run the drill on a cadence. Quarterly, against at least one non-incumbent model, even when no swap is planned. That converts "we think we're portable" into standing evidence and keeps the muscle warm.

None of that is exotic. All of it is the kind of thing that never gets prioritized until the day it is urgent, which is the worst day to start.

They wrote these down and accepted them, instead of pretending the policy was free. The third one is the one people underestimate.

Most teams reading this will be worse off than the team I described, and the gap is not where they expect. They will assume they need to go build an abstraction layer.

Maybe. But the harder question is the one this team could not answer despite having built everything correctly.

If the honest answer is "we'd look at the output and see how it feels," you have the same gap they do, and you probably have less architecture underneath it.

True portability resides in architecture and planning, not in how you purchase. The planning half is the half everyone skips, because the architecture half is the one that feels like engineering.

Chris