Model orchestration: how AI systems choose and combine models

Golden rule: Do not orchestrate models because it sounds clever. Do it when different kinds of work demonstrably have different requirements, and you can measure when the cheaper path is enough, when it should hand off to a stronger path, and when the system should stop.

What it is and what it is not

Model orchestration is the operational layer that selects a model, configuration and next step for each part of a task. It may choose among a small fast model, a more expensive frontier model, a local model, or deterministic code. It enforces constraints around budget, latency, data sensitivity, output quality and failure handling.

It is not another name for a multi-agent system. A fixed workflow can use three models and remain a workflow. It is not a universal proxy that merely rewrites endpoint names. Models differ in tool use, context limits, structured output, safety behaviour and failure modes. Nor is it a reason to split one simple prompt across five models.

Start with a baseline built around one sensible model. Add orchestration only when logs reveal a specific problem: routine tasks cost too much, one task class performs poorly, some data must stay local, or an independent check is required.

A routing decision tree

A router does not need to begin as another LLM. The first version is often easier to understand as a small set of rules:

Is the task deterministic?
├─ yes → code, SQL, search or a validator without an LLM
└─ no
   Does the input contain data that must stay in a region or device?
   ├─ yes → local or region-approved model
   └─ no
      Is the outcome irreversible or high risk?
      ├─ yes → stronger model + verification + possibly a human
      └─ no
         Does the cheaper model pass evals for this task class?
         ├─ yes → cheaper model
         └─ no  → stronger model

After every step:
- schema violation or tool error → repair once, then hand off
- low verifiable confidence → retrieval, stronger model or human
- timeout or provider outage → compatible backup path
- security issue → stop rather than improvise

Routing signals should be available before calling the expensive model: request type, language, input size, data source, required tools, tenant, SLA and risk class. Model self-assessment can be one signal, but not the only evidence. A model can confidently approve its own mistake.

A practical architecture blueprint

A useful design has a small number of explicit layers:

1. Input normalisation removes unnecessary data, assigns the tenant and creates a run identifier.
2. Policy gate determines allowed providers, region, data classification, maximum spend and actions requiring approval.
3. Router chooses a path using versioned rules or a classifier. It also returns the reason for its decision.
4. Model adapter maps the internal message, tool and structured-output format to a provider API.
5. Executor runs models and tools with timeouts, step limits and an idempotency key.
6. Verifier checks the schema, citations, business rules or another machine-verifiable result.
7. Handoff controller chooses repair, escalation, another model, human review or a stop.
8. Observability and eval store records prompt version, model, routing reason, latency, cost, verification result and final state.

Domain logic should not know provider names. A compact internal contract might contain task, risk, data_class, allowed_tools, output_schema and budget. Adapters handle provider differences. The abstraction must not flatten everything to the lowest common denominator, however. If one model offers a capability that measurably improves the outcome, expose it through an explicit capability flag and maintain a backup path.

Handoffs and fallbacks without silent chaos

A handoff transfers work to another path for a substantive reason: the input proved more difficult, verification failed, or a different capability is needed. Pass structured state, not the entire raw transcript. The next model needs the original goal, verified facts, tool results, previous failures and remaining budget.

A fallback handles unavailability or a known failure mode. Its order must be defined in advance. For example: retry the same model only for a transient network error; repair output after a schema violation; use a stronger model after a failed factual check; use another provider during an outage; require a human for an irreversible action. An endless retry chain is not resilience. It is a hidden bill.

Every transition must preserve the security policy. A backup model must not receive data that the primary model was barred from receiving for locality reasons. A fallback must not turn “approval required” into “try it automatically”.

Evals and cost per success

Price per million tokens is a purchasing input, not a system metric. The useful measure is cost per successfully completed task:

total run cost + retrieval + tools + verification + human repair
─────────────────────────────────────────────────────────────────
number of tasks that passed the defined success criterion

An eval set should include real task classes, edge cases, long inputs, multiple languages and security scenarios. For each route, track success rate, latency, attempts, handoff frequency, human intervention and error types. Compare the orchestrator with a single-model baseline. A router optimised only for cost learns to fail cheaply; one optimised only for average quality sends everything to the frontier model.

Change one routing decision at a time and replay the same eval set. Monitor production separately because a change in request mix can invalidate an old routing policy. Do not treat public benchmarks as proof for your product. Performance on your tasks under your definition of success is what matters.

Security and data locality

The router is security-sensitive infrastructure. It sees the metadata used to decide where data goes. Data classification must therefore happen before provider selection and be enforced by a policy layer, not merely written in a prompt.

For every model, record approved regions, retention terms, training-data policy, supported encryption, audit features and allowed tools. Minimise or redact personal data and secrets before any model call. Logs must not recreate a leak by storing the whole prompt. Keep tool credentials outside model context and authorise each tool for the specific run.

A local model is not automatically safe, and a cloud model is not automatically unsafe. The complete flow matters: where inference runs, where telemetry goes, what remains in caches, who can access logs and whether actions can be audited.

Common failure modes

• Routing by intuition: “hard” tasks have no definition. Fix: task taxonomy, eval set and versioned policy.
• Frontier model for everything: premium quality is assumed even where it adds nothing. Fix: a cheaper baseline and cost-per-success measurement.
• Cheapest model for everything: savings disappear into retries and human repair. Fix: count the full run, not the first API call.
• Silent fallback: an incident looks like success. Fix: record the reason, route and final state, and surface degraded operation.
• Lost state during handoff: the second model repeats work or trusts an unverified claim. Fix: a structured handoff package.
• A model grades itself: the same blind spot passes twice. Fix: deterministic checks, an independent evaluator or a human according to risk.
• The abstraction owns the workflow: the product depends on a visual Agent Builder, its memory and proprietary tool definitions. If the service disappears, so does the exit path. Fix: keep state, prompts, schemas, evals and policies in a versioned format you control.
• Provider switch without evals: compatible JSON does not imply compatible behaviour. Fix: replay representative tasks before switching.

Worked example: customer request triage

A company receives support email in several languages. The system must assign a category, retrieve relevant documentation and prepare a reply draft. Contract changes and refunds require a human.

1. Code strips signatures, detects attachments and marks personal data.
2. A small local model classifies language, topic and risk into a fixed JSON schema.
3. The policy gate blocks sensitive attachments from external APIs. It permits an approved cloud provider for an ordinary query.
4. A cheaper model receives retrieved articles and produces a cited draft.
5. The verifier checks that cited documents exist, the answer makes no prohibited promise, and every product claim is supported by a source.
6. A schema violation gets one targeted repair. A factual mismatch hands the original goal, sources and verification result to a stronger model. A high-risk case goes directly to a human.
7. The system records the route, reason, total cost and whether the human accepted, edited or rejected the draft.

After several eval cycles, the team may find that the cheaper model handles routine questions safely but frequently needs repair for ambiguous complaints. The router then escalates based not on email length, but on a combination of topic, risk and retrieval result. That is evidence-based orchestration, not a hierarchy of brand names.

Sources

• Building effective agents (Anthropic) - a practical distinction between simple workflows and agentic systems, with a strong case for the simplest design that works.
• RouteLLM (LMSYS) - research on routers that choose between stronger and cheaper paths according to the query.
• FrugalGPT (Stanford University) - early work on model cascades, budgets and quality-cost optimisation.
• NIST AI Risk Management Framework - a framework for mapping, measuring and managing AI risk, including operational accountability.
• OWASP Top 10 for LLM Applications - a catalogue of attacks and operational risks that routers, tools and fallbacks must respect.
• OpenAI Evals - an open framework and examples for describing eval tasks and comparing model behaviour.

What to remember

Orchestration is not a collection of models. It is the control layer for decisions, handoffs, verification and stopping. Start with one model and a measurable baseline. Route from evidence, not brand prestige. Count cost per success, including retries and human work. Enforce security policy before the model and preserve it across fallbacks. Keep state, evals and tool contracts in a format you control. Good orchestration is boring, auditable and replaceable.