An Evaluation Framework for LLM Routing Quality

Start with observable failure

Before selecting an evaluator, define what a bad route does to the application. Examples include invalid JSON, missing citations, incorrect labels, unusable tool arguments, unsafe advice, style violations, or reduced task completion.

Deterministic validators should run first because they are reproducible and inexpensive. A model judge can assess dimensions that cannot be expressed as code, but its own bias and variance must be tested.

Build a representative golden set

Sample real prompts across common, difficult, and high-risk cases. Remove sensitive content or keep the dataset in an approved environment. Label expected behavior, not only one ideal prose answer.

Version the dataset and keep a held-out portion. If the same examples repeatedly shape prompts and policy, they stop measuring generalization.

Use layered evaluation

A practical pipeline checks parse and schema validity, task rules, retrieval grounding, pairwise preference, and human review in that order. Track disagreement between judges and humans instead of treating a judge score as ground truth.

Evaluate the full routed system, including classifier, cache, retries, and fallback. Testing model outputs in isolation misses the failures caused by orchestration.

• Task success and hard failure rate
• Quality delta against baseline
• Escalation and retry rate
• Latency distribution
• Realized cost per successful outcome
• Results by cohort and risk class

Turn evaluation into a control loop

Set a minimum sample count and rolling quality floor for each enforced route. When the floor is breached, roll the cohort back to monitor mode and preserve the failing samples. A route without a tested rollback is an experiment without containment.