Retries can multiply across layers
An SDK, application, gateway, and provider client can each retry the same request. Without coordination, a brief outage creates duplicate calls, higher latency, and an unexpected bill.
Choose one layer to own retries, propagate deadlines, and cap total attempts. Use idempotency where the provider and operation support it.
Fail over by capability
A fallback must support the required context, tools, structured output, modalities, region, and safety policy. A nominally stronger model is not a valid fallback if its API behavior breaks the application.
Maintain tested fallback groups by route family. Re-run contract tests when providers or model versions change.
Use circuit breakers and health state
Stop sending test requests to a provider that is clearly failing. A circuit breaker opens after a defined failure threshold, waits, and probes recovery in a controlled state. Keep health scoped by provider, region, and endpoint.
Distinguish rate limits, authentication errors, content rejections, timeouts, and provider failures; they require different actions.
- End-to-end deadline
- Maximum provider attempts
- Capability-compatible fallback set
- Circuit state and cooldown
- Idempotency and cancellation
- Actual billed cost for every attempt
Price reliability honestly
The effective cost of a route includes failed attempts and fallback calls. Report nominal model cost separately from realized request cost. Reliability improvements that reduce duplicates can be as valuable as model downgrades.
Frequently asked questions
How many retries should an LLM request have?
Use the minimum justified by failure data and the user deadline. Coordinate across layers so the total, not each component, is bounded.
Should a rate limit trigger model substitution?
It may trigger a compatible provider or model fallback, but the substitute must meet route capability and policy requirements.
Do failed calls count in AI spend?
If the provider bills them, yes. The request ledger should retain all attempts and invoice reconciliation should resolve differences.
Sources and further reading
FrugalAI uses primary documentation and published research where possible. Product capabilities and prices can change; verify vendor details before procurement or production changes.