AI Gateway RFP Checklist: 40 Questions for Vendors

Compatibility and architecture

Ask which endpoints, streaming modes, tools, structured outputs, modalities, and provider-specific fields are supported. Request contract-test results and a documented behavior for unsupported fields.

Understand where traffic is processed, how regions are chosen, what happens during vendor outage, and whether a pass-through path exists.

• Which SDKs and API versions are tested?
• How are timeouts, retries, and cancellation coordinated?
• Can we pin providers and models per route?
• What deployment and data-residency options exist?
• How do we export policy and request records?

Routing, caching, and quality

Ask how the router is evaluated on customer traffic, how prices and capabilities are versioned, and how a decision is explained. For caches, require tenant and policy namespace details, TTL and invalidation controls, and false-hit evaluation.

A vendor should distinguish modeled opportunity from realized savings and describe the cost of classifiers, judges, retries, and embeddings.

• Can routing begin in shadow mode?
• What hard constraints run before cost optimization?
• How are quality floors and rollback defined?
• Are exact, semantic, and provider prompt caches distinguished?
• Can sensitive routes bypass all content storage?

FinOps and governance

Require organization, team, app, environment, and route allocation. Test a budget threshold and temporary approval. Ask how provider invoices, credits, and off-gateway usage reconcile with request estimates.

• What money precision is stored?
• Can baseline model and savings logic be audited?
• Are showback and chargeback exports available?
• How is unattributed spend represented?
• Which roles can view prompts, change routes, and approve budgets?

Commercial and exit terms

Model the vendor fee at current and expected volume. Ask about request, token, seat, retention, support, and egress limits. Confirm ownership and deletion of logs, embeddings, evaluations, and derived routing data.

The proof of concept should use representative traffic, defined acceptance criteria, and an exit test. A gateway is critical infrastructure; migration ability is part of reliability.