Tokens are an input unit
Token prices help compare model calls, but two systems using the same number of tokens can create different value. One may resolve a support case; the other may generate an answer that requires escalation and rework.
Choose an outcome the application already measures and define success consistently. Avoid vanity outcomes such as raw generations when acceptance or task completion is available.
Build the full numerator
Include model and embedding calls, retries, judge calls, cache and gateway infrastructure, retrieval services, and direct human review attributable to the workflow. Decide how shared platform cost is allocated and publish the rule.
Use request and trace identifiers to connect gateway spend with application outcomes. Late outcomes may require a window or cohort model rather than immediate attribution.
Segment before optimizing
Average cost per outcome can hide difficult languages, customer tiers, document types, or product versions. Segment by the same cohorts used in quality evaluation.
A cheaper model that lowers first-pass acceptance can raise total cost. Conversely, a more expensive model may improve unit economics if it prevents retries and human escalation.
- Cost per resolved support ticket
- Cost per accepted marketing draft
- Cost per correctly extracted document
- Cost per successful agent task
- Cost per active user served
- Gross margin after AI variable cost
Use the metric for decisions
Compare routing policies, prompt versions, cache eligibility, and model catalogs on both cost per successful outcome and quality. The purpose is to choose a better operating point, not to force every workload to the lowest token rate.
Frequently asked questions
What if outcomes are subjective?
Define a proxy such as accepted without revision, human score above threshold, or downstream completion. Document limitations and improve it over time.
Should fixed engineering cost be included?
For product margin and build-versus-buy decisions, yes. For request-time routing, focus first on marginal and directly attributable cost.
Can cost per outcome be real time?
Some outcomes are immediate; others arrive later. Use provisional request cost and update the cohort when the outcome is known.
Sources and further reading
FrugalAI uses primary documentation and published research where possible. Product capabilities and prices can change; verify vendor details before procurement or production changes.