Token Metering Architecture for SaaS: Capture, Attribute, Reconcile

Capture what the provider reports, not what you estimate

OpenAI's token counting guide describes an endpoint, POST /v1/responses/input_tokens, that returns the input tokens a request will use before you send it, and it says the count includes formatting tokens for message roles and boundaries that you might not see when tokenizing locally. That is the right tool for budget checks and routing. It is a forecast, and a ledger built on forecasts will drift.

The same guide is explicit about output. Reported output token usage includes all tokens the model generates, not only the visible text, and the max_output_tokens and max_completion_tokens limits count non-visible tokens too. So the number to meter is the usage object on the response, written as it arrives. Anthropic's Usage and Cost API page makes the same distinction from the other side: it lists precise token counts as the reason to use the report instead of relying solely on response token counting, which tells you to keep both and compare them.

• Use pre-request counts for limits and routing decisions.
• Write the provider-reported usage on the response as the billing-grade record.
• Store token categories separately (uncached input, cached input, cache creation, output) because Anthropic reports them as distinct measures and they price differently.

Design the usage event

LiteLLM's spend log is a useful reference for what a minimal row holds. Its documented entry carries a hashed API key, the owning user and team, request tags, the end user, the model group, the spend, total, prompt and completion tokens, and metadata that records attempted fallbacks and the originally requested model group. The proxy also returns the calculated cost in an x-litellm-response-cost response header.

Borrow that shape and add three fields of your own. A request identifier that is unique per attempt, so a retry is a second row rather than an overwrite. A price version, so a row can be recomputed after a rate change. And a status, so a failed or cut-off call is visible instead of missing.

• Identity: tenant_id, user_id, feature or route name, assigned by your server.
• Call: request_id, attempt number, provider, requested model, served model, status.
• Usage: uncached input, cached input, cache write, output, and any per-call tool charges.
• Money: computed cost, price version, currency.

Assign the tenant on the server

LiteLLM's documentation carries a warning worth copying into your own design review. End users can supply the user parameter in a request body, and doing so increments the spend reported for that self-declared user rather than for the user who owns the key. The docs conclude that if you need to track user spend you must set the user on the key when you create it and use keys issued for that user every time you call on their behalf.

The general rule is that any identity field a client can set is a label, not a fact. Resolve the tenant from your authenticated session, write it into the event, and treat a client-supplied value as optional metadata at most. This matters more once you bill on the data, since a spoofable tenant is a billing dispute waiting to happen.

Reconcile against the provider

Your ledger can be wrong in ways it cannot detect by itself. Streams can end before usage is read, retries can go unrecorded, and a model can be missing from your price table. LiteLLM documents the last case directly: a request that carries usage but prices to $0 on a model with a non-zero rate is still logged with spend 0, and it flags the gap with a warning and a Prometheus counter named litellm_zero_cost_requests_total.

For the provider side, Anthropic's usage endpoint supports 1m, 1h and 1d buckets, can group by API key, workspace, model and service tier, and says data typically appears within 5 minutes of a request completing, with polling supported once per minute. Its cost endpoint is daily only. Two attribution details matter for reconciliation: usage in the default workspace has a null workspace_id, and playground usage has a null api_key_id even when you group by that dimension.

Run reconciliation as a scheduled job. Sum your ledger by hour, model and key, pull the provider report for the same window, and alert when the gap passes a threshold you choose.

A worked reconciliation (hypothetical numbers)

Suppose your ledger shows 1,000,000 input tokens and 200,000 output tokens for one model on one day, and the provider report shows 1,030,000 and 205,000. The input gap is 30,000 tokens, or 3.0 percent of your ledger. The output gap is 5,000 tokens, or 2.5 percent. These figures are placeholders, not measurements.

If your threshold is 2 percent, both lines alert. The next step is diagnosis by bucket. Compare hourly: a gap concentrated in a few hours points to an incident such as a retry storm or a deploy that dropped a log write, while a steady gap of a few percent in every hour points to a systematic miss such as streams where you never read the final usage, or traffic from a key that bypasses your wrapper.

Because the provider report is the invoice-side source, treat it as the authority and the ledger as the attribution layer. The goal is not a perfect match. It is a known, bounded gap and an alert when it widens.

Where metering usually breaks

LiteLLM's docs add two operational limits that apply to any SQL-backed ledger. Its legacy spend logs endpoint truncates to the 10,000 most recent matching rows, and the paginated v2 endpoint caps the advertised total at 10,000, with a recommendation to chunk large exports into smaller time windows. Its daily activity counts also record each upstream attempt separately, so request counts and billable requests are not the same number.

Plan for these before the table is large: partition by day, export in windows, and decide up front whether a retry is one billable request or two.

• Write the event before you return the response, or queue it durably, so a crash does not lose it.
• Alert on zero-cost rows for models that have a rate.
• Version your price table and keep the version on each row.
• Keep provider invoices as the final reconciliation source.