Do You Need an LLM Gateway or Just Cost Monitoring?

What monitoring can do, and where it stops

Provider usage and cost APIs are the lowest-effort form of monitoring. Anthropic's Usage and Cost Admin API needs an Admin API key and no change to the application. Its usage endpoint supports 1 minute, 1 hour, and 1 day buckets and can filter and group by API key, workspace, model, service tier, context window, data residency, and speed. Anthropic says data typically appears within 5 minutes of a request finishing and that the API supports polling once per minute.

The cost endpoint is coarser. Anthropic documents it as daily granularity only, grouped by workspace or description. So token usage can drive a near-real-time alert, while dollar cost is a daily view. In both cases the data describes requests that already ran. Monitoring can tell you at 2:05 that a loop started at 2:00. It has no way to refuse the request at 2:06.

The other limit is attribution. The dimensions are the ones the provider knows. If one API key serves every customer, no provider report will split them.

What a gateway adds, and what it costs you

A gateway changes two things. It can act before the request leaves, and it can attach a dimension the provider lacks. Solo.io describes the pattern for its agentgateway reference design as four steps: validate the caller and overwrite trusted identity headers, reserve estimated budget before the request leaves the gateway, settle the reservation using provider-reported token usage, then alert and forecast from real spend. Its example rejects the next provider request once a $500 allocation for a runaway evaluation job is exhausted, while a support agent on a different budget is unaffected.

LiteLLM's proxy shows the same idea in a product. Budgets can be set globally, per key, per team, per team member, and per customer, where the customer is whatever value your application passes in the user field, with no key needed for each customer. Before a request, LiteLLM reserves an estimated cost, using max_tokens when present, and replaces it with the actual cost afterward.

The costs are operational and each is documented. LiteLLM states that every budget on its page is enforced against spend read from a database, so none of them cap anything on a deployment without one, and its global max_budget fails open there. It also warns that disabling reservation can let concurrent requests exceed a budget, and offers a fail_closed_budget_enforcement setting for cases where the budget must stay a hard ceiling if Redis is unavailable or stale. A gateway is something you run, size, and secure, and a client that skips the base URL is not governed by it.

A worked example of the gap

Take a hypothetical retry loop that spends $60.00 per hour. With only a daily cost report, you may not see it until the next day's bucket, which could be up to 24 hours of spend, or $1,440.00. With a usage alert polled every minute, you might page someone within 10 minutes, about $10.00, and a person still has to act. With a gateway budget of $50.00 for that service, the request that would exceed the budget is rejected, so the loop is bounded at roughly the budget no matter when anyone looks.

The numbers are invented, but the ordering is not. Daily reports bound how fast you learn, alerts bound how fast a human reacts, and a gateway budget bounds the spend itself.

A decision rule

Ask what you need the system to do when spend is wrong. If the answer is tell us, monitoring is enough, and it is the right first step because it takes an admin key rather than a rollout. If the answer is stop it, or give each customer their own limit, you need something in the request path. Many teams end up with both, using the provider report as the authoritative total and the gateway ledger for attribution and limits, and comparing the two to find traffic that bypasses the gateway.

• Only need totals by project, key, or model: provider usage and cost API.
• Need an alert when usage spikes: poll the usage endpoint, no gateway required.
• Need a per-customer or per-team limit: gateway with budgets and a database.
• Need a hard stop before spend: gateway with reservation, and decide whether it fails open or closed.