OpenAI Batch API: When the 50% Discount Is Worth the Latency

What the Batch API trades for the discount

OpenAI's Batch guide describes three benefits compared with synchronous calls: a 50% cost discount, a separate and higher rate limit pool, and completion within 24 hours, often sooner. The completion window can only be set to 24h, and the Batch API FAQ states that OpenAI cannot change that period. You upload a .jsonl file where each line is one request with a unique custom_id, create the batch, poll its status, and download an output file.

The constraints are specific. One input file can only include requests to a single model. A batch may hold up to 50,000 requests and a 200 MB file, and you can create up to 2,000 batches per hour. Streaming is not supported, and output lines may not match input order, so results must be joined on custom_id. The output file is deleted 30 days after the batch completes.

The separate rate limit pool matters on its own. OpenAI states that batch usage does not consume tokens from your standard per-model rate limits. A team that is throttled by synchronous limits during the day can move bulk work to a batch and free that capacity for interactive traffic.

The rates, read from the pricing page today

OpenAI's pricing page lists Batch rates beside Standard. For short-context requests, gpt-6.1-sol is $2.00 input and $10.00 output at Standard and $1.00 and $5.00 at Batch. gpt-6-luna is $0.10 and $0.50 at Standard and $0.05 and $0.25 at Batch. Cached input is halved too: gpt-6.1-sol is $0.10 per million cached tokens at Standard and $0.05 at Batch.

Anthropic runs a comparable program. Its pricing page states that the Batch API gives a 50 percent discount on both input and output tokens, which puts Claude Sonnet 5.5 at $1 and $5 per million tokens against $2 and $10 standard, and Claude Haiku 5.5 at $0.05 and $0.25 for prompts up to 100,000 tokens. Anthropic also documents that prompt caching discounts stack with the batch discount.

One check comes first: not every model supports batches. OpenAI's FAQ says the Batch API is widely available across most models but not all, and points to its model reference to confirm yours.

A worked nightly job

Take a hypothetical nightly evaluation run: 20,000 requests, each with 3,000 input tokens and 500 output tokens. That is 60 million input tokens and 10 million output tokens. On gpt-6.1-sol at Standard rates the run costs 60 x $2.00 + 10 x $10.00 = $220. At Batch rates it costs 60 x $1.00 + 10 x $5.00 = $110, so the batch saves $110 per run.

Now run the same job on gpt-6-luna. Standard is 60 x $0.10 + 10 x $0.50 = $11, and Batch is 60 x $0.05 + 10 x $0.25 = $5.50. The percentage is identical but the saving is $5.50 per run.

Whether that is worth building depends on what the asynchronous path costs you. Suppose, hypothetically, it takes 8 engineering hours at $100 per hour, or $800, to build submission, polling, retry of failed lines and result joining. The gpt-6.1-sol job repays that in 8 runs ($800 / $110 is about 7.3). The gpt-6-luna job needs 146 runs ($800 / $5.50 is about 145.5). The rule falls out of the numbers: the discount is a percentage, but the decision is made on absolute dollars per run and runs per month.

Flex: the same rates without the file pipeline

If the job can wait but you do not want a file-based workflow, OpenAI's Flex processing may fit. The Flex guide says tokens are priced at Batch API rates, with additional discounts from prompt caching, in exchange for slower response times and occasional resource unavailability. You set service_tier to flex on a Responses or Chat Completions request, and OpenAI's examples raise the client timeout to 15 minutes. The pricing page shows gpt-6.1-sol at $1.00 and $5.00 under Flex, matching Batch.

Flex is in beta with limited model availability, so check the pricing page for your model before planning around it. The trade is reversed from Batch: Flex keeps your code synchronous and needs no file pipeline, but you must handle the occasional unavailable response yourself, while a batch gives you a defined 24-hour window and a record of which requests expired.

Budget and retention gaps to plan for

Batches complicate spend control. Anthropic warns that, because of high throughput and concurrent processing, batches may go slightly over the spend limit configured for a workspace. Treat a workspace limit as a soft ceiling for batch work, and size jobs so that a small overshoot is acceptable.

Expiry is billed differently by each provider. OpenAI says that when a batch expires, unfinished requests are cancelled, completed responses are available in the output file, and you are charged for tokens consumed by completed requests. Anthropic's batch documentation says expired requests are not billed, because they never reached the model. Either way, record custom_ids of expired lines and decide whether to resubmit them.

One compliance note: OpenAI's FAQ says zero data retention does not apply to the Batch API. Organizations with ZDR can still create batches, but batch input files, outputs and errors are retained under the configured Batch and File Service retention policies. If a workload is in scope for ZDR, check this before moving it to batch.

• Take the discount when no user waits on the result, the job runs often, and absolute savings per run exceed the cost of the pipeline.
• Prefer Flex when the job is small, you want synchronous code, and you can tolerate occasional retries.
• Keep a cost report that tags batch usage separately, so the discount shows up in your own numbers and not only on the invoice.