When to Use an AI Batch API Instead of Real-Time Requests

Most AI API examples follow the same pattern: send a request, wait for the response, then move on. That works well when a person or application needs the result immediately.

But many jobs do not need to happen in real time. If you have thousands of reviews to classify, product descriptions to rewrite, documents to summarize, or test prompts to run, waiting for every response individually can add avoidable cost and coordination work.

That is where a Batch API fits. Instead of handling each request as an immediate conversation, you submit independent requests together, let them run in the background, and collect the results later. On NanoGPT, supported batch token usage is priced 50% lower than its normal synchronous equivalent. Non-token charges such as provider-native web search remain at their standard rate.

What makes a job suitable for batching?

A workload is a good fit when:

• It contains many separate requests
• No request needs another request's result
• All requests can use the same model
• Nobody is waiting on an immediate response
• You can process the results later

Good examples include:

• Classifying customer feedback by topic or sentiment
• Summarizing a collection of documents
• Extracting the same set of details from many records
• Generating descriptions for a product catalog
• Running evaluations against a fixed prompt set
• Producing draft tags, titles, or metadata
• Describing or categorizing images provided as supported image_url content

The important word is independent. One request should not need the answer from another request in the same batch.

When real-time requests are better

Use normal API requests when:

• A user is waiting for the answer
• You want the response to stream as it is generated
• Each step depends on the previous result
• The model needs a provider-hosted tool or an interactive tool loop
• A single batch would need more than one model
• The job uses an endpoint other than Chat Completions or Responses
• The Responses request needs stored state, a prior response, or background mode

A chatbot is the clearest example. Its next request depends on what the user says after seeing the previous answer, so putting an entire conversation into one batch would not make sense.

The same is true for many agents. If an agent must inspect a result, choose a tool, and decide what to do next, it needs an interactive request loop rather than a queue of independent prompts.

A batch is a list of normal requests

NanoGPT uses JSONL files for file-backed batch input. JSONL means one JSON object per line. Each line needs a unique custom_id, method: "POST", a supported url, and an endpoint-specific request body. Use /v1/chat/completions with messages, or /v1/responses with input. Every row in one batch must use the same endpoint and model.

For example, a two-request file could look like this:

{"custom_id":"review-001","method":"POST","url":"/v1/chat/completions","body":{"model":"your-supported-model","messages":[{"role":"user","content":"Classify this review as positive, neutral, or negative: The setup was easy and everything works well."}],"max_tokens":12}}
{"custom_id":"review-002","method":"POST","url":"/v1/chat/completions","body":{"model":"your-supported-model","messages":[{"role":"user","content":"Classify this review as positive, neutral, or negative: Delivery was late and the package was damaged."}],"max_tokens":12}}

Replace your-supported-model with a model currently available through the Batch API. Every Chat Completions row must include max_tokens or max_completion_tokens, which caps how many tokens each response can use. File-backed Responses rows instead require max_output_tokens of at least 16.

A Responses batch uses the same envelope with a different row URL and body:

{"custom_id":"summary-001","method":"POST","url":"/v1/responses","body":{"model":"gpt-4.1-mini","input":"Summarize this document in one sentence.","max_output_tokens":128}}

You do not combine all your source material into one enormous prompt. Each line remains its own request, with its own messages and result.

The practical NanoGPT workflow

The Batch API page handles the main workflow in one place:

1. Select the API key that should own the job.
2. Upload the JSONL file.
3. Create the batch.
4. Follow its status while it runs.
5. Download the output and error files when it finishes.

The output is also JSONL. Every completed line includes its original custom_id, so your script can reconnect each answer to the correct review, document, product, or test case.

You can cancel an active job, although cancellation may take time if work is already in progress.

Cost and timing

Batch jobs draw from your account balance. When you create one, NanoGPT checks and reserves enough balance for a conservative maximum estimate based on the input and each row's output limit. The final cost is based on actual completed usage, not the maximum estimate.

The only accepted completion_window value is 24h. A job may finish earlier, but batching is the wrong choice if a result is needed in the next few seconds or minutes.

Lower pricing does not automatically make a workload efficient. Keep prompts focused, choose a model appropriate for the task, and avoid requesting far more output than each row needs.

Using the API directly

For smaller automated jobs, use the inline endpoint at https://api.nano-gpt.com/api/beta/batches. Send one JSON body containing a batch-level endpoint, model, and a requests array:

{
  "endpoint": "/v1/chat/completions",
  "model": "openai/gpt-4o-mini",
  "requests": [
    {
      "custom_id": "review-001",
      "body": {
        "messages": [{ "role": "user", "content": "Classify this review: Excellent." }]
      }
    }
  ]
}

Creation returns 202 Accepted. Poll GET /api/beta/batches/{batch_id}; once the job completes, the same response contains a results array. A request body can omit model and inherit the batch-level model. The inline API also accepts endpoint: "/v1/responses" with Responses request bodies. If an inline row omits its endpoint-specific output cap, NanoGPT applies a 4096-token default.

The original OpenAI-compatible file workflow remains available for reusable JSONL inputs and clients that already use OpenAI's Batch API. Use https://api.nano-gpt.com/api/v1 for the calls that upload, create, poll, cancel, and download a file-backed batch. Inside the JSONL file, every row's url must match the batch endpoint: /v1/chat/completions or /v1/responses.

The basic flow is:

1. Upload a JSONL file with purpose=batch.
2. Create a batch for /v1/chat/completions or /v1/responses with completion_window set to 24h.
3. Poll the batch status or cancel it if necessary.
4. Download the output and error files.

Batch requests are non-streaming. Chat Completions batches support text and compatible image_url content across selected OpenAI, Claude, Gemini, managed, and Fireworks Batch API models. Responses batches currently use direct OpenAI models and support function or custom tools, structured text output, and remote or data-URL images. They are stateless: store is forced to false, and prior-response chains, conversations, background mode, provider-hosted tools, file references, and NanoGPT-only extensions are rejected. Use separate batches for different endpoints or models, and validate a small job before building a large one because model availability changes.

Choosing between batch and real time

Use the normal API when the answer is part of a live interaction. Use the Batch API when you already have a list of independent jobs and can collect the answers later.

If the work can wait, batching runs the same per-item prompts asynchronously at the lower batch rate.

Open the NanoGPT Batch API page to upload and manage a job, or see the API documentation when integrating the workflow into your own application.