Google Context Caching

Google's Context Caching is not a traditional batch API, but it achieves a similar cost reduction for workloads where the same (or very similar) prompt is reused across many requests. By caching a long prompt or system instruction, you avoid re-processing it on every call. For repeated prompts with small input variations, this can reduce costs by up to 75%.

DeepSeek: Already Cheap

DeepSeek's standard pricing is already among the lowest in the market. At $0.14 per 1M input tokens and $0.28 per 1M output tokens, batch processing offers only marginal additional savings. If you are already using DeepSeek for cost-sensitive workloads, the engineering effort to implement batch processing may not be justified.

Cost Savings Calculator: Batch vs Real-Time

Here is a concrete example showing the cost difference between real-time and batch processing for a GPT-4o workload. Assume 1,000 requests per day with an average of 800 input tokens and 400 output tokens per request.

At this scale, switching to batch processing saves over $1,000 per year. At higher volumes, the savings become even more significant.

When to Use Batch Processing

Batch processing is ideal for any workload where the response does not need to be delivered instantly. Common use cases include:

• Data labeling and classification: Tagging thousands of records with categories, sentiments, or metadata. The labels are needed eventually but not immediately.
• Report generation: Producing summaries, analytics reports, or insights from large datasets. These typically run on a schedule (daily, weekly) and do not require real-time responses.
• Content moderation: Reviewing user-generated content for policy violations. A delay of a few hours is acceptable for most moderation workflows.
• Embeddings generation: Creating vector embeddings for search, recommendation, or RAG systems. This is a one-time or periodic computation that does not need instant delivery.
• Translation pipelines: Translating large volumes of text where turnaround time is measured in hours, not milliseconds.
• Fine-tuning data preparation: Generating training data, cleaning datasets, or creating synthetic examples for model fine-tuning.

When NOT to Use Batch Processing

Batch processing is not a universal solution. Avoid it when:

• Real-time chat applications: Users expect instant responses. Batch processing introduces unacceptable latency for conversational AI.
• User-facing applications: Any application where the end user is waiting for a response in real time — search, recommendations, autocomplete, or interactive tools.
• Time-sensitive workflows: Fraud detection, alert systems, or any application where a delay in processing could result in missed opportunities or security risks.
• Low-volume workloads: If you are making fewer than 100 requests per day, the engineering effort to set up batch processing may not pay for itself.

Implementation Guide: OpenAI Batch API

Here is a practical guide to implementing batch processing with OpenAI's Batch API using Node.js and the native fetch API.

Step 1: Prepare Your Batch File

Create a JSONL file with one request per line. Each line must include a custom ID and the standard chat completion request format.

// prepare-batch.js
const fs = require('fs');

const requests = [
  'Summarize this product review: ...',
  'Classify this customer email: ...',
  'Extract entities from this document: ...',
];

const jsonl = requests.map((prompt, i) => {
  return JSON.stringify({
    custom_id: `request-${i}`,
    method: 'POST',
    url: '/v1/chat/completions',
    body: {
      model: 'gpt-4o',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 500,
    },
  });
}).join('\n');

fs.writeFileSync('batch-requests.jsonl', jsonl);
console.log(`Created batch file with ${requests.length} requests`);

Step 2: Upload and Create the Batch

// create-batch.js
const API_KEY = process.env.OPENAI_API_KEY;

// Step A: Upload the file
const formData = new FormData();
formData.append('file', fs.createReadStream('batch-requests.jsonl'));
formData.append('purpose', 'batch');

const fileRes = await fetch('https://api.openai.com/v1/files', {
  method: 'POST',
  headers: { Authorization: `Bearer ${API_KEY}` },
  body: formData,
});
const { id: fileId } = await fileRes.json();

// Step B: Create the batch
const batchRes = await fetch('https://api.openai.com/v1/batches', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    input_file_id: fileId,
    endpoint: '/v1/chat/completions',
    completion_window: '24h',
  }),
});
const { id: batchId } = await batchRes.json();
console.log(`Batch created: ${batchId}`);

Step 3: Poll for Completion

// check-batch.js
const checkBatch = async (batchId) => {
  const res = await fetch(
    `https://api.openai.com/v1/batches/${batchId}`,
    { headers: { Authorization: `Bearer ${API_KEY}` } }
  );
  const batch = await res.json();
  console.log(`Status: ${batch.status}`);

  if (batch.status === 'completed') {
    // Download results
    const resultRes = await fetch(
      `https://api.openai.com/v1/files/${batch.output_file_id}/content`,
      { headers: { Authorization: `Bearer ${API_KEY}` } }
    );
    const results = await resultRes.text();
    fs.writeFileSync('batch-results.jsonl', results);
    console.log('Results saved!');
  }
};

Batch Processing Pitfalls

Batch processing is not without its challenges. Here are the most common pitfalls and how to avoid them:

Retry Logic

Individual requests within a batch can fail independently. OpenAI marks these as failed in the output file. You need logic to extract failed requests, diagnose the cause (rate limits, malformed input, content policy violations), and resubmit them in a new batch. Do not assume a "completed" batch means every request succeeded.

Error Handling

Every request in the output includes an error field when it fails. Common errors include invalid_prompt (malformed JSON or missing fields), content_policy_violation (content filtered by safety systems), and max_tokens_exceeded. Build a pipeline that parses these errors and routes them appropriately — some are fixable (bad input), some are permanent (policy violations).

Monitoring

Batch processing runs asynchronously, which means you cannot rely on request-response timing for monitoring. Set up polling or webhooks to track batch status. Monitor for batches that stay in in_progress longer than expected, and alert on batches with high failure rates. Without monitoring, failed batches can go unnoticed for hours.

Concurrency Limits

OpenAI limits the number of concurrent batches you can run (typically one active batch at a time per organization). Plan your batch scheduling to avoid bottlenecks. If you have multiple daily workloads, stagger them or queue them sequentially.

Completion Window

OpenAI batches must specify a completion_window (e.g., 24h). If your batch cannot complete within that window, some requests may be left unprocessed. Size your batches appropriately and monitor completion times to avoid this.

Cost Comparison: Real-Time vs Batch at Scale

Here is what your monthly costs look like at three common volume levels using GPT-4o, assuming 800 input tokens and 400 output tokens per request:

At 1M requests per month, batch processing saves $3,000 every month — that is $36,000 per year. Even at the lower end, $30 per month adds up to $360 per year of savings for very little engineering effort.

If your workload can tolerate a few hours of latency, batch processing is the single easiest way to cut your AI API bill in half.

— See if you're overpaying for AI APIs