Common Failure Patterns and Fixes

1. Context window overflow

The most expensive error. You build a massive prompt, send it, and get a 400 error because you exceeded the model's context window. The fix: always count tokens before sending.

// Before sending, validate token count
const maxTokens = {
  'gpt-4o': 128000, 'gpt-5': 272000, 'gpt-5.5': 1000000,
  'claude-sonnet-4.6': 1000000, 'claude-haiku-4.5': 200000,
  'gemini-2.0-flash': 1000000, 'deepseek-v4-pro': 1000000
};

const limit = maxTokens[model] || 128000;
if (estimatedTokens > limit * 0.9) {
  // Truncate or summarize before sending
  prompt = truncateToTokenLimit(prompt, limit * 0.85);
}

2. Streaming connection drops

Streaming responses (SSE) can disconnect mid-response, especially on long outputs. The result: you've consumed input tokens but get no complete output. Fix: implement streaming resume or fall back to non-streaming for critical requests.

// Streaming error handling
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 60000);

try {
  const stream = await fetch(url, {
    ...options,
    signal: controller.signal
  });

  let fullResponse = '';
  for await (const chunk of stream.body) {
    fullResponse += parseChunk(chunk);
    clearTimeout(timeout); // Reset timeout on each chunk
  }
} catch (error) {
  if (error.name === 'AbortError') {
    // Timeout — retry with non-streaming
    return await callWithRetry(() => fetchNonStreaming(options));
  }
  throw error;
}

3. Concurrent request collisions

Sending many requests simultaneously often triggers rate limits. The fix: implement a request queue with concurrency limits.

class RequestQueue {
  constructor(concurrency = 5) {
    this.concurrency = concurrency;
    this.running = 0;
    this.queue = [];
  }

  async add(fn) {
    if (this.running >= this.concurrency) {
      await new Promise(resolve => this.queue.push(resolve));
    }
    this.running++;
    try {
      return await fn();
    } finally {
      this.running--;
      if (this.queue.length) this.queue.shift()();
    }
  }
}

Cost-Aware Error Handling

The biggest mistake: retrying expensive requests blindly. If you're retrying a request that costs $0.50 per attempt, three retries cost $1.50 for a single logical operation.

Model fallback chains

When your primary model fails, falling back to a cheaper model saves both time and money:

— See if you're overpaying for AI APIs

Monitoring and Alerting

Handle errors in code, but monitor them in production. Set up alerts for:

• Error rate >5% — Indicates a systemic issue, not random failures
• Retry rate >20% — You're retrying too aggressively, wasting money
• P95 latency >30s — Approaching timeout thresholds
• Monthly error cost >$50 — Set a hard cap and alert on it

Use our AI API cost monitoring guide to set up real-time tracking of error-related spend.

Provider-Specific SDK Retry Defaults

Most providers have built-in retry logic in their official SDKs. Here's what you get by default:

Important: Don't rely solely on SDK defaults. OpenAI and Anthropic retry on 429 and 500 errors, but they won't implement cost-aware retry logic or model fallbacks. Build your own wrapper on top.

The Complete Error Handler

Here's a production-ready error handler that combines everything:

async function resilientLLMCall(options) {
  const { model, prompt, maxCost = 0.50 } = options;
  const costPerToken = getModelCost(model);
  let totalCost = 0;

  const fallbackChain = getFallbackChain(model);

  for (const currentModel of fallbackChain) {
    const cost = getModelCost(currentModel);
    if (totalCost + cost > maxCost) break;

    try {
      const result = await callWithRetry(
        () => callLLM(currentModel, prompt),
        currentModel.includes('gpt') || currentModel.includes('claude') ? 2 : 1
      );
      return { ...result, model: currentModel };
    } catch (error) {
      totalCost += cost;
      console.warn(`${currentModel} failed: ${error.message}. Falling back.`);
    }
  }

  throw new Error('All models in fallback chain failed');
}

Quick Reference: Error Handling Checklist

• Do: Retry 429, 500, 502, 503, 529 with exponential backoff
• Do: Add jitter to prevent thundering herd
• Do: Set a max budget per operation
• Do: Implement model fallback chains
• Do: Validate token count before sending
• Do: Monitor error rates and costs
• Don't: Retry 400, 401, 403, 404 errors
• Don't: Retry without checking if input tokens were billed
• Don't: Use the same retry count for expensive and cheap models
• Don't: Ignore Retry-After headers