Before vs. After: Real Cost Comparison

Let's see the impact on a real workload — 1,000 requests per day with a mix of complexity levels:

How to Classify Requests

You don't need a complex ML system to classify requests. Three approaches, from simplest to most accurate:

1. Keyword-Based Routing (Easiest)

Route based on simple patterns in the input:

• Contains "summarize", "translate", "format" → budget model
• Contains "analyze", "compare", "explain" → mid-tier model
• Contains "plan", "design", "write" → premium model

Accuracy: ~70%. Good enough for most applications.

2. Length-Based Routing (Simple)

Shorter inputs are usually simpler tasks:

• < 200 tokens input → budget model
• 200-1000 tokens input → mid-tier model
• > 1000 tokens input → premium model

Accuracy: ~65%. Works well for chat applications.

3. Classifier Model Routing (Most Accurate)

Use a tiny, fast model to classify request complexity before routing:

• Run input through a small classifier (GPT-4o mini, ~$0.0001/classification)
• Classifier returns: simple, moderate, or complex
• Route to appropriate model

Accuracy: ~85-90%. Best for high-stakes applications.

Implementation: Simple Router Pattern

Here's the core routing logic — it fits in a single function:

The key addition is a quality fallback: if the budget model's response doesn't meet a quality threshold (e.g., too short, contains errors), automatically retry on the next tier. This ensures quality while still saving on the 80%+ of requests that budget models handle well.

Quality Fallback: The Safety Net

The biggest concern with routing is quality degradation. A quality fallback handles this:

• Response length check: If the response is suspiciously short (< 50 tokens for a question), retry on a higher model
• Confidence scoring: Some models return confidence scores — use them to trigger retries
• User feedback loop: Track thumbs-down rates per model and route more to higher models if quality drops
• A/B testing: Run 10% of traffic through a single premium model to measure routing quality

Provider-Specific Routing Tips

OpenAI Ecosystem

Route GPT-5 for critical reasoning, GPT-4o for general tasks, GPT-4o mini for simple ones. Use batch API for background tasks (50% discount).

Anthropic Ecosystem

Route Claude 4 Opus for complex analysis, Sonnet for code generation, Haiku for classification and extraction. Prompt caching saves 90% on repeated prefixes.

Cross-Provider Routing

Don't limit yourself to one provider. Mix and match for optimal cost:

• Gemini 3 Flash for the cheapest simple tasks ($0.10/$0.40)
• Claude Haiku 4.5 for mid-tier tasks with great quality ($1.00/$5.00)
• Claude Sonnet 4.6 for complex reasoning ($3.00/$15.00)

Measuring Success

Track these metrics after implementing routing:

• Cost per request — should drop 50-70%
• Average quality score — should stay the same or improve
• Fallback rate — if > 15%, your classifier needs tuning
• Latency — budget models are often faster, so TTFT should improve

The Bottom Line

Multi-model routing is the single most impactful cost optimization you can implement. Start with simple keyword-based routing — it captures most of the savings with minimal engineering effort. Add a classifier model and quality fallback as you scale.

The math is simple: if 40% of your requests are simple, routing them to a model that costs 90% less saves you 36% on total costs immediately. Add moderate request routing and you're at 50-60% savings.

— See if you're overpaying for AI APIs