All Topics
All Topics
Technology
Technology
AI
AI
Business
Business
Entertainment
Entertainment
News
News
Programming
Programming
Science
Science
Design
Design
Environment
Environment
Finance
Finance
Crypto
Crypto
Politics
Politics
Sports
Sports
Education
Education
Gaming
Gaming
Art
Art
Music
Music
Health
Health
Security
Security
Books
Books
Food
Food
Travel
Travel
Personal
Personal
Bluesky
Twitter

Prioritize routing over model choice to slash AI costs

The article argues that teams building AI agents make a critical mistake by choosing the model first and the architecture second. It advocates for prioritizing a "router" — a small piece of code that determines which tier of model handles each request. When the router is designed correctly, 70-80% of traffic can run on cheap local or async models, dramatically reducing AI costs. It cites Brian Armstrong's example of Coinbase cutting AI spend in half while token usage grew, reinforcing the principle that most AI work doesn't need expensive frontier models.

Tomasz Tunguz4h ago3 min readenOpinion
Read on tomtunguz.com

Key quotes

Most teams building agents pick the model first & the architecture second. That is backwards.
What matters is the router, a small piece of code that decides which tier of model handles each request.
Get the router right & 70-80% of traffic runs on local models that cost nothing per call, or on async models that reduce AI spend by 90%+.
How to keep AI spend flat while token usage grows exponentially

From the article

"Prioritize routing over model choice. Most AI work runs on cheap local models."
Continue reading on tomtunguz.com

You might also wanna read

Model routing cuts AI costs by matching tasks to the right model, challenging OpenAI and Anthropic's premium pricing model

Model routing is an emerging practice where companies match AI tasks to appropriately sized models—sending complex problems to expensive fro

cnbc.com·1mo ago

Coinbase CEO Brian Armstrong outlines AI cost-saving strategy using cheaper models

Coinbase CEO Brian Armstrong outlined a strategy to keep AI costs "roughly flat" by routing prompts to cheaper AI models when appropriate, r

businessinsider.com·1mo ago

Companies shift from indiscriminate AI spending to cost-effective model routing strategies

Companies are moving away from "tokenmaxxing" — the practice of indiscriminately using expensive AI models for every task — and adopting "mo

businessinsider.com·4d ago

AI Inference Routers: How vLLM Semantic Router Enables Model Collaboration as the New Frontier

This article discusses the emerging paradigm of AI inference routing, where "routers" act as a control plane between user requests and AI mo

vllm.ai·10d ago

AI Inference Routers: How vLLM Semantic Router Enables Model Collaboration as the New Frontier

This article discusses the emerging paradigm of AI inference routing, where "routers" act as a control plane between user requests and AI mo

vllm.ai·10d ago

AI Inference Routers: How vLLM Semantic Router Enables Model Collaboration as the New Frontier

This article discusses the emerging paradigm of AI inference routing, where "routers" act as a control plane between user requests and AI mo

vllm.ai·10d ago

Rising costs push AI industry toward smaller, cheaper models

The AI industry has long operated on the assumption that bigger models are more powerful and therefore superior. However, mounting costs are

techcrunch.com·29d ago

Workweave Router: A model routing proxy for AI agent systems that cuts costs by 40-70%

Workweave's Router is a model routing tool for agentic AI systems that acts as a drop-in proxy for Anthropic, OpenAI, and Gemini APIs. It us

github.com·12d ago

Comments

Sign in to join the conversation.

No comments yet. Be the first.