DEV Community

Walker Miller
Walker Miller

Posted on Edited on Originally published at loopandretry.github.io

Cheap first, smart later: model routing that cuts cost without cutting quality

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

Here's the asymmetry that makes model routing worth the trouble: most of what an agent handles in production is easy — a well-formed tool call, a summary of a short document, a classification with an obvious answer — and you're routing all of it through the same model you needed for the 10% of requests that are actually hard. A cascade fixes that by trying the cheap model first and escalating only when it's warranted. Done right, this cuts the token bill 40-70% with no quality loss. Done wrong, it just moves your failures downstream where they're harder to see.

This post covers the pattern itself, the one design decision that determines whether it works (the escalation trigger), the failure modes that show up when you get that decision wrong, and when the added architectural complexity isn't worth it.

The pattern

A routing cascade is a pipeline, not an agent — the control flow is yours, not the model's, which matters because it means you can reason about it and test it (see when not to build an agent for why that distinction is the whole ballgame). The shape:

def route(request, cheap_model, expensive_model, escalate_fn):
    cheap_response = cheap_model.complete(request)
    if escalate_fn(request, cheap_response):
        return expensive_model.complete(request)
    return cheap_response
Enter fullscreen mode Exit fullscreen mode

That's the entire pattern. Everything that makes it work or fail lives inside escalate_fn. A cascade with a bad escalation trigger is worse than not having one, because it adds latency (you paid for the cheap call and the expensive one) without saving money on the requests that needed escalating anyway, or worse — it fails to escalate the ones that did.


Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article

Top comments (0)