DEV Community

Cover image for Bigger Isn't the Question: Picking a Local Model per Task
Mikalai Karytka
Mikalai Karytka

Posted on

Bigger Isn't the Question: Picking a Local Model per Task

The instinct, when a local run is disappointing, is to reach for a bigger model. For agent workloads that instinct is usually wrong. The question that actually decides the outcome is not how large the model is. It is whether its throughput matches how your workload hits it.

I learned this the expensive way. A strong 27B model produced nothing over two long runs. Swapping in a dense 8B with five slots multiplied the throughput and ended the timeouts, though it did not make the run finish either. Here is how I pick a local model per task now, the mechanics that make the choice fairly predictable, and what the numbers do and do not show.

The throughput numbers below are relative to one machine: a single Apple M4 Max with 64 GB of unified memory, serving GGUF quants through LM Studio's bundled llama.cpp. No cloud, no rented GPU.

The short version

Scenario Model class Launch
Deep mode, multi-agent swarm dense 8B, large native context CONTEXT_LENGTH=131072 PARALLEL=5
Quick, targeted check (1-2 agents) 7B coder-security CONTEXT_LENGTH=32768 PARALLEL=2
Max reasoning, time not critical 27B dense CONTEXT_LENGTH=131072 PARALLEL=1, not deep mode
Deep mode on a 27B not on a single machine n/a

The rest of this is why, because the reasoning transfers to models and hardware that are not in that table.

What my own runs showed

The most convincing evidence was not a vendor benchmark. It was the behavior on my own hardware, my own target, in the mode I actually run: the deep scan mode of Strix, an open-source agentic pentest framework (I compared it with eleven others here). Its prompt tells the agent to fan out into as many sub-agents as it can. The numbers come straight from each run's own output.

A 27B dense cyber model in deep mode, single slot, ran for 8.9 hours on one attempt and 8.0 on another, roughly 17 hours in total, and produced zero reviewed attack surfaces and zero findings. All of that time went into recon that never finished, with requests dying on an idle timeout before the first token even arrived.

Three observations, each one load-bearing.

It managed about six to seven requests per hour. A 27B dense model in a single thread physically cannot turn the crank of a multi-agent deep scenario fast enough. Over 8.9 hours it got through 61 requests.

Time to first token was over 20 minutes. One log line showed 23.6 minutes spent just processing the prompt before any output began, and the request was then killed by the idle timeout. Dozens of requests per run died the same way.

Zero surfaces, zero findings across every completed run. Deep mode never left the recon phase. The swarm launched, but each of its requests sat in a queue behind the single model slot and timed out before testing anything.

The conclusion from the facts: a "27B plus one slot plus deep mode" configuration was unworkable, not because the model was bad, but because its throughput did not match how deep mode loads a model.

Why a big dense model in one thread is so slow

Three things stack up, and they stack up specifically in a multi-agent mode.

Prefill dominates. The average input was 64,000 to 70,000 tokens per request. On a dense 27B, preprocessing that much context takes minutes, and it happens from scratch on every turn, because the prompt prefix was not being cached. That alone is where the 20-minute time-to-first-token comes from.

Deep mode deliberately runs a swarm. The mode's own prompt tells the model to scale out to maximum parallelization, so it spins up several sub-agents that all hit one local model at once. A cloud API absorbs that. A single local server does not.

One slot serializes the swarm into a queue. With a single slot, every concurrent request lines up one behind another, and any that waits longer than the idle timeout without a byte of response is cancelled as if it hung, even though nothing broke.

You cannot just raise the slot count on a 27B either. Each extra slot is roughly another KV cache sized to your full context, which at long context is tens of gigabytes, and the server hits a memory guardrail. So a big dense model leaves you pinned between "one slot and painfully slow" and "more slots and out of memory." The way out is a smaller model.

Match the model to how the work hits it

The dense 8B is the workhorse for deep mode. A dense 8B with a large native context (no YaRN rope-scaling crutch) is the key. Its small size makes the KV cache per slot cheap, so PARALLEL=5 genuinely fits in a 64 GB budget. The swarm gets real slots, requests stop queuing for 20 minutes, and the idle-timeout cascade disappears. You pay for it in the quality of each single step, since an 8B reasons less deeply than a 27B. In a multi-agent run that is the right trade: five agents that actually work beat one agent that hangs.

A 7B coder-security model is the scalpel. Smaller still, and tuned for code and offensive coding, but with a small native context. Above that window it leans on rope-scaling and quietly degrades. That makes it wrong for a long deep session with 60K-plus input, and right for a quick, targeted check: one or two agents, a short session, a concrete hypothesis like "look at this one endpoint for SQL injection."

A 27B dense model is for depth you pay for with time. It belongs in non-deep, single-slot work where one agent's reasoning quality matters and waiting is fine: walking a tricky finding by hand, building an exploitation chain on top of recon that is already done, reasoning through complex logic. Run it with PARALLEL=1 and keep it out of deep mode. Feed it finished recon and a narrow task, so the per-request input stays small, the prefill stays short, and you spend its strength where it earns its keep.

What the 8B actually changed

I ran the same deep scan against the same target with the dense 8B and PARALLEL=5. In 8.7 hours it made 461 requests, against 61 for the 27B in 8.9 hours: roughly 53 an hour instead of 7. The idle-timeout cascade was gone, with two cancellations in the whole run instead of dozens.

It still had not reviewed a single attack surface when I stopped it. Throughput was the bottleneck I could fix by picking a smaller model, and fixing it was necessary. It was not sufficient. The 8B was now turning the crank fast, but each of those requests carried about 77,000 input tokens (35 million in total), because every turn of an agent sends its whole growing context again. That is what the habit below is for.

The one habit that makes all of this cheaper

For the big model especially, do not make it redo work. Hand it finished recon and a tight instruction (something like "do not repeat basic recon, go straight into active testing on the surface already found"). Smaller input per request means shorter prefill, and the model's real strength, reasoning, is aimed at the part that actually needs it. Most of the pain above came from making a slow model re-prefill 70K tokens on every turn.

Honest caveats

I did not measure finding quality between models. Neither model completed a deep review: both 27B runs stalled in recon, and the 8B run, which I stopped after 8.7 hours, had not reviewed a surface yet. Throughput is what the data supports, not depth of findings.

Vendor benchmarks for the "cyber" models were self-reported and general, with no independent, security-specific evaluation. They are not a reason to pick a model.

Native context is a real limit, not a setting you can just raise. Pushing CONTEXT_LENGTH past a model's trained window turns on rope-scaling and degrades quality silently. Check the GGUF metadata's own context length before trusting a bigger number.

And the throughput figures are from my hardware and my target. On a different machine the absolute numbers move, but the shape holds: a dense 27B in deep mode chokes, and a dense 8B with real parallelism keeps the swarm moving.

This is the second piece in the local-LLM series, after running a capable LLM on your own machine. Still to come: why tool-calling breaks on small models, and spreading one workload across several Macs. The thread through all of it is the same: match the model to the shape of the work, not to the size on the model card.

Top comments (0)