A follow-up to my original 16GB GPU experiment, this time with consistent quantization, repeated benchmarks, and a few unexpected results.
I recently published an article about experimenting with Mixture-of-Experts (MoE) offloading on my laptop.
I tested five different models, compared traditional layer offloading against selective expert offloading, and came away with some pretty interesting results.
Some models performed better with selective offloading. Others were considerably faster using traditional layer splitting.
And one model, LFM2.5, performed so badly with selective offloading that I really wanted to figure out what was going on.
Well, the article got some great feedback.
People pointed out things I hadn't properly accounted for, questioned some of my assumptions, and suggested better ways to test what I was seeing.
Which, honestly, is exactly the kind of feedback I love getting.
So naturally, I went right back down the rabbit hole.
And this time, some of the results were very different.
First, what needed fixing?
If you haven't read the original article, you can find it here:
MoE Offloading Isn't Always Faster: I Tested 5 Models on a 16GB GPU
In that experiment, I was testing whether selective MoE offloading could improve performance when running models that were too large to fit entirely in my GPU's VRAM.
The idea is to keep certain operations running on the GPU while offloading selected expert weights and their computation to the CPU.
I used llama.cpp with the --n-cpu-moe option and compared it against traditional layer splitting using -ngl.
But there were a couple of problems with my original testing.
The models weren't all using the same quantization.
I had Phi-3.5 running at Q4_K_M, Apex at Q5_K_M, Qwen at Q6_K, and Gemma and LFM at Q8_0.
That made it difficult to know whether the performance differences were related to the model architectures, quantization, or something else.
A couple of people brought this up in the comments, particularly regarding LFM's terrible performance.
I also wasn't doing enough repeated measurements.
Another person pointed out that laptop thermals, background activity, and ordinary measurement noise could influence the results. He suggested interleaving the configurations rather than running one strategy completely before moving to the other.
That was a really good point.
My original benchmarks were useful for spotting interesting behavior, but some of the smaller differences weren't necessarily meaningful.
So I decided to do another round of testing.
The new test setup
I'm still using the same laptop:
GPU: NVIDIA RTX 4090 Laptop GPU, 16GB VRAM
CPU: Intel Core i9-14900HX, 24 cores / 32 threads
RAM: 32GB DDR5
OS: CachyOS / Arch Linux
Inference engine: llama.cpp with CUDA and Flash Attention enabled
But I changed several things about the testing process.
All five models were now running at Q4_K_M.
I also updated llama.cpp, adjusted the offloading configurations to accommodate the smaller quantized models, and kept enough free VRAM to avoid pushing the GPU right to its memory limit.
For each model, I used the same prompt:
"What are 3 practical benefits of using Arch Linux?
Each configuration received a warmup request before measurement.
I then ran three repeated measurements per strategy in an interleaved order, with short cooldowns between runs.
Instead of relying on a single result, I used the median generation speed.
This doesn't make the experiment perfectly controlled. In particular, changing the quantization, offload settings, and llama.cpp build means I can't attribute every difference to quantization alone.
But it does give me a much better picture of how these models behave under a consistent Q4_K_M setup.
The new results
Here's what happened with the initial 100-token Q4_K_M tests:
| Model | Traditional Split | Selective MoE | Result |
|---|---|---|---|
| Phi-3.5-MoE | 6.17 tok/s | 5.15 tok/s | Traditional +19.8% |
| Apex-35B | 8.20 tok/s | 9.70 tok/s | Selective +18.3% |
| Qwen3.5-35B | 7.44 tok/s | 9.00 tok/s | Selective +21.0% |
| LFM2.5-24B | 32.96 tok/s | 35.44 tok/s | Selective +7.5% |
| Gemma 4 26B | 13.58 tok/s | 13.58 tok/s | Tie |
Well, that was interesting.
In my original tests, traditional splitting was faster on Qwen, LFM, and Gemma.
Now selective offloading was ahead on Qwen and LFM, and Gemma was essentially tied.
Phi-3.5 had also gone in the opposite direction.
But I wasn't quite ready to declare those results final.
Some of the individual runs had pretty significant variation, including one unusually fast Apex result and one unusually slow LFM result.
So I extended the testing to 512-token generations on a few of the models.
And that's where things got even more interesting.
Qwen: Quantization really does seem to matter
This was probably the result I was most interested in investigating.
In my original tests, Qwen3.5-35B at Q6_K performed better using traditional layer splitting.
But Apex-35B, which uses a related 256-expert MoE architecture, performed better with selective offloading at Q5_K_M.
I wondered whether the quantization difference was contributing to that behavior.
The new Q4_K_M results certainly suggested it might be.
So I went a little further.
I tested the same Qwen3.5-35B checkpoint at Q4_K_M and Q6_K, using the same llama.cpp build and longer 512-token generations.
Here's what I found:
| Quantization | Traditional Split | Selective MoE |
|---|---|---|
| Q6_K | 6.79 tok/s | 6.75 tok/s |
| Q4_K_M | 6.47 tok/s | 8.38 tok/s |
At Q6_K, the two strategies were practically tied.
At Q4_K_M, selective MoE offloading was 29.5% faster.
That's a substantial difference.
And unlike the original Apex-versus-Qwen comparison, this time I was comparing different quantizations of the same model, on the same inference engine build.
I still can't say quantization was the only factor, because the layer placement also needed adjusting to fit the different model sizes.
But this is much stronger evidence that the combination of quantization and memory placement has a major influence on selective offloading performance.
It also changed how I think about fine-grained MoE architectures.
Originally, I suspected that models with 256 routed experts might be particularly problematic for CPU offloading.
These results suggest it's not nearly that simple.
A 256-expert model can perform quite well with selective offloading, depending on the configuration.
LFM: The biggest surprise needed another look
If you read my first article, you might remember LFM2.5.
That was the model that managed a rather miserable 2.11 tokens/sec with selective offloading at Q8_0.
Traditional splitting achieved 9.03 tok/s.
More than four times faster.
It was such a dramatic difference that several people commented on it.
Mudassir suggested that the heavier 8-bit expert weights might be putting too much pressure on CPU memory bandwidth.
I suspected something similar, but didn't have the evidence to separate that from the architecture itself.
Well, switching to Q4_K_M produced a massive improvement.
The initial 100-token run reached 35.44 tok/s with selective MoE offloading.
That's approximately 16.8 times the original result.
Traditional splitting also improved considerably, reaching 32.96 tok/s.
But there was a problem.
One of the selective runs dropped all the way to 11.23 tok/s, while the other two were above 35 tok/s.
That's a pretty ridiculous amount of variation.
So I repeated the experiment using longer 512-token generations.
This time, the results looked quite different:
| Strategy | Median Generation |
|---|---|
| Traditional Split | 21.72 tok/s |
| Selective MoE | 21.88 tok/s |
Essentially a tie.
The weird 11.23 tok/s dip didn't reappear in those longer runs, but both configurations were considerably slower than their original 100-token medians.
That raises questions about workload length, sustained performance, and what might be changing during generation.
What I can say is that LFM clearly isn't inherently terrible at selective expert offloading.
Under the new Q4_K_M configuration, it performed reasonably well.
But whether the improvement came primarily from quantization, different layer placement, changes in llama.cpp, or a combination of all three still needs more investigation.
And the longer test also reminded me of something important.
A configuration that looks faster over 100 tokens might not have the same advantage over 512.
That's something I'll definitely keep in mind when benchmarking models in the future.
Then I tried speculative decoding with Gemma 4
Because apparently five models and two offloading strategies weren't enough to keep me entertained.
I also wanted to see whether Multi-Token Prediction (MTP) speculative decoding could improve performance when using selective MoE offloading.
Gemma 4 26B-A4B has a separate MTP drafter that can propose multiple tokens for the main model to verify.
The idea is that generating and verifying several tokens together can sometimes be faster than generating them individually.
In theory, that sounded promising.
In practice, my results were not particularly encouraging.
Without MTP, selective offloading reached:
13.58 tokens/sec
With the MTP drafter enabled:
7.24 tokens/sec
So instead of speeding things up, it reduced throughput by approximately 46.7%.
I wanted to understand why.
During a longer 512-token test, I captured the drafter telemetry:
draft acceptance = 0.11678103 accepted / 882 generatedmean len = 1.35
Only 11.68% of the proposed draft tokens were accepted.
That's a pretty awful acceptance rate.
The longer MTP run also measured just 6.03 tok/s.
Speculative decoding has overhead. If the drafter isn't making predictions that the target model can accept often enough, the extra work can outweigh whatever speedup you hoped to achieve.
The low acceptance rate gives me a much more concrete reason to investigate the slowdown than simply assuming CPU/GPU synchronization was responsible.
I still want to check whether the drafter configuration or some other compatibility issue contributed to the poor acceptance rate.
So I wouldn't conclude from this that speculative decoding and CPU offloading are fundamentally incompatible.
Just that this particular configuration definitely didn't help.
Something else the repeat testing revealed: Noise
One of Riccardo's main suggestions was to account for measurement variation.
And after seeing these results, I'm glad I did.
For example, the three Apex selective offloading runs were:
9.21 tok/s9.70 tok/s25.50 tok/s
That last run was affected by early end-of-sequence generation, making it a poor comparison with the other runs.
LFM also produced that unusually slow 11.23 tok/s result.
And Gemma's traditional offloading runs varied from 8.98 to 16.85 tok/s.
Those are not small differences.
Interleaving the configurations helped make the testing process fairer, but it didn't magically remove every source of variation.
Using medians helped reduce the influence of individual outliers, although three measurements still aren't enough to establish a particularly reliable performance distribution.
The longer tests were especially useful because they showed that some of the apparent advantages became much smaller when the workload changed.
Going forward, I think longer generations and more repeated measurements are going to be standard practice for my experiments.
It's a little more work, but clearly worth it.
So what did I actually learn?
I went into the original experiment wanting to find out whether selective MoE offloading was a better way to run large models on a GPU with limited VRAM.
My first conclusion was that it depended on the model.
That still seems true, but now I'd add quite a bit more to it.
Quantization can change the outcome.
Qwen went from traditional splitting performing slightly better at Q6_K to selective offloading gaining a substantial advantage at Q4_K_M.
Offloading configuration matters.
Two configurations using the same model can place very different amounts of work on the CPU and GPU. The amount of VRAM occupied doesn't tell the whole story.
Short benchmarks can be misleading.
LFM's apparent 7.5% selective advantage in the 100-token tests practically disappeared over the longer generations.
Speculative decoding isn't a free speed boost.
Gemma's MTP drafter had a very low acceptance rate and made generation considerably slower under my tested setup.
And perhaps most importantly, a measurement and an explanation aren't the same thing.
I can measure that something runs faster or slower.
Explaining exactly why sometimes requires considerably more testing.
That distinction is becoming more obvious the deeper I get into this.
What's next?
I'm not quite finished experimenting.
I'd still like to test more quantization levels on the exact same checkpoint, ideally with more carefully controlled offloading configurations.
I'm also interested in monitoring actual CPU memory bandwidth and GPU behavior during inference.
That might help answer some of the questions about where these slowdowns are coming from instead of relying on educated guesses.
And eventually, I still want to repeat some of these tests on a machine with 64GB of RAM.
I'm especially curious how much more flexibility that provides for keeping expert weights in system memory without running into swap pressure.
But for now, I think I've learned quite a bit from these two rounds of testing.
Final thoughts
One of my favorite things about sharing experiments publicly is that sometimes people notice things you missed.
I thought my original results were interesting, and I still do.
But several people asked good questions about the methodology, the quantization differences, and whether the measurements were reliable enough to support some of my conclusions.
They were right to ask.
And when I went back and tested those things, some of my conclusions changed.
I'm glad they did.
I'd much rather discover that my original explanation wasn't quite right than keep repeating it because it sounded reasonable.
That's half the fun of experimenting with this stuff.
You start with one question, find an interesting result, and suddenly you're investigating three completely different things you hadn't even considered.
And occasionally you realize you've spent an entire weekend benchmarking language models instead of doing literally anything else.
You know, perfectly normal behavior.
A big thank you to everyone who took the time to read the original article and offer thoughtful feedback. It genuinely helped make this second round of experiments better.
And if anyone has suggestions for what I should test next, particularly around quantization, CPU memory bandwidth, or speculative decoding, I'd love to hear them.
Apparently I haven't spent enough time staring at tokens-per-second measurements yet.
Top comments (36)
One thing I'd be curious to see in the next round is how context depth changes the offloading results. A setup that looks fine at 512 tokens might behave very differently once the KV cache grows and starts competing for VRAM with the model weights.
I'd run a few
llama-benchtests with fixed generation lengths but different context depths, using-dand-pgto separate prompt processing from generation. That could help distinguish whether the slowdown is coming from expert weight transfers, attention/KV-cache pressure, or the interaction between them.Especially with LFM, where the 100-token and 512-token results told quite different stories, I think that would be a useful next experiment.
I actually did some context scaling tests with Phi-3.5 and Apex in my original experiments, going from 4K to 16K, but those were changes to the configured context window rather than tests with prompts that actually filled it.
I haven't explored how growing the actual context affects the different offloading strategies yet, and I think that would be really interesting, especially with the differences I saw between the shorter and longer LFM runs.
Using llama-bench to separate prompt processing and generation at different context depths sounds like a good next step. I'd love to see whether the KV cache starts affecting which offloading configuration performs better as the context grows.
Thanks for the suggestion! My list of things to test keeps getting longer.
That distinction between configured context size and actually filling the context is important. A 16K context window doesn't necessarily tell us much about memory pressure if the prompt only uses a fraction of it.
I'd be particularly curious about what happens when the KV cache starts competing with expert weights for GPU memory. In an MoE setup, that could change the offloading trade-off quite noticeably, especially if some experts are already being moved between CPU and GPU during generation.
It'd be interesting to see whether the crossover point between different offloading strategies shifts as the actual context grows. That might explain some of the differences you observed in the LFM runs.
Seeing someone revisit their own conclusions publicly is interesting to me, especially when the newer results complicate the original explanation. That part of research doesn't get discussed enough.
I think it's easy to get attached to an explanation once you've written it all out, but sometimes the more you investigate, the more complicated things get. I actually enjoyed discovering that some of my original assumptions weren't quite right. It gave me a lot more to explore, and I learned more from going back and questioning the results than I probably would have otherwise.
Your reply reminds me of that old Bertrand Russell observation about intelligent people being full of doubt. I’ve always found that interesting because the deeper you investigate something, the more you realize how much you might be missing. Being willing to question your own conclusions seems like one of the most important parts of research.
The early-ending Apex run makes me want actual output-token counts beside each timing. Were the 512-token tests all reaching 512, or was that just the limit? That would help tell whether the longer runs are measuring comparable amounts of work before digging into why they slowed down.
That's a really good point, and something I should have documented more carefully. The 512 tokens was the configured generation limit, but I didn't include the actual output-token counts for each run in my results.
The early-ending Apex run made me realize how much that can affect a short benchmark, and you're right that it's important to verify the longer runs were actually doing comparable amounts of work too.
I think my next step is going to be logging the actual generated token counts alongside the timing data, rather than relying on the configured limit. Thanks for pointing that out.
The 'a measurement and an explanation aren't the same thing' is the most important line in this article. So many benchmark write-ups stop at the number and never dig into why it happened. The fact that you went back and re-tested after feedback makes this much more credible.
Thank you! That's actually one of the biggest things I've taken away from all this. It's one thing to observe a performance difference, but understanding exactly what's causing it is a whole different challenge. And I've definitely caught myself coming up with explanations that sounded perfectly reasonable until I did more testing. I'm glad I went back and revisited the results. It's been a really interesting learning experience.
This level of intellectual honesty is rare to see in benchmark posts. It takes a lot of discipline to go back, rerun the tests, and publicly admit that the initial "perfectly reasonable" narrative was wrong. It's a great reminder that performance metrics are just numbers until you actually understand the mechanism behind them.
I'm curious — when you were digging into the data this time around, did you find that the MoE offloading behavior was more dependent on the specific prompt structure, or was it mostly the hardware memory bandwidth limits doing the heavy lifting?
Thank you. I really appreciate that. I think one of the most interesting parts of experimenting with things like this is discovering that an explanation you were pretty confident about doesn't necessarily hold up once you start testing further.
As for the prompt structure versus memory bandwidth, I haven't isolated those yet. I've deliberately been using the same prompt across the models to keep that part consistent, so I can't really say how much different prompts or expert routing patterns might affect performance.
The results definitely suggest that quantization and memory placement play a significant role, but I haven't measured memory bandwidth directly either, so I don't want to claim that's the whole explanation.
Testing different prompt types and lengths while tracking expert activation patterns would actually be a really interesting next step. You've given me another thing to add to my growing list of experiments.
Keeping the prompt constant was definitely the smart move for establishing a baseline. Varying prompt lengths while tracking expert activation is a brilliant next step—I'd be fascinated to see if the routing patterns actually shift. Can't wait to read about those results if you post them!
three interleaved runs with medians is a real step up, but you also changed the quantization, the llama.cpp build and the offload settings in the same pass, so a flipped result could come from any of the three. i would pin the build first and rerun one model at both quants, since that isolates the quant effect cheaply.
on a laptop 4090 the thermal throttle also tends to show up in run three, not run one, so are you logging clocks alongside tokens per second?
You're absolutely right about the multiple variables changing, and that's something I tried to be careful about in the follow-up conclusions. I actually did another round of 512-token tests using the same Qwen3.5 checkpoint at Q4_K_M and Q6_K, both on llama.cpp build 11985. Selective offloading was about 29.5% faster than traditional splitting at Q4, while the two were practically tied at Q6.
The offload configurations still needed adjusting for the different model sizes, though, so it's not a perfectly isolated quantization comparison yet.
And no, I haven't been logging GPU clocks alongside the throughput measurements. That's a really good suggestion, especially considering some of the variation I saw between runs. I'd like to capture clocks and temperatures alongside the actual output-token counts in the next round so I can better separate thermal effects from the offloading behavior.
Thank you for the feedback. You've given me quite a bit to investigate.
the q4 vs q6 split is useful. selective offload winning by 29.5% at q4 and tying at q6 reads like the gain shrinks once more of the model fits in vram. since the offload configs differ per model size, i would log how many layers sit on gpu for each run next to tokens per second, then the gap is explained by one number. when you add clocks and temps, will you log them per run or sample them every few seconds during decode?
That's a good point about logging the GPU layer counts alongside throughput. I have the offload configurations recorded, but having the actual placement information right next to each result would make those comparisons much clearer.
And I was thinking of sampling clocks and temperatures every few seconds during decoding rather than just once per run. That way I could see whether performance changes line up with clock drops or temperature increases, especially during the longer generations.
I think adding that telemetry along with actual output-token counts would make the next round of tests a lot more useful. Thanks for all the suggestions. I'm learning quite a bit about benchmarking from these discussions alone.
sampling clocks every few seconds beats one reading per run, a throttle halfway through a 512 token decode would otherwise get averaged away into the final tok/s. i would log it on the same row as tok/s and layer counts so one csv answers whether a slow run was a placement problem or a thermal one. are you capturing cpu clocks too, since selective offload leans on the cpu side more?
The Gemma MTP result is the part I keep coming back to: an 11.68% draft acceptance rate turning a supposed speedup into a 46.7% slowdown (13.58 down to 7.24 tok/s) suggests acceptance rate should be reported before throughput in any speculative decoding benchmark. Did you notice whether acceptance changed between the shorter and the 512-token runs, or was it uniformly low? If it stays that low regardless of length, that points more to a drafter/target mismatch than to offloading overhead.
Unfortunately, I only captured the acceptance telemetry during the longer 512-token test, so I don't have the numbers from the original 100-token runs to compare against.
I was pretty surprised by how low that 11.68% acceptance rate was, though. It definitely makes me wonder whether there's something going on with the drafter configuration or compatibility with the target model, rather than the slowdown being caused primarily by offloading overhead.
I'd like to repeat those tests while logging acceptance rates at both generation lengths, and ideally compare against a configuration without CPU offloading too. That should help narrow down what's actually causing the poor performance.
Thanks for bringing that up, I hadn't originally planned to investigate speculative decoding this deeply, but apparently I've found another rabbit hole.
Quite fascinating to see this! I understand you are perhaps limited by your hardware here and it costs when you go do these on real cloud infrastructure.
May I ask, are you trying specific kind of workloads and does the results vary? I understand these models are a bit generic but even then are you seeing specific differences? Just curious! Do you know if the models have published data on, is that SWE Bench, or wherever they do? I used to track them until 6 months or so ago
Apologies, I have not read all your posts.
I'm not really testing the models against specific tasks or evaluating their actual capabilities in these experiments. I'm more interested in the hardware side of things, specifically how different MoE architectures behave when they're too large to fit entirely in my GPU's VRAM.
I've been using the same prompt across all the models so I can compare generation speeds, prompt processing, and the differences between traditional layer offloading and selective expert offloading. And yes, even with a fairly generic workload, I've seen some pretty dramatic differences depending on the model, quantization, and offloading configuration.
The models do have their own published benchmarks for things like coding and reasoning, but I haven't been evaluating those here. I'm mostly just curious how much performance I can squeeze out of my 16GB GPU without relying on cloud infrastructure. It's turned into a much bigger experiment than I originally planned.
Thanks - now I get the intent, and that makes sense - "I'm mostly just curious how much performance I can squeeze out of my 16GB GPU without relying on cloud infrastructure." - I did this too for various LLM choices with OMLX (not MOE) to know where I can use local LLM and save money.
"It's turned into a much bigger experiment than I originally planned." - this is what I discovered and had to scale back tests and ended up looking for other people's benchmarks for my use cases to pair down the LLM choices.
The confound you flag at the end is the real story: quant, offload config, and llama.cpp build all changed at once, so the Qwen and LFM2.5 flips can't be attributed to quantization alone. Isolating it needs the two-quant A/B (Q4_K_M vs Q6_K) on all five models with everything else pinned - you only ran that pair on Qwen. Also, with --n-cpu-moe the payoff depends on how skewed per-token expert activation is. Did you log distinct experts activated per token? Gemma tying at 13.58 tok/s both ways smells like uniform expert usage - nothing for selective scheduling to exploit. A histogram of expert hits would explain the tie and predict which architectures benefit before you benchmark.
You're right that changing multiple variables makes it difficult to isolate exactly what caused the performance differences. That's why I was careful to point that out in the follow-up.
I did run an additional Q4_K_M vs. Q6_K comparison on the same Qwen checkpoint using the same llama.cpp build and longer 512-token generations, which helped narrow things down a bit. The offload configurations still differed, though, so there's definitely more to investigate.
I haven't logged individual expert activation frequencies yet, but that's actually a really interesting idea. I'd be curious to see whether certain experts are consistently selected more often and how that relates to offloading performance.
I'm not sure Gemma's tie necessarily means uniform expert usage, since there could be several reasons the two configurations performed similarly. But looking at the routing distribution might help answer some of those questions.
Thanks for the suggestions.
Redoing it all at Q4_K_M and interleaving the runs is the honest fix — the first version's mixed quantizations made it impossible to tell what was architecture and what was just precision. The LFM2.5 swing is what I'd want to see next though: did it actually get faster once the quantization was consistent, or is selective offloading just a bad fit for that one regardless?
That LFM result was actually one of the biggest surprises for me. At Q8_0, selective offloading was only getting 2.11 tok/s compared to 9.03 with traditional splitting. But when I switched to Q4_K_M, the initial runs jumped to 35.44 tok/s with selective offloading.
I did some longer 512-token tests afterward, and both strategies settled around 22 tok/s, so they were practically tied. That definitely suggests selective offloading isn't inherently a bad fit for LFM.
I can't attribute the improvement entirely to quantization since I also adjusted the offload configuration and updated llama.cpp between tests. But it's a pretty dramatic difference, and I'd love to isolate those variables further. Apparently this rabbit hole isn't finished with me yet.
Retesting until the conclusion changes is exactly the evidence developers need to see. The reusable artifact would be a benchmark manifest that locks model revision, routing policy, host topology, warm-up, workload mix, and cost accounting; otherwise an offloading result can drift while every individual run still looks plausible.
That's a really good suggestion. One thing these experiments have definitely taught me is just how many variables can affect the results, even when you think you're keeping things consistent. I started out just wanting to compare two offloading methods, and ended up learning quite a bit about benchmarking itself along the way.
I've been keeping detailed experiment logs, but a standardized manifest that captures the exact model, configuration, hardware, and workload for each run would make comparisons and reproduction much easier. Honestly, that sounds like a pretty useful little tool to build too, and now you've got me thinking.
Thank you. I really appreciate the feedback.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.