Why a Sparse 35B MoE Runs Faster on a CPU Than a Dense 27B Model

2026-11-01 · Ginigen AI

aiedge-aillmmoe

TL;DR

On a CPU, how fast a language model writes text depends mostly on how many bytes of weights it must read for each token, not on how many parameters it has in total. A sparse Mixture-of-Experts (MoE) model reads only the experts that are active for that token. That is why POCKET-35B, a 35B MoE model from our partner VIDRAFT, generates 3.13x faster than the dense Bonsai-27B on a MacBook M3 Pro CPU in the same stock llama.cpp build (13.8 against 4.4 tok/s, measured).

The market has noticed: the three POCKET repositories were downloaded 1,113,164 times in the last 30 days (Hugging Face public API, 1 November 2026).

What limits token speed on a CPU?

Generating one token means running the whole network once. For each layer, the CPU must:

1. read the layer's weights from RAM,
2. multiply them with a small activation vector,
3. move on to the next layer.

Step 2 is cheap on a modern CPU. Step 1 is not. A desktop or laptop moves tens of gigabytes per second from RAM, while the weights of a large model are several gigabytes even after quantization. In practice the CPU waits for memory. A useful rule of thumb is:

tokens per second  ~  memory bandwidth  /  bytes of weights read per token

This rule tells you where to look for speed. You can buy more bandwidth (new hardware), or you can read fewer bytes per token.

How does a Mixture-of-Experts model read fewer bytes?

A dense model uses every weight for every token. An MoE model splits its feed-forward layers into many small "experts" and a router picks a few of them per token. According to the POCKET-35B model card, the model has 256 experts per MoE layer and activates 8 per token.

So the total parameter count (35B) sets how much RAM you need to hold the model, but the active parameters set how many bytes you read per token. A 35B MoE can read far fewer bytes per token than a 27B dense model, and that difference turns directly into speed on bandwidth-bound hardware.

What do the measurements say?

Both models run in the same stock llama.cpp build on the same machine. These are the numbers published on the POCKET-35B card.

MacBook M3 Pro, 18 GB:

Xeon CPU, 16 threads: 27.0 against 10.1 tok/s generation (2.69x).

H100 GPU: 197 against 89 tok/s generation (2.22x), but Bonsai wins prompt processing (1816 against 753 tok/s).

Why does the dense model win prompt processing on an H100?

The H100 result is the exception that proves the rule. Prompt processing (prefill) handles many tokens at once. With a batch of tokens, each weight read from memory is reused many times, so the work becomes compute-bound instead of memory-bound. An H100 has a huge amount of compute, and a dense model keeps all of it busy with large, regular matrix multiplications. The MoE router splits the batch across experts, which leaves the GPU's compute less evenly used.

On a laptop GPU that compute headroom is gone, and MoE wins every axis again, including prefill (3.28x on Metal). For edge devices the lesson is simple: edge hardware is bandwidth-poor, so sparsity pays.

Does extreme quantization destroy quality?

Quantization reduces bytes per weight, which also helps speed. The cost shows up in quality, so it must be measured. On GPQA Diamond (198 questions, greedy decoding):

The POCKET-26B row is the interesting one for edge builders: a mixed-precision 2-bit build keeps almost all of the base score because the most quality-critical weights stay at higher precision. A plain uniform 2-bit quantization of the same model collapses to about 44% according to its card.

How do I pick threads and files on my own machine?

From the published runs, three practical rules:

# CPU only: use the physical core count, not every hardware thread
llama-cli -m POCKET-35B-Q2_K.gguf -p "Summarize this paragraph." -ngl 0 -t 8

# measure your own bandwidth-bound speed the same way the card does
llama-bench -m POCKET-35B-IQ1_M.gguf -p 128 -n 64 -ngl 0 -t 16

• Threads: set -t to physical cores. Oversubscribing threads can slow generation sharply, because threads fight over the same memory channels.
• File size vs RAM: Q2_K (13 GB) fits a 16 to 24 GB machine; IQ1_M (8.2 GB) fits 16 GB; Q4_K_M (21 GB) wants 32 GB.
• 18 GB Macs: run Q2_K on the CPU (-ngl 0); the card reports 19.5 tok/s on the M3 Pro CPU.

FAQ

Why is CPU inference memory-bound? Each generated token requires reading the model's weights from RAM, and RAM bandwidth is far lower than the CPU's arithmetic throughput, so the CPU mostly waits for data.

Why does a bigger MoE model run faster than a smaller dense one? Speed depends on bytes read per token. A sparse MoE reads only the active experts (8 of 256 per layer for POCKET-35B), so it can read fewer bytes than a smaller dense model that reads everything.

When does a dense model win? In compute-bound work such as prompt processing on a high-end GPU, where batched tokens reuse each weight many times. On an H100, Bonsai-27B prefills faster than POCKET-35B.

How much does 2-bit quantization cost? It depends on the method. POCKET-26B's mixed-precision 2-bit build scores 67.2% on GPQA Diamond against 67.7% for its base; a uniform 2-bit build of the same model falls to about 44%.

How popular is POCKET? 1,113,164 downloads in the last 30 days and 3,154,913 in total across POCKET-35B, POCKET-26B and POCKET-Darwin-180B (Hugging Face, 1 November 2026).

Download measurement method: vidraft.net/hf-usage.html

GINIGEN AI works with VIDRAFT, the developer of POCKET, within the same AI ecosystem.