Developer Tarek Ziadé published an experiment running Qwen3.8-Flash-Next with approximately 180 billion parameters on a MacBook with 32 GB of unified memory. The breakthrough hinges on the Mixture of Experts (MoE) architecture and streaming active neural blocks directly from an SSD. In the final iteration, generation reached 13.9 tokens per second, closely matching the specialized Strata inference engine on the same hardware.
The result is notable because the majority of the model is never loaded into system RAM all at once. However, a common misconception should be addressed immediately: a 180B model does not "fit into 32 GB" in the conventional sense. A significant portion of its weights remains on disk, and execution depends heavily on quantization, aggressive caching, and SSD bandwidth. This represents a research prototype rather than an off-the-shelf way to deploy any massive model on consumer laptops.
Why 180 Billion Parameters Typically Demand Far More Memory
In dense neural networks, processing each token engages virtually every parameter. For a 180-billion parameter model, even an 8-bit quantized representation would require roughly 180 GB of memory just for weights, excluding activation buffers, KV cache, and runtime overhead. In 4-bit precision, that footprint hovers around 90 GB, though practical deployments with metadata and mixed precision often exceed that.
The dynamics change under Mixture of Experts (MoE). Instead of activating all parameters, the network houses numerous modular sub-networks — experts — with an internal router activating only a sparse subset for each token. In Qwen3.8-Flash-Next, the author notes 48 layers with 512 experts each, but only 10 experts activate per layer per token. While shared attention layers and routers must remain constantly active, selected expert weights can theoretically be retrieved on demand.
Crucially, experts are abstract mathematical sub-networks, not distinct "coding" or "biology" specialists. Their specialization emerges naturally during gradient descent and rarely mirrors human occupational categories.
How the Streaming Experiment Works
The implementation relies on expert streaming. The core strategy is maintaining a strictly bounded in-memory cache on the fast device while keeping inactive expert weights on disk. As the router selects experts, the runtime checks the cache, streams missing weights from the SSD, and proceeds with computations. The higher the cache hit rate and the more accurate the prefetching, the less time the GPU spends idling on I/O.

Figure 1. Editorial diagram of SSD streaming and limited cache operation. Figures based on Tarek Ziadé's technical report; diagram simplifies per-layer details.
In the default configuration, Qwen3.8-Flash-Next's expert layers take around 121 GB in FP8. To run on the laptop, the author quantized them down to a 3-bit format, fitting the weights into roughly 49 GB on disk, and allocated a 12 GiB cache pool in RAM. Dense layers were quantized to 8 bits, occupying roughly 5.4 GB. Another hurdle was a 51 GB n-gram table, requiring custom on-disk row-streaming logic.
Following initialization, baseline footprint measured around 18 GiB, rising to a peak of 25.4 GiB when processing a 6,000-token prompt. On a 32 GB machine, background browsers and applications had to be terminated to prevent macOS from aggressively thrashing swap memory. Preparing the quantized weight files took roughly five hours.
Modifying the Transformers Runtime
Standard offloading routines in Transformers and Accelerate were ill-equipped for sparse MoE pipelines. When evicting an entire layer, they frequently swap out dozens of inactive experts alongside needed ones. Once RAM saturates, continuous disk thrashing causes inference throughput to plummet.
To overcome this, the author authored an experimental experts_offload implementation in a pull request to Transformers. Rather than evicting broad layer blocks, the engine preallocates a fixed slot pool in memory. A mapping table correlates expert IDs to active slots, preserving compatibility with optimized batched matrix multiplication (grouped_mm) kernels.
Disk reads are parallelized, while adjacent layers attempt to predict forthcoming weight requirements. The author also implemented custom Metal kernels for architectural paths lacking efficient macOS implementations and adjusted GPU dispatch scheduling. These granular optimizations were vital; simply issuing read calls to an SSD does not yield usable throughput.
The open-source prototype, conversion utilities, and detailed benchmarks are available in the moe-expert-streaming Hugging Face Space and on GitHub. The PR to Transformers remains under active discussion with maintainers and should not be considered a stable library feature.
Benchmark Results: Tokens Per Second
Testing on an identical Mac M5 with 32 GB unified memory, the author compared the optimized Transformers pipeline against Strata 0.1.60 across identical prompts. In final benchmarks, Transformers clocked 13.9 tokens/s generating 200 tokens, closely trailing Strata's 14.8 tokens/s. During 4,000-token prompt prefill, Transformers achieved 228 tokens/s versus Strata's 220 tokens/s. Following a 6,000-token document prompt, generation speeds stood at 12.4 and 13.7 tokens/s respectively.

Figure 2. Two benchmarks on an identical Mac M5. Prompt processing and generation speeds are displayed separately as distinct operations.
Early iterations were considerably more sluggish. Initial tests began at around 10 tokens/s before the author fine-tuned cache eviction, quantization, speculative loading, and GPU dispatch. Ultimately, the gap relative to Strata narrowed to roughly 6% on short response generation.
Nevertheless, a sustained rate of 14 tokens/s remains modest for demanding automated coding agents. The author explicitly notes that this speed falls short of his daily production workflow, keeping him reliant on high-end remote compute clusters. The experiment proves technical feasibility rather than an outright replacement for cloud APIs.
Comparison with llama.cpp and vLLM
The report includes secondary benchmarks on another MoE architecture, Qwen3.6-35B-A3B, tested on an A10G GPU with 24 GB of VRAM. There, Strata with Q8_0 hit 82–88 tokens/s, while llama.cpp in --fit on mode registered 63–64 tokens/s. Various vLLM configurations with CPU RAM offloading posted different trade-offs, whereas vanilla Transformers + Accelerate lagged substantially.
Crucially, these figures cannot be conflated with the 180B Mac benchmark: they evaluate a distinct model size, hardware tier, quantization schemes, and execution backends. The takeaway lies in architectural positioning: llama.cpp excels when experts fit comfortably in system RAM for CPU inference, while vLLM targets multi-tenant enterprise serving. When system RAM itself is insufficient, selective SSD weight streaming with aggressive hit caching emerges as the only viable alternative.
As MoE parameter counts expand, this design could yield major memory dividends, though SSD latency, cache misses, and long context windows can still introduce bottlenecks.
Reproducing the Setup
This configuration requires Apple Silicon / MPS, the specific Qwen architecture, and custom-quantized weights. Beyond 32 GB of unified memory, an NVMe SSD with ample free storage is necessary for weight storage and intermediate formatting. Running standard pip install transformers will not enable this pipeline; reproduction demands the experimental PR branch, dedicated scripts, and custom Metal kernels.
Developers interested in testing should consult the Hugging Face technical overview and the model weight repository. Practical adoption depends heavily on expectations: while 14 tokens/s is acceptable for conversational prompts, iterative software agent loops will quickly feel constrained.
Why This Milestone Matters
Local AI runtimes have long focused on aggressive quantization and CPU/RAM offloading. This project demonstrates the next evolution: treating SSD storage as an active memory tier for sparse MoE models, coupled with predictive prefetching. This approach could significantly broaden access to massive open-weight models on memory-constrained local workstations.
Still, parameter scale alone does not guarantee model intelligence, nor does runnable execution promise snappy performance. The project's primary accomplishment lies in closing the gap between general-purpose Transformers code and dedicated engines like Strata, establishing a template for sparse offloading. Long-term impact will depend on whether upstream frameworks adopt these patterns and support broader model families.
