2026-11-18 · ← News
Transformers runs a 180B MoE on a 32GB Mac, trading memory pressure for SSD traffic
A Hugging Face experiment ran the 180 billion parameter Qwen3.8-Flash-Next on an M5 Mac with 32 GB of unified memory. It streams selected experts from SSD and reached 13.9 tokens per second on short generation, about 6% behind the specialised Strata engine.
The image could not be loaded.
Hugging Face has demonstrated an experimental way to run the 180 billion parameter Qwen3.8-Flash-Next on an M5 Mac with 32 GB of unified memory. The system keeps most experts on SSD, loads only those selected by the router and reached 13.9 tokens per second on short generation.
Three bit experts and SSD storage squeezed the model into 32 GB
Qwen3.8-Flash-Next has 48 layers with 512 experts in each layer and activates 10 experts per token. The experts occupy 121 GB in FP8, while the model also carries a 51 GB n-gram embedding table. Conventional offloading alone could not fit the setup.
The experiment quantised the experts to three bits and stored their 49 GB on disk. It kept a 12 GiB device cache, reduced dense layers to eight bits and read n-gram table rows from SSD as needed. Memory use was about 18 GiB after loading and peaked at 25.4 GiB with a 6,000 token prompt. Weight conversion took roughly five hours.
The router turned SSD into another tier of memory
The standard Transformers path through Accelerate moves an entire layer's expert bank even when the router uses only 8 or 10 experts. The prototype instead maintains a bounded cache and reads only the selected experts. It also runs routers for the next two layers early to predict what should be prefetched.
Its central trick is a pool of preallocated slots. Expert IDs are remapped to those slots, allowing the existing grouped_mm operation to treat the cache like a normal expert bank. The design avoids a custom expert matrix multiplication implementation and reuses the fast path already available in Transformers.
Impressive capacity depends on a prototype and a carefully tuned Mac
On the same M5, the final build reached 13.9 tokens per second for short decode against Strata's 14.8. Prefill on a 4,000 token prompt was slightly faster at 228 tokens per second against 220, while decode after a 6,000 token document reached 12.4 against 13.7. The author still considers roughly 14 tokens per second too slow for his coding agent workflow.
The experts_offload feature remains in pull request #50717 and its API is still under discussion with Transformers maintainers. The result also relies on a model specific package, three bit quantisation, Metal kernels and special handling for the n-gram table. It is a measured experiment on one configuration rather than a finished general feature in a standard Transformers installation.
Adoption hinges on broader hardware tests and the pull request landing
The first signal to watch is the status of the pull request and the shape of a stable API for models with the standard expert layout. Tests across other Apple chips, SSDs and MoE architectures matter too, because cache hit rates and storage stalls can change substantially.
Local deployments should measure the whole workflow: conversion time, disk capacity, prefill speed, long generation and the quality effect of quantisation. The experiment establishes that a 180 billion parameter model can operate within a 32 GB machine. Broader measurements will determine which workloads justify five hours of preparation and generation near 14 tokens per second.
Lilith's verdict
Hugging Face fitted a 180B model into a 32GB Mac by leaving most of the weight waiting on SSD. It is a clever moving trick, but at 14 tokens per second the coding agent is still carrying the boxes one at a time.