slotstream
What it is
Runs Qwen3.8-Flash-Next — a 125B MoE that is 104GB even at 4-bit — on Apple Silicon with a fraction of that in RAM, by streaming mixture-of-experts weights from SSD into a fixed cache of expert slots. MLX and Swift, a CLI plus Ollama- and OpenAI-compatible APIs, vision input included; a one-line installer pulls the ~88GB model. The README's own numbers: about 12 tokens/sec warm on a 48GB M5 Pro, and a technically-it-runs 1.41 tokens/sec on a 16GB M2 mini.
Featured in
Issue #11 — Mark Your Own Homework · — the full issue has that week's other tools, reads, and big-lab releases.
More local AI tools we've covered
- Magnitude Local inference engine for coding agents that tunes its kernels to your exact hardware before running a model.
- Cactus Needle 3 A foundation model for phones, wearables, and microcontrollers, built for automation rather than chat: tool calls, structured extraction, and embeddings. The trick is an 'intelligence ladder' — one set of weights where every depth from 2 to 20 layers is a usable model of its own, shipped as 2-bit binaries between roughly 9 and 29 MB — so you choose accuracy versus footprint per device without training a family. Open weights on Hugging Face under Apache-2.0; Cactus sells a one-off $19 hosted fine-tune.
- Desert Ant Labs A new European lab, launched this week by Paul Veugen, shipping eighteen specialised on-device models for audio, vision, and text: transcription it claims runs 4.7× faster than Whisper, audio enhancement in 9MB, PII redaction across 27 languages in 12MB, language identification in 2MB. Everything runs on the phone's neural engine or in-browser WASM; Swift, Kotlin, and JS SDKs; free up to 100,000 monthly active devices, no tokens, no logins.
- kimi-k3-in-c A portable C99 implementation that runs the 2.78-trillion-parameter Kimi K3 on one CPU in 8.24 GB of RAM. Six C files, AVX2 and OpenMP, no GPU and no BLAS — dormant experts stream off disk while 16 of 896 fire per token.