The archive · AI & Models · Technical decision · 2026
Swiftlet streams an 80B Qwen in 4.3 GB of RAM on a Mac and 35B on iPhone
An open Swift+Metal runtime runs the real Qwen3-Next-80B in 4.3 GB of peak RAM on a Mac and a 35B on an iPhone by streaming MoE experts from disk.
Swiftlet (leonickson1)
What the business is
An open-source (Apache-2.0) Swift and Metal inference runtime for Qwen3-Next and Qwen3.5/3.6 MoE models on Apple Silicon, offering a Swift package, CLI, OpenAI-compatible local server and the Priv AI iOS app that embeds SwiftletCore.
How it started
Posted by developer leonickson as a Show HN on 2026-08-03, Swiftlet was built from scratch in about 10,000 lines of Swift and Metal, written with Claude Code against mlx-lm references and validated layer by layer, after earlier expert-streaming projects such as TurboFieldfare proved the thesis for Gemma.
What happened
The thread (312 points, 141 comments) debated prefill limits, SSD read-disturb and whether disk-streamed speeds could ever be practical. The author conceded prefill is the weak spot and put batched expert reads for long prompts atop the roadmap, alongside a phone speed pass and a 6-bit container tier.
How it ended up
Working end to end as of 2026-09-05: the runtime ships qpack containers for 35B, 80B and 397B Qwen models, and the open-source Priv AI iOS app embeds SwiftletCore so end users can download a 35B model and chat fully on-device.
Background
Swiftlet is an open-source (Apache-2.0) Swift and Metal inference runtime for the Qwen3-Next and Qwen3.5/3.6 Mixture-of-Experts model families on Apple Silicon. It ships as a Swift package, a CLI, an OpenAI-compatible local server, and the Priv AI iOS app, which embeds SwiftletCore so end users can download a model and chat on-device.
The runtime's insight is that MoE models activate only a few billion parameters per token, so the rest of the weights do not need to stay in RAM. Swiftlet keeps the dense core resident, stores the routed experts on disk as fixed-stride .qpack blobs, and streams each token's experts in with a bounded LFU-plus-recency cache.
The result is that the real 4-bit Qwen3-Next-80B (42 GB on disk) runs in about 4.3 GB of peak RAM at 4.5-5 tok/s on an M5 Mac, and a 35B model runs on an iPhone 17 in about 2.5 GB at roughly 1 tok/s. A community benchmark verified an 8-bit 80B serving from 6.9 GB on a 64 GB M4 Max.
Developer leonickson posted Swiftlet to Hacker News on 2026-08-03, where it drew 312 points and 141 comments. As of 2026-09-05 the runtime works end to end; the roadmap targets batched prefill, faster phone decode and a 6-bit container tier.
What has to be true
- MoE sparsity is a real door: only about 3B of an 80B model's parameters activate per token, so a runtime that fetches just the routed experts from disk can slash peak RAM by an order of magnitude.
- One fetch, one read: fixed-stride .qpack blobs turn expert loading into a single SSD pread, and runtime-compiled Metal shaders keep the forward pass on the GPU without a build-time toolchain.
- Publishing honest limits — prefill at decode speed, about 1 tok/s on a phone — set expectations so the thread argued about SSD physics instead of dismissing the demo as fake.
- An installable app made the claim testable: Priv AI embeds SwiftletCore, so anyone with an iPhone 17 can download a 35B model and verify the runtime themselves.
What can be applied
Running a real frontier-class model on ordinary hardware is a stronger argument than a demo: an 80B in 4.3 GB changed what local AI meant, and publishing slow-prefill trade-offs built credibility.
Aftermath
As of 2026-09-05 Swiftlet works end to end and is still a solo, unfunded open-source project. The README documents validated output for 35B, 80B and 397B containers, community benchmarks contributed by @Avicennasis, and an explicit roadmap: batched prefill (the acknowledged weak spot), a phone speed pass, and a 6-bit tier. Priv AI on the App Store embeds SwiftletCore with streaming models under an Experimental setting. No company, funding round or press coverage beyond technology blogs has been reported in the sources reviewed.
Sources
- Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
- Swiftlet: Run an 80B Qwen on a Mac or iPhone
spotted an error? The archive wants to know.
Your turn
You just read one. Describe what you are building, and see who is betting on the same thing.
Free account · 3 free questions · no card