The archive · AI & Models · Technical decision · 2025-2026
RunAnywhere bets hand-written Metal kernels make on-device AI the default
YC W26 startup skips generic runtimes, hand-writes Apple Metal and Qualcomm NPU kernels, open-sources the SDKs, and sells a fleet control plane.
RunAnywhere
What the business is
On-device AI infrastructure: hand-written inference engines for Apple GPUs (MetalRT) and Qualcomm Hexagon NPUs (QHexRT), open-source SDKs for iOS, Android, React Native, Flutter, TypeScript, and C++, plus a hosted console for model rollouts and fleet observability.
How it started
Sanchit and Shubham founded RunAnywhere in 2025 and joined YC's Winter 2026 batch. Their premise was that demoing on-device AI is easy but shipping it is brutal: teams chain STT, LLM, and TTS, each stage adds latency, and most fall back to cloud APIs not because local models are bad but because local inference infrastructure is.
What happened
They built MetalRT, an inference engine with custom Metal compute shaders, zero runtime allocations, and one unified engine for LLMs, speech, and TTS, then open-sourced RCLI, an end-to-end voice pipeline. Published M4 Max benchmarks: Qwen3-0.6B at 658 tok/s vs 295 for llama.cpp and 552 for MLX, 6.6ms time-to-first-token, 70 seconds of audio transcribed in 101ms, and 178ms TTS. At the March 2026 launch the open-source SDKs had roughly 10.1k GitHub stars.
How it ended up
Still live. RunAnywhere expanded to a second engine, QHexRT, running LLM, VLM, STT, TTS, and embeddings fully on Qualcomm Hexagon NPUs, and sells a control plane for hybrid routing - local-first with policy-based cloud fallback - across device fleets.
Background
RunAnywhere is a YC Winter 2026 startup building on-device AI infrastructure. Instead of wrapping generic runtimes, it hand-writes inference kernels for consumer silicon - MetalRT for Apple GPUs and QHexRT for Qualcomm Hexagon NPUs - and open-sources the runtimes, SDKs, and a CLI voice pipeline under the RCLI project.
Founders Sanchit and Shubham started from a simple observation: most teams fall back to cloud APIs not because local models are bad, but because local inference infrastructure is. In a voice pipeline, latency compounds - three chained models, and if each adds 200ms the user waits 600ms before hearing a word.
At its March 2026 Launch HN, RunAnywhere published M4 Max numbers: Qwen3-0.6B decoding at 658 tok/s versus 295 tok/s for llama.cpp and 552 for MLX, 6.6ms time-to-first-token, 70 seconds of audio transcribed in 101ms, and 178ms TTS. The open-source SDKs already had about 10.1k GitHub stars at launch.
The company now pairs the research engines with a hosted console: OTA model updates, policy-based hybrid routing that tries local first and falls back to the cloud, and observability across thousands of devices, targeting iOS, Android, macOS, Windows, Linux, web, and embedded.
What has to be true
- Voice pipelines make latency compound: three chained models mean each stage must be fast on one device, which generic runtimes with framework overhead cannot deliver.
- The bet is that silicon is not the problem - Apple GPUs and Qualcomm NPUs are already fast, but generic runtimes leave them idle, so hand-written kernels are the unlock.
- Publishing every benchmark makes the performance claim the product: engineers can verify 658 tok/s before adopting, which is stronger than marketing.
- Open-sourcing RCLI and the SDKs is the distribution wedge: developers install, measure, and then pay for the control plane and fleet operations.
- Hybrid routing hedges the bet - local-first with policy-based cloud fallback means on-device AI does not have to win every task on day one.
What can be applied
When the incumbent abstraction is the bottleneck, go below it: RunAnywhere hand-writes kernels because generic runtimes leave consumer silicon idle, and publishes every benchmark.
Aftermath
As of September 2026, RunAnywhere remains active with a five-person team in San Francisco. The product line has grown from MetalRT to include QHexRT for Qualcomm Hexagon NPUs, one shared C++ core with six SDKs, and a hosted console for model management, policy routing, and telemetry. The engines and SDKs are open source, and the company frames its market as enterprise on-device AI across health, finance, voice, and agents.
Sources
- Launch HN: RunAnywhere (YC W26) - Faster AI Inference on Apple Silicon
- RunAnywhere: The default way of running on-device AI at Scale
spotted an error? The archive wants to know.
Your turn
You just read one. Describe what you are building, and see who is betting on the same thing.
Free account · 3 free questions · no card