brain
projects·playbook

Local-first architecture (for latency & privacy)

The pattern

Run the latency-critical and privacy-sensitive parts of the pipeline on-device. Push only what benefits from cloud scale — and even then, make it optional.

Projects exemplifying this

  • wispr-flow — fully local. Whisper on Apple MLX + Dolphin3 via Ollama. No cloud, sub-300ms end-to-end. The entire value prop is "nothing leaves your machine."
  • vera — partially local. STT and TTS run on-device (no internet for speech); the planning LLM (Groq / OpenRouter) is hosted. Hybrid: local where latency / privacy matter, hosted where reasoning quality matters.

Why local-first wins here

  • Latency budget: voice UX has a ~500ms feel-instant threshold. A single cloud round-trip eats 200-400ms by itself. Doing STT locally and only sending text to the cloud preserves budget for actual reasoning.
  • Privacy: dictation includes sensitive material (passwords, medical, personal). Cloud transcription is a non-starter for many real users.
  • Offline resilience: local-first products keep working when the user's network is bad, the API is down, or they're on a plane. Hosted products don't.

Apple Silicon specifics (from wispr-flow)

  • MLX beats PyTorch on Apple Silicon for narrow inference: 3-4x faster because MLX skips the CPU→GPU memory copy that PyTorch forces even on unified-memory machines.
  • INT4 quantization is worth the quality hit for latency-critical paths.
  • Multi-threading beats asyncio for real-time audio. Python asyncio doesn't give hard real-time guarantees; dedicated threads with sync primitives do.

Hybrid pattern (from vera)

  • Local: STT, TTS, health data (HealthKit is on-device).
  • Hosted: the LLM that does actual planning and reasoning.
  • Fallback chain for the hosted part (Groq → OpenRouter) but no fallback needed for the local part — it's already on the device.

When this playbook misleads

  • Don't force local for things that aren't latency-critical or privacy-sensitive. The engineering cost is high; the payoff is zero if the user doesn't notice.
  • Don't assume local is always faster. A well-tuned cloud model can beat a local one on cold-start or for complex reasoning. Measure.
  • Don't over-index on privacy rhetoric if the product doesn't need it — "local-first" is a choice, not a badge.

Related