brain
projects·playbook

AI model choice: local vs hosted, fallbacks, structured output

Three sub-patterns from the corpus

1. Local models for latency-critical on-device work

  • wispr-flow — Whisper via Apple MLX, Dolphin3 via Ollama, fully offline, sub-300ms end-to-end.
  • vera — on-device STT + TTS for voice capture (even though the planning LLM is hosted).

Rule: if the user experience has a sub-second budget and the modality is voice, local STT/TTS is non-negotiable. Network round-trips are too slow and too unreliable.

2. Hosted models for reasoning, with multi-provider fallback

  • vera — Groq llama-3.3-70b primary, OpenRouter fallback when rate-limited.
  • amboras — OpenRouter → GPT-4o-mini.

Rule: pick a primary provider for speed/cost (Groq is the current sweet spot for fast, cheap Llama). Keep a secondary provider wired in from day one for rate limits and outages. Treat the fallback as a first-class path, not an emergency hatch.

3. Structured output via tool calls, not free-form JSON

  • amboras — tool_choice forces a typed StoreConfig. No parsing risk, no partial failures.
  • orqys — multi-agent system where each agent returns typed outputs to the next.

Rule: if your downstream pipeline consumes the LLM output programmatically, pay the upfront schema cost. You save an entire class of "the model returned almost-valid JSON" bugs. Same schema can drive initial generation and iterative editing (Amboras does this — AI editor patches the same StoreConfig the generator produced).

Provider notes (snapshot as of this ingest, 2026-04-18)

  • Groq: primary for speed/cost on Llama-family models (Vera).
  • OpenRouter: universal fallback + access to GPT-4o-mini (Vera, Amboras).
  • OpenAI direct: embeddings and chat (Portfolio OS, Synapse).
  • Ollama: local model serving across Apple Silicon (Wispr Flow, Synapse portfolio narrative).
  • Apple MLX: hardware-accelerated on-device inference for Apple Silicon (Wispr Flow).

When this playbook misleads

  • Don't multi-provider for experiments — adds configuration overhead without payoff.
  • Don't force structured output for open-ended creative tasks — typed schemas lose nuance.
  • Don't default to local — hosted is usually faster and better quality; local is a choice for latency/privacy, not a default.

Related