Building on-device voice assistants with QVAC

Every voice assistant you've used sends your voice to a data center.

You speak. Audio uploads. A server transcribes it, a model reasons about it, another server synthesizes a reply, and audio comes back. Your voice — the most identifying biometric you produce casually, hundreds of times a day — makes a round trip through someone else's infrastructure.

QVAC ships a voice assistant capability that closes the loop entirely on-device. Speak, think, respond, without a single byte leaving the hardware in your hand.

Here's how it works and why it's harder (and more interesting) than it sounds.

The pipeline

A voice assistant is three models in a trench coat. QVAC's voice assistant capability is a real-time pipeline chaining:

Transcription (speech-to-text) — turn the user's audio into text
Text generation (LLM) — reason about it and produce a reply
Text-to-speech — synthesize that reply as audio

QVAC provides all three as first-class capabilities, and composes them into one loop. The backends:

Transcription: a customized Whisper engine, or NVIDIA Parakeet
Generation: qvac-fabric-llm.cpp (the llama.cpp fork)
TTS: ONNX Runtime with Chatterbox / Supertonic neural TTS models

What makes this notable is that you're not integrating three separate libraries with three lifecycles and three sets of platform quirks. It's one SDK, one worker, one memory model.

Why "on-device voice" is a bigger deal than on-device text

Text is what you chose to write. Voice is what you sound like.

Your voice is biometric. A voice sample identifies you, and increasingly can be used to synthesize you. Every cloud voice assistant maintains, by architectural necessity, a pipe carrying your voiceprint to a server.

Voice assistants are always listening. That's the product. The always-on microphone is only tolerable if you trust what's on the other end of it — and "trust" is doing an enormous amount of work in that sentence when the other end is a company's data center.

Latency is the whole experience. A voice interface with a network round trip between each turn feels sluggish in a way text chat doesn't. Conversation has a rhythm; hundreds of milliseconds of round-trip breaks it. Local inference removes that entirely.

It works where you are. In a tunnel. On a plane. In a basement. At a campsite. Cloud assistants become paperweights the moment connectivity drops — which is precisely when hands-free assistance is most useful.

The engineering realities

Having built voice features on QVAC for the upcoming Local Money (voice-logged expenses), here's what actually determines whether this feels good or bad.

Three models, one memory budget. You now have an ASR model, an LLM, and a TTS model, all wanting to be resident. On a phone, you cannot naively keep all three warm forever. This is where model lifecycle stops being theory. Load and unload deliberately; use QVAC's runtime suspend/resume around app background/foreground.

Latency is a chain, not a number. Total perceived latency is transcription + generation + synthesis. Each stage adds. The single most effective lever is streaming: don't wait for the LLM to finish before starting TTS. Begin synthesizing the first sentence while the model is still writing the second. This is the difference between "responsive" and "awkward pause."

Time-to-first-audio is the metric that matters. Not total latency. The user forgives a long answer; they don't forgive a long silence. Optimize for how quickly sound starts.

Small models, tight prompts. An on-device LLM is smaller than a frontier cloud model. For a voice assistant this matters less than you'd think, because voice interactions are usually short, contextual, and task-shaped. Good retrieval and a well-scoped system prompt beat raw model size. (If you need heavy reasoning, see delegated inference below.)

Cancellation is a feature. Users interrupt. They change their mind mid-sentence. QVAC lets you cancel in-flight inference by requestId — wire this to barge-in, or your assistant will keep talking over someone who's already moved on.

The delegated inference twist

Here's the capability nobody else has.

The standard complaint about local voice assistants is that the small on-device model gives shallow answers. QVAC's delegated inference offers a third path: run transcription and TTS on the phone — they're light — and delegate the heavy generation step to a peer device, like your desktop, over an encrypted P2P connection.

You get a large model's reasoning with a phone's microphone, and nothing touches a cloud server. When no peer is available, fall back to the local model. Same code, graceful degradation.

That architecture — light on-device, heavy on a peer you own — is genuinely new, and voice assistants are its most obvious application.

What to build

A private assistant. The obvious one. An always-listening assistant you'd actually be comfortable having always listening.

Hands-free data entry. Say "twelve dollars on lunch" and have it logged, parsed, and categorized — entirely on-device. This is what Local Money (coming soon) will do, and it's the feature that convinced us the voice loop was ready.

Accessibility tools. Voice interfaces for users who can't use a screen, working offline, without a subscription, without a cloud dependency.

In-car and field applications. Anywhere connectivity is unreliable and hands are busy.

Voice for robots. Pair the voice loop with VLA and you have a robot you can talk to, that hears and reasons and acts without leaving the room.

The honest limits

On-device TTS quality is good but generally not yet indistinguishable from the best cloud voices. Small LLMs will sometimes fumble a question a frontier model would handle. Battery and thermal budgets are real, and a continuously-running voice loop on a phone will make itself known.

QVAC is pre-1.0. The voice assistant capability composes three capabilities that are each evolving. Verify against the docs before shipping.

But the trade is clear-eyed: slightly less polish, in exchange for a voice assistant that never sends your voice anywhere. For a device that listens to your home, that seems like a very good deal.

See also: The Complete Guide to QVAC's Capabilities, On-Device Transcription: Whisper vs Parakeet, and On-Device Text-to-Speech with QVAC.