On-Device RAG Over Your Own Notes: How We're Building Chat-With-Your-Notes on QVAC

"Chat with your notes" is easy to demo and hard to ship — especially if you refuse to send the notes anywhere.

The usual way to build retrieval-augmented generation (RAG) is: embed the documents with a hosted embeddings API, store the vectors in a cloud vector database, and call a hosted LLM at query time. Three server round trips, three places your private notes now live that aren't your device.

We're building Local Notes' chat feature with zero of those. Here's how the pipeline works on QVAC, and where the real engineering effort is going. This is written for developers evaluating QVAC or local-first AI generally — if you just want to know what the feature will do, the app's page (coming soon) is the better read.

Note: code below is illustrative pseudocode to show the shape of the pipeline, not a copy-paste drop-in. Check the QVAC docs for current API signatures.

The constraint that shapes everything

One rule drove every decision: no note content leaves the device, ever. Not to embed it, not to search it, not to answer questions about it.

That rule rules out the entire standard RAG stack. No OpenAI embeddings endpoint. No Pinecone. No hosted LLM. Every stage — chunking, embedding, indexing, retrieval, generation — has to run on the phone or laptop the user is holding. QVAC is what makes that feasible from a single codebase, because it exposes embeddings, vector search, and LLM completion through one interface that runs the same on iOS, Android, and desktop.

The pipeline, stage by stage

1. Chunking

Notes aren't uniform. A note might be three words or three thousand. We chunk on structure first (headings, blocks) and fall back to a token-bounded splitter with overlap so a to-do buried mid-note stays retrievable. Chunking is pure application logic — no model needed — but it's where retrieval quality is quietly won or lost.

2. Embedding — on device

Each chunk becomes a vector using QVAC's on-device embeddings capability. The important part: the text never goes to a network call to become a vector.

import { loadModel, embed, unloadModel } from "@qvac/sdk";

const embedder = await loadModel({ modelType: "embeddings" });

async function embedChunks(chunks) {
  const vectors = [];
  for (const chunk of chunks) {
    const { embedding } = await embed({ modelId: embedder, input: chunk.text });
    vectors.push({ id: chunk.id, embedding, meta: chunk.meta });
  }
  return vectors;
}

We embed incrementally — when a note changes, only its chunks get re-embedded — so the cost is amortized across editing rather than paid in one big batch.

3. Indexing and retrieval — on device

Vectors live in a local index on the device. At query time we embed the user's question with the same model, run similarity search locally, and pull the top-k chunks. QVAC's RAG tooling handles the ingestion-and-search loop; you can also bring your own local vector store if you want tighter control over the index.

The scope selector planned for Local Notes — everything / one notebook / one note — will just be a filter on this retrieval step. "Ask this note" will be the same pipeline with the candidate set narrowed to one note's chunks.

4. Generation — on device

The retrieved chunks get assembled into a prompt with the user's question, and a local LLM generates the answer via completion.

const llm = await loadModel({ modelType: "llm" });

const context = topChunks.map(c => c.text).join("\n\n");
const response = completion({
  modelId: llm,
  history: [
    { role: "system", content: "Answer using only the user's notes provided as context. If the notes don't contain the answer, say so." },
    { role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` },
  ],
});

Streaming the response token by token matters a lot for perceived speed on-device — you want the first words on screen while the rest generates, exactly as you would with a cloud model.

What was actually hard

Model lifecycle and memory. On a phone you can't keep an embeddings model and an LLM resident forever. Deciding when to load, when to unload, and how to keep the app responsive during a model download is most of the real work. QVAC's download lifecycle controls (pause/resume, sharded models) matter here more than any single inference call.

First-run experience. With no server, the model download on first launch is your onboarding. We leaned into QVAC's download-a-model-on-first-run pattern rather than hiding it — a short, honest "setting up on-device AI" beats a fake instant start that stalls later.

Retrieval quality with small models. Local models are smaller than frontier cloud models. Good chunking and tight retrieval do more for answer quality than model size — a well-scoped context makes a small model punch above its weight.

Latency budget. No network latency is a gift; on-device compute latency is the new budget. Warm models, incremental embedding, and streaming are the three levers that keep it feeling instant.

Why we're making this choice

We could ship chat-with-your-notes in a weekend with a hosted stack. We'd also create a database full of strangers' private notes — the exact thing our future users are choosing us to avoid.

Building it on QVAC means the privacy promise is structural, not a policy. There will be no server log to subpoena, no vector store to breach, no retention policy to trust. The notes will live on the device, the model will run on the device, the answer will be computed on the device. That's not a feature we can toggle off under pressure — it's the shape of the system.

If you're building local-first AI and weighing whether the on-device RAG path is viable in production: it is. This pipeline is what Local Notes will run on at launch.

---

localhost/AI is building consumer apps where the AI will run entirely on your device, powered by QVAC. See it in Local Notes, coming soon.