A Local-Embedding RAG Chatbot, a Report the LLM Can't Edit, and Serverless Surprises

Quotient has three places where an LLMLLMLarge language model — a neural network trained on large amounts of text to predict the next token, used for generation, summarisation, classification and more. or its neighbours caused real design decisions: the report, the chatbot, and getting both to run on a serverlessserverlessA hosting model where the platform runs your code on demand, per request, and manages servers and scaling for you; instances are short-lived and keep no local state. host. None of these are novel, but each had a trade-off worth writing down.

The Report: Facts Are Deterministic, Prose Is Optional

The report is generated by rules from the scores — which domains are strengths, what each domain measures, practice tips. That output is complete on its own. An optional second pass sends it to a model through OpenRouter to make the writing feel warmer.

The constraint that matters: the model may only replace narrative fields — the overview and each section's "what it measures / how you did / everyday example". Scores, percentiles, strength flags, tips and the disclaimer are never sent back for rewriting. If the call fails or times out, the deterministic text is shown. That gives a clear failure mode ("plain but correct") instead of a risky one ("fluent but wrong about someone's score").

One incidental finding: the rewrite uses a plain fetch POST rather than the openai SDK. Calls through the SDK silently hung inside next dev until its own client timeout; a direct request behaved normally. I didn't dig into the root cause, so I'd call it "worked around" rather than "understood".

The Chatbot: Retrieval Without a Second Vendor

For grounding, the chatbot retrieves from a small markdown knowledge base — one file per domain plus an overview. The pipeline:

1. Ingest: split the markdown by paragraph into overlapping chunks (150 characters of overlap) and embed each with Xenova/all-MiniLM-L6-v2 via @xenova/transformers, which runs locally and produces 384-dimension vectors. The model downloads once (roughly 80 MB) and is cached as a singleton.
2. Store: vectors go into LanceDB, an embedded vector databasevector databaseA database built to store embeddings and run fast similarity search over them, usually with an approximate nearest-neighbour index. that lives in a local directory — no server to run.
3. Query: embed the question, take the top 5 matches, and place them in the system prompt along with the user's own results.

EmbeddingembeddingA list of numbers (a vector) representing a piece of text, an image or audio so that similar meanings land close together — the basis of semantic search. is done sequentially on purpose, to avoid memory spikes. The system prompt instructs a warm tone, short answers, no clinical language and a repeated reminder that it's a self-insight tool, and it injects the retrieved snippets with a short truncation so the prompt stays bounded. Streaming responses are parsed chunk by chunk, with malformed chunks skipped rather than crashing the stream.

The model choice is a configuration value. I started from a plan that named one provider and switched to OpenRouter for both the rewrite and the chat, recording the change in a decision log so the "why" didn't vanish.

Serverless Surprises

SQLite on a read-only filesystem. Locally, DATABASE_URL=file:./dev.db is fine. On Vercel, the filesystem is read-only outside /tmp, so writes fail. For the hosted preview, the database is bootstrapped from a bundled snapshot into /tmp at runtime. It is a reasonable demo trick and explicitly not a production database — data there is ephemeral.

Environment variables are per environment. The chatbot said "no API key set" in production because the key existed only in local .env. Adding the variables to both Preview and Production fixed it. Obvious in hindsight; easy to forget.

PDF export and bundle size. Plain puppeteer downloads a full Chromium that doesn't fit a serverless function. The fix is puppeteer-core plus @sparticuz/chromium, a build of Chromium trimmed for functions (on the order of tens of MB). A related fix: render the report HTML on the server and hand that to the headless browser, instead of having the headless browser load the live page.

Takeaways

• Keep a deterministic path that is correct on its own, and treat the LLM as optional polish.
• Local embeddings plus an embedded vector store keep a small RAGRAGRetrieval-Augmented Generation — fetching relevant passages from a document store at query time and handing them to a language model as context, so its answer is grounded in those sources. project self-contained.
• Test on the real host early; filesystem, env vars and binary size are all invisible until you deploy.