Lately I’ve been focused on building local-first AI apps. And no, it’s not about saving money. Compared with cloud services, running locally means living with limited hardware and compute, so you give up a lot of capability. The real reasons are:
- Privacy: in some industries, compliance rules say business data can’t touch the internet.
- Uncensored content generation and processing: no explanation needed.
- Speed: in some scenarios, speed trumps everything. If the requirements don’t change much, a fine-tuned local model is more than enough.
Beyond those three, the bigger factor is that open-source models keep getting better, which is what makes local deployment viable in the first place. It also opens the door to a hybrid setup: handle what you can locally, and hand the rest off to the cloud.
The Stack
Taking my current project as an example, here’s a typical stack:
Development
- Frontend: Next.js
- Agent: eve
- Sandbox (in-memory): just-bash
- Generative UI: json-render
- Document processing: Docling + Python
- Database: pgvector + Drizzle
Testing
- Unit: node:test
- E2E: Playwright
- Evals: eve
- Linting: i18n-check + Biome
Local Models
- Embedder: Qwen/Qwen3-Embedding-0.6B
- Reranker: Alibaba-NLP/gte-multilingual-reranker-base
- LLM: Qwen/Qwen3.6-35B-A3B-FP8
Serving
- Embedding + reranking: HF TEI
- LLM: vLLM
- Hardware: NVIDIA DGX Spark
Q&A
- Why
eve?
Because it’s good, and it’s from Vercel. The frontend is already on Vercel’s Next.js, so it was the natural pick. The developer experience is great too, and evals are built in, which saves a lot of hassle. The real clincher is that it’s a one-stop shop for agent development, sandbox and generative UI included. With advantages that obvious, why would I pick anything else?
Related reading : The Three Camps of Open-Source AI Harnesses
- Why
just-bashfor the sandbox?
First, local-first rules out all the cloud sandbox providers, which leaves self-hostable options like Docker. But I don’t need anything that heavy right now, so the in-memory just-bash was the obvious choice. If I need more later, swapping in Docker won’t be a big deal.
BTW, it’s also from Vercel.
- Why
json-renderfor generative UI?
Also from Vercel, and it supports mcp-app, which I need.
- How did you pick the models?
I need CJK support, so the embedder and reranker pretty much chose themselves.
A quick word on the reranker: I started with Qwen Reranker, but the results never quite cut it. After digging in and talking it over with AI, I realized it wasn’t a BERT-style cross-encoder at all, as I’d simply assumed, and it was a poor fit for my use case. After another round of research, I landed on the current one because it outperforms BGE (which, admittedly, was also the conclusion of a chat with AI).
The LLM was also a natural pick after some research. Why not the new 27B? Cost. This one Spark is already running vLLM + Qwen alongside TEI + embedder + reranker, and I plan to add an image model down the road. After talking it through with AI, I decided to hold off on upgrading. Besides, in practice both speed and quality are acceptable.
- Why not Ollama?
Correction: I do use Ollama. My evals use Ollama + Qwen as the LLM judge (I couldn’t find a decent local jev-style model, so Qwen fills in). But as a server, Ollama isn’t a great choice; vLLM and SGLang have much better reputations. As for vLLM vs. SGLang, I honestly didn’t think too hard about it. Once I realized Ollama wasn’t production-ready, I just went with vLLM.
In a nutshell: Ollama for development, vLLM or SGLang for production serving. For anything more involved, go ask AI. Nobody reads long posts these days anyway 😄.
Gotchas
- After running into a few snags, these are the Docker images we settled on:
- vLLM:
timothystewart6/vllm-gb10 - TEI:
ghcr.io/huggingface/text-embeddings-inference:121-latest
- vLLM:
Both have been optimized or patched for the Spark.
Note: I set this up a while ago (3+ months), so they may no longer be necessary. Think of them as a last resort.
thinkingisn’t free, and it’s no silver bullet.
The word thinking is misleading. It makes it sound like flipping a switch makes the model smarter. It doesn’t. Used poorly, it spits out a pile of rambling before getting to the actual answer, so users feel the system is slow, and the answer might even be wrong! Too much of a good thing.
For most local scenarios, like RAG, image understanding and recognition, or document generation and processing, there’s really no need to turn it on. And with agent harnesses getting more capable all the time, it’s even more redundant in most cases.
After setting the thinking budget to 0, not only did speed improve dramatically, but output quality got noticeably better too.
- Running locally isn’t free either.
Local AI still costs money, and a quick tally makes that clear: hardware, securing access, electricity, and reliability. You have to weigh all of it.
Wrapping Up
That’s a from-memory rundown of what I’ve worked with and the pitfalls I’ve hit building local AI apps over the past few months. If I’ve missed anything, I’ll cover it in a future post, haha.