OfferTurnkey landing from $150
All posts
  • ai
  • rag
  • llm
Applied AI

RAG without the hype

Retrieval-augmented generation that stays operable — chunking, evals, and failure modes that actually show up in production.

  • MS
  • AK
Authors

RAG demos look magical. Production RAG looks like data contracts, eval harnesses, and boring monitoring.

Chunking is product design

Chunk size is not a hyperparameter you copy from a blog. It depends on:

  • document structure (policies vs chat logs)
  • citation UX (can a human verify the snippet?)
  • latency budget for retrieval + generation

A simple relevance score

For a query embedding qq and chunk embedding dd:

s(q,d)=q⋅d∥q∥ ∥d∥s(q, d) = \frac{q \cdot d}{\lVert q\rVert\,\lVert d\rVert}

Rank by ss, then filter with a floor threshold so the model is not forced to invent from noise.

Minimal retrieval loop

ts
async function answer(query: string) {
  const hits = await vector.search(query, { k: 8 });
  const context = hits
    .filter((h) => h.score > 0.72)
    .map((h) => h.text)
    .join("\n---\n");

  return llm.chat([
    { role: "system", content: "Cite only from context. If missing, say so." },
    { role: "user", content: `Context:\n${context}\n\nQuestion: ${query}` },
  ]);
}

Eval before polish

Track at least:

MetricWhat it catches
GroundednessHallucinations
Recall@kBad chunking / index drift
Latency p95Infra regressions

Closing

Ship the evaluation loop first. Fancy agents can wait.