An eval harness before the demo

Demos forgive mistakes. Production keeps the receipts. An eval harness is how you stop shipping vibes.

Minimum bar. If you cannot re-run yesterday’s cases after a prompt change, you do not have quality control — you have hope.

Start with a golden set#

Twenty to fifty real cases beat a thousand synthetic ones. Tag each case with:

• intent
• expected constraints (must cite / must refuse / must format)
• severity if wrong

A simple pass rate#

For $N$ cases and $p$ passes:

Track it per release. A drop is a regression — even if the new demo “feels smarter”.

Wire it into CI#

const report = await runEvals(goldenSet, { model, prompt });
if (report.passRate < baseline - 0.03) {
  throw new Error(`Eval regression: ${report.passRate}`);
}

Keep the gate boring. Fancy dashboards can grow later.

What to measure#

Closing#

Ship the harness with the first LLM feature — not after the first incident.