An observability budget that scales

Observability fails in two ways: too little signal, or too much noise. Both are expensive.

Start from user journeys#

Instrument the path a customer cares about. Everything else is optional until that path is green.

SLO sketch#

For availability target $a$ over window $T$:

Example: $a = 0.999$ over 30 days ≈ 43 minutes of budget.

Sampling without blindness#

// Prefer head sampling with a floor for errors
func shouldSample(trace Trace) bool {
  if trace.HasError() {
    return true
  }
  return rand.Float64() < 0.05
}

Ownership table#

Closing#

Budget the telemetry like you budget CPU. If nobody owns a dashboard, delete it.