LLMOps: Evaluation and Observability for Production Agents

The constraint

A prompt is production code with none of production code's safety net. No type system, no unit tests, and a one-word edit can silently degrade quality for a whole class of inputs while looking fine on the three examples you happened to check.

Teams ship prompt changes on vibes. Then quality drifts, nobody can say when it started, and the instinct is to blame the model.

Approach

Treat prompts as versioned artifacts under regression test. Three layers:

Golden set. 150+ real queries drawn from production logs, each with a graded reference answer. Every prompt or model change runs the full set and reports per-category deltas. A drop beyond threshold fails CI. Building this from logged queries rather than imagination is the whole ballgame — see the RAG case study for what happens otherwise.

Trace-level observability. Each agent run emits a structured trace: which sub-agent ran, what it retrieved, token counts, latency, validation results. When a user reports a bad answer you can replay the exact path rather than guessing.

Cost attribution. Tokens logged per turn, aggregated into BigQuery by feature and tenant. Without this, LLM spend is a single opaque line item and you cannot tell which feature is expensive.

Architecture

Evaluation is a Cloud Run job triggered by CI. It loads the golden set, runs the candidate prompt against the current model, and scores with a rubric-based grader plus deterministic checks — did it cite sources, did it stay in schema, did it refuse when it should. Scores land in BigQuery; the workflow posts a comparison table to the pull request.

The rubric grader is itself an LLM call, which is uncomfortable but workable: it is graded against human labels periodically to confirm it still correlates. When it drifts, it gets recalibrated.

What broke

Early evaluation used a single aggregate quality score. It hid everything. A change that improved general answers by 5% while breaking edge-case refusals entirely showed up as a net positive.

Now scores are reported per category — factual, refusal, multi-hop, out-of-scope — and a regression in any category fails the build regardless of the average. Aggregate metrics are how you ship regressions with confidence.

Outcome

Every prompt change is regression-tested before merge. Bad answers are debuggable to the retrieval step. Cost is attributable per feature, which turned an unbounded line item into something a finance team can actually plan against.