Stage 09 of 11 · Production Engineering · 3 min · Reviewed Aug 2026
Production Engineering: Tracing, Evals, Local Dev, Deploy
Production means you can see a run, score a run, reproduce a run, and ship a run. Observability, evaluation harnesses, local development against traces, and boring deployment (versioned graphs, flags, rollback) are how agents leave the demo.
- Tracing
- Observability
- Evals
- Local development
- Deployment
What you learn
How to make agents reliable enough to ship: traces, evals, and a deploy path with rollback.
If you cannot see it, you cannot ship it
Trace every run: input, route, model, token counts, tool names, arguments (redacted), observations (redacted), retries, cost, latency, stop reason. OpenTelemetry gen-ai semantic conventions and products like LangSmith, Langfuse, Phoenix, and vendor consoles all speak this language.
A screenshot of a good chat is not observability. You need IDs you can join to logs, bills, and tickets.
Visual
Agent Observability Pipeline
Every agent run emits structured traces. A collector aggregates them into dashboards and alerts so you can debug, bill, and audit any run by ID.
Evals are unit tests for probabilistic systems
Offline evals: a frozen set of cases with assertions (contains, JSON schema, rubric, tool-was-called). Online evals: sample live traffic, score, alert. LLM-as-judge is useful and gameable — pair it with deterministic checks.
promptfoo, DeepEval, custom pytest, and CI gates belong here. If the eval is optional, the prompt will drift the week after launch.
- id: refund-paid-order
assert:
- type: is-json
- type: javascript
value: output.decision === "escalate"
- type: tool-called
value: get_orderInteractive
Eval Pipeline Execution
Load test case
Frozen input with expected assertions is loaded from the eval suite.