Safety & ops
What is Eval Harness?
An eval harness runs many agent or prompt variants against fixtures and assertions.
promptfoo is one harness. Without fixtures, you cannot detect regressions.
Why it matters
An eval harness runs many agent or prompt variants against fixtures and assertions. It is the automated testing system for agent behavior — catching regressions before they reach production.
Without an eval harness, you are testing agents manually, which does not scale and misses edge cases.
Key takeaways
- 1Eval harnesses are automated testing for agents.
- 2They catch regressions before production.
- 3Without fixtures and assertions, evaluation is guesswork.
Common mistakes
- ✕Running evals without assertions — execution without expected outcomes is not evaluation.
- ✕Testing on too few fixtures to be representative.
Related terms
Concept neighborhood
Terms linked from Eval Harness in the glossary graph.
- Eval Harness
- Prompt Evaluation
- Agent Evaluation
- promptfoo