Prompt workshop
What is promptfoo?
promptfoo is an eval harness for prompts and models: configs, assertions, diffs, and CI.
It measures prompts. It does not by itself search for better wording.
Why it matters
promptfoo is an eval harness for prompts and models. It lets you define test cases with assertions, run them against multiple prompt variants and models, and compare results in a matrix.
promptfoo matters because it brings CI/CD practices to prompt engineering — you can catch regressions before they reach production.
Key takeaways
- 1promptfoo brings CI/CD testing to prompts.
- 2It compares variants across models with assertions.
- 3It measures prompts — it does not optimize them (that's DSPy's job).
Common mistakes
- ✕Confusing promptfoo (evaluation) with DSPy (optimization).
- ✕Writing assertions that are too loose to catch real regressions.
Related terms
Concept neighborhood
Terms linked from promptfoo in the glossary graph.
- promptfoo
- Prompt Evaluation
- Prompt Engineering