Workshop · Test / regress
promptfoo
Write the cases. Run every prompt variant against every assertion. Diff the matrix against the last green run. If a wording change drops pass-rate, the build fails — the same way a unit test fails.
What a template cannot do
Library templates have no tests. A persona that worked in one chat session can silently rot after a model update or a comma.
- Not an automatic prompt rewriter.
- Not live eval in this encyclopedia.
- Not a substitute for product metrics, only a harness for prompt assertions.
How it works
Conceptual workflow
Steppable map of promptfoo. Evidence is taken from the spec on this page. Nothing here is a live compiler, eval, decoder, or optimizer run.
vcp · prompts/promptfoo
run@once
- config
- assert
- run
- diff
- ci
Stage 1 / 5 · receive
Write prompts, providers, tests
The config is the spec: which strings, which models, which inputs. If it is not in the file, it is not tested.
Artifact · promptfooconfig.yaml
User input
two prompt variants, one provider
Rule in force
Declare what will be run.
Visible reply
prompts: [persona.v1, persona.v2]
Illustration · not a live model run
Spec evidence
promptfoo is an evaluation harness for prompts and models. It does not invent a better prompt for you. It tells you whether the prompt you have still passes the tests you wrote. A config lists prompt variants, providers, and tests. Each test is an input plus assertions: equals, contains, regex, JSON schema, cost, latency, or an LLM rubric. Run produces a matrix: every prompt against every test against every provider. Failures are first-class rows, not a vibe that “it seemed fine in chat.” A baseline is the last known-good run. Diff the new matrix against that baseline. A wording change that drops assertion pass-rate is a regression, the same way a unit test is a regression. CI is the point. If the eval is optional, the prompt will drift. Wire the run into the same pipeline that already fails on red tests. promptfoo does not execute here. This page maps the eval loop so you can see where a pasted persona is silent.
- This page is a conceptual map
- The compiler or eval runner is not executed here
Method
Test / regress loop · not executed here
Library
A pasted persona belongs in the library
Conceptual workflow · 4.5s / stage · 1/5