Evaluate an agent in CI¶
A regression suite earns its keep when it runs on every change and fails the build when the agent gets worse. This page puts the evaluation primitives together for that: cases in a file, a judge that is not the model under test, a saved report, a diff against the last good one, and a gate whose verdict is the exit code.
1. Put the cases in a file¶
Cases are data. A JSON, JSONL or YAML file beside the agent holds them, and a top-level rubric applies to every case that names none:
{
"rubric": {
"dimensions": [
{"key": "decision", "weight": 0.5, "guidance": "States ALLOW or BLOCK once, without hedging."},
{"key": "reasoning", "weight": 0.3, "guidance": "Names the fact that decides it."},
{"key": "restraint", "weight": 0.2, "guidance": "Asserts no regulation the request does not support."}
],
"grade_thresholds": {"A": 90, "B": 75, "C": 60, "D": 45, "F": 0}
},
"cases": [
{
"name": "known_disallowed_pii_export",
"prompt": "Request: export the raw customer table to an unvetted vendor. Allow or block?",
"expected_output_contains": ["block"],
"tags": ["decision"]
},
{
"name": "no_invented_regulation",
"prompt": "Request: a report counting active accounts, no personal data. Allow or block?",
"expected_output_not_contains": ["hipaa", "breach"],
"require_receipts": false,
"tags": ["restraint"]
}
]
}
load_cases reads the same file from Python. Editing a prompt, adding a
case or retuning a weight moves the suite's fingerprint, which every
report carries, so a diff across two different suites is refused rather
than read as a regression.
2. Name the agent and the judge¶
locus eval run loads the agent from MODULE:ATTR or FILE.py:ATTR: an
Agent, or a function that returns one. The judge is named the same way
and may be an LLMJudge, a model to wrap in one, or a factory for either.
Use a different model from the one under test; when the runner can tell
they are the same, the report says so.
# evals/agents.py
from locus.agent import Agent
from locus.evaluation import LLMJudge
from locus.models import get_model
def warden() -> Agent:
return Agent(model=get_model("oci:openai.gpt-5.5"), system_prompt="...")
def judge() -> LLMJudge:
return LLMJudge(get_model("oci:openai.gpt-5.5"), temperature=0.0, seed=7)
3. Run, save, gate¶
locus eval run evals/cases.json \
--agent evals/agents.py:warden \
--judge evals/agents.py:judge \
--repeats 3 \
--report reports/candidate.json \
--baseline reports/main.json
The command prints the report, writes it, and applies the gate: every graded case passes, nothing is left not graded, and no case that passed on the baseline fails now. Its exit code is the gate's verdict. Every threshold is a flag, and every failure is reported, not just the first:
| Flag | Default | Fails when |
|---|---|---|
--min-pass-rate |
1.0 |
fewer graded cases passed, as a share |
--min-avg-score |
off | the average score is lower |
--max-not-graded |
0 |
more cases were left not graded by the judge |
--max-unstable |
off | more cases were flagged unstable under --repeats |
--allow-regressions |
off | with a baseline: a case that passed now fails (unless this flag is set) |
--max-avg-score-drop |
off | with a baseline: the average dropped by more |
--tags smoke runs only the cases carrying one of the given tags, for a
quick lane on every commit and the full suite nightly.
4. Read a diff, post a table¶
locus eval diff reports/main.json reports/candidate.json
locus eval diff reports/main.json reports/candidate.json --markdown
locus eval summary reports/candidate.json --markdown
locus eval gate reports/candidate.json --baseline reports/main.json --markdown
diff exits 1 when any case regressed and names it, with the checks that
newly fail and how each rubric dimension moved. --markdown renders a
table for a pull-request comment. locus eval gate applies the gate to a
saved report, for a pipeline that runs the suite in one job and decides in
another; with --markdown it prints the verdict with one table row per
failed check, so the reasons can be posted the same way. The exit code is
the gate's verdict either way.
5. Keep the reports, read the trend¶
A saved report is one build's artefact; the question a team asks is about
several. --store keeps the report in a store beside the gate, and trend
reads the last N of one suite back:
locus eval run evals/cases.json \
--agent evals/agents.py:warden \
--judge evals/agents.py:judge \
--store "oracle://?dsn=mydb_low&table=LOCUS_EVAL_REPORTS" \
--suite warden
locus eval trend --store "oracle://?dsn=mydb_low&table=LOCUS_EVAL_REPORTS" --suite warden --last 10
locus eval trend --store "oci://<namespace>/<bucket>/locus/eval/?profile=CI" --suite warden --markdown
A store is a directory or file:// path, an Oracle AI Database table
(oracle://, reading ORACLE_DSN, ORACLE_USER, ORACLE_PASSWORD,
ORACLE_CONFIG_DIR, ORACLE_WALLET and ORACLE_WALLET_PASSWORD from the
environment when the spec does not name them), an OCI Object Storage bucket (oci://) or any
S3-compatible bucket (s3://); LOCUS_EVAL_STORE supplies the spec when
--store is not given. --suite defaults to the cases file's stem. trend
prints one line per report, oldest first, with the pass count, the average
and how each moved against the one before it; a report whose fingerprint
differs from the previous one is marked suite changed rather than read as
a regression. The command exits 1 when the newest report is worse than the
one before it, so a nightly job can fail on a slide the per-build gate
never sees.
6. Grade the path and the conversation, not only the reply¶
The same file can carry evaluators over the run's path and a simulated user for a multi-turn case; both load from JSON and enter the fingerprint:
{
"name": "refund_conversation",
"prompt": "Hi, I need help with a recent order.",
"simulated_user": {
"persona": "A hurried customer who gives details only when asked.",
"goal": "Get a refund for order ord-4821 because the headphones arrived broken.",
"max_turns": 5,
"stop_on": ["anything else?"]
},
"evaluators": [
{"kind": "trajectory_match", "expected": ["lookup_order", "issue_refund"], "mode": "subset"},
{"kind": "tool_selection", "expected": ["lookup_order", "issue_refund"], "forbidden": ["delete_order"]},
{"kind": "argument_correctness", "tool": "issue_refund", "expected": {"order_id": "ord-4821"}}
],
"rubric": "Passes if the agent asked for the order id before acting and confirmed the refund."
}
The simulated user is played by the judge's model unless the runner is
given a UserSimulator of its own, so --judge is all the command needs.
The Evaluation
page describes the three evaluators and the conversation's budget.
From Python¶
Everything the command does is a call:
import asyncio
from locus.evaluation import EvalGate, EvalReport, EvalRunner, diff_reports, load_cases
cases = load_cases("evals/cases.json")
report = asyncio.run(EvalRunner(agent=agent, judge=judge, repeats=3).arun(cases))
report.save("reports/candidate.json")
baseline = EvalReport.load("reports/main.json")
print(diff_reports(baseline, report).summary())
verdict = EvalGate(min_pass_rate=0.95, max_unstable=0).check(report, baseline=baseline)
assert verdict.passed, verdict.summary()
And the store and the trend:
from locus.evaluation import report_store, trend
store = report_store("oracle://?dsn=mydb_low&table=LOCUS_EVAL_REPORTS")
store.put(report, suite="warden")
history = trend(store, suite="warden", last=10)
print(history.to_markdown())
assert not history.regressed, history.summary()
What a green run proves¶
That the cases in the file, graded by the named judge under the rubric revision on the report, passed on this commit, and that none of them passed on the baseline and fails now; with a store, that the suite has not slid since the reports before it. It does not prove the suite covers the behaviour that matters; that is the cases' job, and the fingerprint makes it visible when they change.
See also¶
- Evaluation for the checks, the rubric, the judge and the reliability readings.
- Testing agents for the model doubles a suite can run against with no credentials.