A campaign can end with a better number that does not hold up. Review a campaign before you trust its result.
With a coding agent
Open Claude Code, Codex or pi in the project repo and ask:
The agent follows the piloop-review skill. It reads the database read-only, and never changes the running
campaign. It answers four questions, in this order:
- Is the result real? The trajectory of the head, the noise between sibling candidates, and whether the gains
hold on data that no agent tuned against.
- Did any agent game the evaluation? Edits to frozen files, reads of labels or per-unit results, and tuning
against the evaluate command.
- What did it cost? Tokens and time for each role, and the work that produced nothing.
- Whose fault is each failure? The model, the harness, the environment, or the setup of the campaign.
It tags each claim as measured or inferred. It ends with a report, bug reports for problems in the tools, and inbox
items for the decisions that only you can take.
By hand
The trajectory of the head and every keep:
The commit of a candidate, and its diff from its parent:
Check that no keep changed a frozen file, such as the evaluate command or the data:
An evaluator is defined by the evaluate command and the metrics, not by the contents of evaluate.py. A worker that
edits evaluate.py changes the score without a new evaluator. Check the diffs of the kept candidates.