Skip to main content

​
Candidates

A candidate is one state of the code, identified by its git tree. Each commit that a worker makes is a candidate. The baseline that piloop init measures is the first one. Because a candidate is a tree, a commit that goes back to a known state is that known candidate, and piloop does not measure it again. A candidate can have several measurements. Its score is their mean under the current evaluator.

​
The head and the bar

The head is the latest kept candidate on the trunk. New runs start from it, unless an idea pins another starting point. Each run is judged against its bar. The bar starts at the base of the run, and moves to each keep that the run makes. For a normal run, the bar is the head. Each keep on the trunk beats the head before it, so the trunk is a chain of improvements.

​
The verdict

The daemon gives each measured candidate one verdict: The rule, in order:
  1. No value for the primary metric: crash.
  2. A guardrail past its floor: discard.
  3. The bar has no score under the current evaluator: keep.
  4. The primary metric is not better than the bar: discard.
  5. The primary metric is better, but its worst case at the confidence level is not above zero: interesting.
  6. A guardrail’s worst case is worse than the bar by more than its margin: interesting.
  7. Otherwise: keep.
Each verdict has a reason, for example:
A candidate whose deciding metrics equal those of its parent has a reason that starts with no-op. The change did not reach what the evaluate command measures.

​
Metric roles

  • primary: the metric to improve. It must beat the bar with confidence for a keep.
  • guardrail: a metric that may get a little worse. margin is the largest regression from the bar that is allowed, in the metric’s own units. floor is an optional hard limit on the value itself: at least the floor for higher, at most the floor for lower.
  • watch: recorded, and never part of the verdict.
A margin applies to each step. Many keeps that each give up a margin can add up to a large regression. A floor stops that. Only an edit of piloop.toml moves a floor.
Put every margin and floor in piloop.toml. A limit that is only in brief.md is a request to the agents. A limit in piloop.toml is enforced by the daemon.

​
Confidence

When the evaluate command writes per-unit results, the verdict uses a paired bootstrap over the units:
  1. Take the units that the candidate and the bar both report, and the difference of each.
  2. Resample the units with replacement 4000 times, and sum the differences each time.
  3. The worst case is the 1 - confidence quantile of these sums. With the default confidence = 0.95, it is the 5th percentile.
So a gain that comes from three units has the spread of three units, not of the whole test set. The random generator has a fixed seed, so the same numbers always give the same verdict. Without per-unit results, or when the two sides report different units, the verdict compares the aggregate values. Any gain then counts, and the reason says aggregate. This is right for a deterministic metric, such as a compressed size. For a noisy metric, write per-unit results.

​
When the head moves during a run

Two runs can start from the same head. When the first one keeps, the head moves. If the second one then beats its own bar, it is interesting, with a reason that starts with keeps against its own bar, but the head moved on. The daemon then merges the second candidate onto the new head, as a commit with two parents. The merge is a new candidate of the same run, and the daemon measures and judges it against the new head. If the merge has conflicts, the daemon files a seed Compose candidate#N onto the head instead.

​
Override a verdict

Any agent or human can change a verdict and say why:
The daemon never overwrites a verdict that it did not set.

​
Notes

A result that regressed can still be a good starting point. A note keeps it in view:
The candidate becomes interesting, with a reason that starts with note:. Each --follow-up files a seed pinned to the candidate. A note is not a keep, and it does not move a bar. A worker notes its own commit with a Piloop-Note: <why> trailer in the commit message.