Verdict rules, stop rules and fact sources

The verdict rule decides which candidates to keep. The stop rule decides whether a campaign goes on. Both are pure: piloop gives them data, and they return data. The fact sources supply what a stop rule or a section reads.

Verdict rules

After piloop measures a candidate, it asks the verdict rule whether to keep the candidate. piloop gives the rule a Case: the measurements of the candidate, the measurements of its bar, and the metrics that decide. The bar is the candidate that this one must beat. The rule returns a Judgement: keep, interesting, discard or crash, and a reason. piloop stores the verdict and the reason on the candidate. The built-in rule is bootstrap. It keeps a candidate when the primary metric beats the bar with confidence and no guardrail metric gets too much worse.

Example: keep only a gain of a fixed size

This rule calls the built-in rule bootstrap and changes one thing: a keep whose gain is below min_gain becomes interesting. min_gain is in the units of the primary metric:

pyproject.toml

[project.entry-points."piloop.verdict_rules"]
min_gain = "piloop_min_gain:MinGain"

piloop_min_gain/__init__.py

"""``min_gain``: the built-in verdict, but a keep needs a gain of at least ``min_gain``. No state, no I/O.

Use it when a tiny gain is not worth a keep. Each keep moves the head, and every later run must beat the head. The
rule is the built-in rule ``bootstrap`` and changes one thing. The confidence, the guardrails and the reasons stay
those of ``bootstrap``.
"""

from piloop.capabilities import Bootstrap, Case, Judgement, Outcome, number


class MinGain(Bootstrap):
    """What ``bootstrap`` decides, but a keep with a gain below ``min_gain`` is ``interesting``."""

    min_gain: float
    """The smallest gain that keeps, in the primary metric's units."""

    def decide(self, case: Case) -> Judgement:
        judged = super().decide(case)
        name = case.primary.name
        if judged.verdict is not Outcome.KEEP or name not in case.bar.means:
            return judged
        gain = case.primary.direction.gain(case.candidate.means[name], case.bar.means[name])
        if gain >= self.min_gain:
            return judged
        return Judgement(Outcome.INTERESTING, f'{judged.reason}; the gain is below min_gain {number(self.min_gain)}')

Write a plugin installs and selects it.

The contract

A verdict rule is a pure provider, a subclass of VerdictRule. Import the types from piloop.capabilities.

Member	What it is
`VerdictRule.decide(case: Case) -> Judgement`	Return the verdict on `case.candidate` against `case.bar`, and the reason.
`VerdictRule.bar(ancestry: Sequence[Node]) -> CandidateId`	Return the bar of a run that starts from `ancestry[0]`. The ancestry goes from it, parent by parent, to its root.
`VerdictRule.raises_bar(node: Node, lineage: Lineage) -> bool`	Return whether `node`, a commit of a run that keeps on `lineage`, becomes the bar of the run’s next commits.
`VerdictRule.head(trunk: Sequence[Node]) -> CandidateId \| None`	Return the head among the trunk’s keeps, oldest first, or None when there is none.
`VerdictRule.on_moved(moved: Moved) -> Rebase \| Leave`	Return what follows a keep that lacks the head. By default its merge onto the head, and a seed on a conflict.
`Case.primary: Criterion`	The primary metric of the candidate’s track.
`Case.guardrails: tuple[Criterion, ...]`	The campaign’s guardrails, then the primary of every other track.
`Case.candidate: Side`	The candidate that the rule judges.
`Case.bar: Side`	What the candidate must beat: its run’s bar (https://piloop.autoresearch.inc/concepts/verdict#the-words).
`Case.parent: Side \| None`	The candidate’s parent, which the candidate changed. None for a candidate without one.
`Criterion.name: str`	The metric’s name, as the evaluate command reports it.
`Criterion.direction: Direction`	Whether a larger or a smaller value is better.
`Criterion.margin: float \| None`	The largest regression from the bar that a guardrail may show, in the metric’s units. None on a primary.
`Criterion.floor: float \| None`	The worst value that a candidate may have, whatever the bar, or None.
`Criterion.fraction: float \| None`	The largest regression from the bar that a guardrail may show, as a fraction of the bar’s value. A guardrail has a `margin` or a `fraction`, never both.
`Criterion.ratio: Ratio \| None`	For a metric that piloop computes: its parts, whose units and samples are in each `Side` by their names.
`Criterion.allowed(bar: float) -> float`	The largest regression from `bar`, the bar’s value, in the metric’s units.
`Criterion.limit() -> str`	How a reason names the regression allowed: `margin 20` or `fraction 0.1`.
`Criterion.past_floor(value: float) -> bool`	Whether `value` is past the floor, in the metric’s direction.
`Side.name: str`	How a reason names it: `candidate#73`.
`Side.means: dict[str, float]`	The mean of each metric over its successful measurements. Empty when none succeeded.
`Side.units: dict[str, dict[str, float]]`	The per-unit results of each metric that has them: unit id → the unit’s mean contribution.
`Side.samples: dict[str, list[float]]`	The samples of each metric that ran in pairs with the other side’s: the i-th of the candidate and the i-th of the bar ran one after the other, on one worker. Empty without pairs.
`Side.stopped: Stopped \| None`	Where its latest climb of the ladder stopped, if it stopped. `means` never include a rung.
`Side.rungs: dict[RungName, dict[str, float]]`	The mean of each metric at each rung of the ladder that it measured.
`Judgement.verdict: Outcome`	`keep`, `interesting`, `discard` or `crash`.
`Judgement.reason: str`	Why, as piloop stores and shows it with the verdict.
`Node.id: CandidateId`	The candidate’s id.
`Node.verdict: Outcome \| None`	None while it is pending, and for a commit that is never judged.
`Node.made: Lineage \| Imported`	Where it was made: on the side of the run that made it, or imported. The baseline is on the trunk.
`Node.means: dict[str, float]`	The mean of each metric over its successful measurements under the current evaluator.
`Branch.root: CandidateId`	The candidate that the branch run started from.
`Moved.candidate: Node`	The keep that lacks the head.
`Moved.head: Node`	The head that it lacks.
`Rebase.conflicts: Seed \| Leave`	What follows when the merge conflicts.

A rule has no error case: it returns a verdict for every case. The rule also decides the lineage: bar, raises_bar, head and on_moved read Nodes (a candidate’s id, verdict, where it was made, Trunk, Branch or Imported, and its means), and each has the built-in behaviour as its default. piloop applies the answers: it moves the head, merges, files the seed, and applies the Piloop-Note of a commit.

Built in

bootstrap is the rule of a campaign without [plugins.verdict_rule]. The verdict describes it.

Stop rules

On each pass, the daemon asks the stop rule of each active campaign whether the campaign goes on, holds, or stops. piloop gives the rule a StopCase: the facts that the rule asked for. The rule returns one of four answers:

Answer	What piloop does
`Go(left)`	The campaign goes on: piloop starts what it needs, within `left`.
`Hold(reason)`	piloop starts no agent and no evaluation for the campaign until a check answers `Go` again.
`Stop(reason)`	piloop stops the campaign.
`Escalate(reason, then)`	piloop posts one escalation at priority 0 for each distinct `reason`, and then acts on `then`.

A hold is not a pause. piloop asks the rule again on each pass, and the hold ends when the rule answers Go. piloop does not stop a paused campaign, but it holds one. The built-in rule is limits. It holds the campaign while a fact of hold_at is unknown or at its limit. Else it stops the campaign at the first of its other limits that the campaign reaches.

Example: end the campaign after a number of keeps

This rule stops the campaign after keeps candidates are kept. It calls the built-in rule limits and adds one limit, so the limits of limits, such as max_hours and hold_at, still apply:

pyproject.toml

[project.entry-points."piloop.stop_rules"]
after_keeps = "piloop_stop_after_keeps:AfterKeeps"

piloop_stop_after_keeps/__init__.py

"""``after_keeps``: stop the campaign after ``keeps`` keeps, or at a limit of the built-in rule. No state, no I/O.

Use it for a campaign that wants a few good changes and no more, for example "make the build pass on the new
compiler". It stops when enough candidates are kept. The rule is the built-in rule ``limits`` with one more limit, so
``max_hours``, ``max_tokens``, ``hold_at`` and the others still apply.
"""

from piloop.capabilities import KEEPS, Escalate, Fact, FactValue, Go, Hold, Limits, Stop, StopCase


class AfterKeeps(Limits):
    """The campaign ends after ``keeps`` keeps. ``limits`` holds it and stops it at its own limits."""

    keeps: int
    """The number of keeps that ends the campaign."""

    def limits(self) -> dict[Fact[FactValue], float]:
        return super().limits() | {KEEPS: self.keeps}

    def check(self, case: StopCase) -> Go | Hold | Stop | Escalate:
        answer = super().check(case)
        keeps = case.facts.get(KEEPS)
        if not isinstance(answer, Go) or keeps is None or keeps < self.keeps:
            return answer
        return Stop(f'after_keeps: {keeps} keeps')

[plugins.stop_rule]
provider = "after_keeps"
keeps = 3
max_hours = 12

Example: escalate when a keep costs too much

This rule compares the fact that spend names, since the latest keep, with its value per keep so far. Past escalate_at times, it answers Escalate once for each keep; past stop_at times, it stops the campaign:

pyproject.toml

[project.entry-points."piloop.stop_rules"]
keep_cost = "piloop_keep_cost:KeepCost"

piloop_keep_cost/__init__.py

"""``keep_cost``: tell a human when a keep costs too much, and stop at a harder line. No state, no I/O.

The rule compares the fact ``spend`` names, since the latest keep, with its value per keep so far. Past ``escalate_at``
times, it asks a human once for each keep; past ``stop_at`` times, it stops the campaign. It is the built-in rule
``limits`` with these two lines, so ``max_hours``, ``hold_at`` and the others still apply. Before the first keep there
is no value per keep, and the rule is ``limits``.
"""

from pydantic import Field

from piloop.capabilities import (
    KEEPS,
    Escalate,
    Fact,
    FactValue,
    Go,
    Hold,
    Limits,
    Stop,
    StopCase,
    per_keep,
    reading,
    since_keep,
)


class KeepCost(Limits):
    """The cost since the latest keep against the cost per keep: escalate at ``escalate_at``×, stop at ``stop_at``×."""

    spend: str = 'spend'
    """The fact that is what the campaign spent so far, which a source of ``[[facts]]`` reads."""
    escalate_at: float = Field(default=2, gt=0)
    """Ask a human once the cost since the latest keep is this many times the cost per keep."""
    stop_at: float = Field(default=4, gt=0)
    """Stop once the cost since the latest keep is this many times the cost per keep."""

    def reads(self) -> list[Fact[FactValue]]:
        return list(dict.fromkeys([*super().reads(), reading(self.spend), KEEPS, since_keep(self.spend)]))

    def check(self, case: StopCase) -> Go | Hold | Stop | Escalate:
        answer = super().check(case)
        each, since = per_keep(case.facts, self.spend), case.facts.get(since_keep(self.spend))
        if not isinstance(answer, Go) or each is None or since is None:
            return answer
        if since >= self.stop_at * each:
            return Stop(
                f'keep_cost: {self.spend} {since:g} since the latest keep, {self.stop_at:g} × the {each:g} per keep'
            )
        if since >= self.escalate_at * each:
            # one escalation for each keep: the reason names the keep, not the amount, which changes on every pass
            return Escalate(
                f'keep_cost: since keep {case.facts.get(KEEPS)}, {self.spend} passed {self.escalate_at:g} × its value'
                f' per keep; it stops at {self.stop_at:g} ×'
            )
        return answer

[plugins.stop_rule]
provider = "keep_cost"
spend = "spend"
escalate_at = 2
stop_at = 4

The contract

A stop rule is a pure provider, a subclass of StopRule. Its fields are its limits and other parameters. Import the types from piloop.capabilities. Most facts cost database queries on every pass, so piloop gets only those of reads. piloop status and the briefs join the parts of describe into the progress line: piloop formats no fact itself.

Member	What it is
`StopRule.while_held: WhileHeld`	What the closing review does when the campaign stops (`piloop campaign stop`) while the rule holds its launches: `run` it anyway, `skip` it, or `wait` until the hold ends.
`StopRule.reads() -> list[Fact[FactValue]]`	Return the facts that `check` and `describe` read: piloop gets only those.
`StopRule.describe(case: StopCase) -> list[str]`	Return where the campaign stands against the rule, in short parts: `run 7/200`, `2.1h/12h`.
`StopRule.check(case: StopCase) -> Go \| Hold \| Stop \| Escalate`	Return whether the campaign goes on, holds or stops, or tell a human why and then do one of those.
`StopRule.check_config(shape: Shape) -> str \| None`	Return why the rule cannot serve a campaign of `shape`, or None. By default, it serves every campaign.
`StopCase.primary: str`	The name of the primary metric of the campaign’s first track.
`StopCase.direction: Direction`	Whether a larger or a smaller value of the primary metric is better.
`StopCase.facts: Facts`	The facts that the rule reads (`reads`), from the campaign’s fact sources.

Go.left says how much of each fact is left before the rule stops the campaign. piloop starts no more runs than RUNS leaves, and cuts each agent’s timeout where HOURS runs out, so a run that starts just before the stop ends with it. An agent whose start comes after the rule answers Stop never starts: the pass that stops the campaign closes its row. check_config refuses a campaign that the rule cannot serve when the file loads: limits refuses a target on a campaign of more than one track. A fact is a typed key, Fact[T], that a fact source of the campaign supplies. case.facts.get(KEEPS) reads one, and case.facts.get(reading('spend')) one that a source reads by name. Every fact that a rule reads needs a source in the campaign, or piloop refuses the configuration. The facts of the built-in sources campaign and optimization, from piloop.capabilities, and the ratios over facts that a rule names (spend, the fact of what the campaign spent; cost, the evaluator’s metric of what one evaluation costs):

Fact	Source	Type	What it is
`HOURS` (`hours`)	`campaign`	`float`	The hours since the first run started, without the time that the campaign was paused.
`TOKENS` (`tokens`)	`campaign`	`int`	The tokens that the agents of the campaign spent, summed over the agents that ended.
`HEAD` (`head`)	`optimization`	`float`	The primary metric of the first track at the head, under the current evaluator.
`RUNS` (`runs`)	`optimization`	`int`	The runs that hold a place under a limit on runs: those that count, and the attempts that may still count.
`UNSETTLED` (`unsettled`)	`optimization`	`int`	The attempts whose outcome can still change: queued, running, or being scored.
`WITHOUT_KEEP` (`without_keep`)	`optimization`	`int`	The finished runs that count since the latest keep, or since the campaign was last set active again.
`KEEPS` (`keeps`)	`optimization`	`int`	The candidates of runs that are kept, on the trunk or on a branch. The baseline is not one.
`EVALUATIONS` (`evaluations`)	`optimization`	`int`	The runs of the evaluate command under every evaluator: each measurement, failed ones and bar samples among them.
`METRIC_SUMS` (`metric_sums`)	`optimization`	`dict[str, float]`	Each metric summed over every measurement. A rule that reads a cost metric names it.
`ATTEMPTS` (`attempts`)	`optimization`	`int`	The runs that are an attempt at their idea: the open ones, and those whose end is one.
`FUNNEL` (`funnel`)	`optimization`	`Funnel`	How far the attempts that ended and are judged got (`Funnel`).

Ratio	What it is
`reading(name: str) -> Fact[float]`	The fact of a value that a source reads under `name`, such as `spend` in a result of `command`.
`since_keep(name: str) -> Fact[float]`	How much the value of `reading(name)` grew since the latest keep: `<name>_since_keep`.
`per_keep(facts: Facts, spend: str) -> float \| None`	The fact `spend` per keep. Reads `reading(spend)` and `KEEPS`.
`per_evaluation(facts: Facts, cost: str) -> float \| None`	The sum of the evaluator’s metric `cost` per evaluation. Reads `EVALUATIONS` and `METRIC_SUMS`.
`worker_spend(facts: Facts, spend: str, cost: str) -> float \| None`	The fact `spend` less the sum of the evaluator’s metric `cost`. Reads `reading(spend)` and `METRIC_SUMS`.
`per_run(facts: Facts, spend: str, cost: str) -> float \| None`	`worker_spend` per attempt. Reads `reading(spend)`, `METRIC_SUMS` and `ATTEMPTS`.

Each is None while a term is, or when its count is 0. A rule that reads a ratio asks for the facts of its terms. A stop that a person requests (piloop campaign stop) stops the campaign whatever the rule answers, also when the rule holds it. The closing review of that stop then follows the rule’s while_held (Spend outside piloop).

Built in

limits is the rule of a campaign without [plugins.stop_rule]. Stop conditions lists its limits.

Fact sources

A fact source supplies the facts that a stop rule or a section reads.

Example: read the spend from a billing endpoint

import json
import urllib.request
from pathlib import Path

from piloop.capabilities import FactReadError, Readings


class Billing(Readings):
    """``billing``: what a billing endpoint says that the campaign spent, as ``spend``."""

    url: str

    def names(self) -> tuple[str, ...]:
        return ('spend',)

    def read(self, repo: Path) -> dict[str, float]:
        try:
            with urllib.request.urlopen(self.url, timeout=30) as answer:
                return {'spend': float(json.load(answer)['total'])}
        except (OSError, KeyError, ValueError) as e:
            raise FactReadError(f'{self.url}: {e}') from e

[project.entry-points."piloop.fact_sources"]
billing = "my_package:Billing"

[[facts]]
provider = "campaign"

[[facts]]
provider = "optimization"

[[facts]]
provider = "billing"
url = "https://billing.example.com/campaigns/enwik8"

hold_at = { spend = 1000 } then holds the campaign at that spend. A fact source is a provider of FACT_SOURCES, a subclass of FactSource. [[facts]] lists a campaign’s sources; a campaign that does not list them has every installed source that is on by default (ON_BY_DEFAULT): campaign, the hours and tokens of any campaign, and optimization, its research. A list that adds a source lists those too. Two sources never supply the same fact. A Readings source (piloop.plugins.fact_source.kit.readings) is one that the daemon reads every interval seconds, outside any transaction. It implements names and read, and piloop does the rest: each reading is a fact.read event on the campaign, its one record. It posts a note at each of its notices, and escalates once when its readings start to fail.

The contract

Member	What it is
`FactSource.ON_BY_DEFAULT: ClassVar[bool]`	Whether a campaign that does not list its `[[facts]]` has this source, with its defaults.
`FactSource.supplies() -> tuple[Fact[FactValue], ...]`	The facts that it supplies. Two sources of a campaign never supply the same one.
`FactSource.gather(s: Session, campaign: Campaign, wanted: frozenset[Fact[FactValue]], now: datetime) -> dict[Fact[FactValue], FactValue]`	The values of the facts in `wanted` that it supplies, as of `now`. It reads with `s`, never writes.
`FactSource.due(s: Session, campaign: Campaign) -> datetime \| None`	When the daemon calls `read` next: a time at or before now reads now.
`FactSource.read(repo: Path) -> dict[str, float]`	Read values by name, at the campaign’s repo root. The daemon calls it outside any transaction, when it is due.
`FactSource.record(s: Session, campaign: Campaign, values: dict[str, float] \| FactReadError) -> None`	Record what `read` returned or raised. The caller commits. By default, nothing.
`Readings.interval: float`	The number of seconds between two readings.
`Readings.notices: dict[str, list[float]]`	Values by the name that it reads. When a value passes one, the daemon posts one note to the inbox, at priority 1.
`Readings.names() -> tuple[str, ...]`	The names of the values that it reads.
`Readings.read(repo: Path) -> dict[str, float]`	Read every value of `names`. The daemon calls it outside any transaction, at the campaign’s repo root.
`Readings.supplies() -> tuple[Fact[FactValue], ...]`	`reading(name)` and `since_keep(name)` for each name of `names`.
`Readings.gather(s: Session, campaign: Campaign, wanted: frozenset[Fact[FactValue]], now: datetime) -> dict[Fact[FactValue], FactValue]`	Each value of the latest reading that has it, None after a failed one, and its growth since the latest keep.
`Readings.due(s: Session, campaign: Campaign) -> datetime \| None`	When the source reads next: `interval` after its latest reading, or now before the first.
`Readings.record(s: Session, campaign: Campaign, values: dict[str, float] \| FactReadError) -> None`	Record a reading as a `fact.read` event, with a note for each notice that it passed. The caller commits.

The built-in command source runs a command that writes {"metrics": {"<name>": <value>}} to $PILOOP_RESULT_FILE, as the evaluate command does. Spend outside piloop configures it. piloop gives no fact a meaning: a stop rule or a section names the facts that it reads.

Get started

Concepts

Guides

Plugins

Reference

Verdict rules, stop rules and fact sources

Verdict rules

Example: keep only a gain of a fixed size

The contract

Built in

Stop rules

Example: end the campaign after a number of keeps

Example: escalate when a keep costs too much

The contract

Built in

Fact sources

Example: read the spend from a billing endpoint

The contract

​Verdict rules

​Example: keep only a gain of a fixed size

​The contract

​Built in

​Stop rules

​Example: end the campaign after a number of keeps

​Example: escalate when a keep costs too much

​The contract

​Built in

​Fact sources

​Example: read the spend from a billing endpoint

​The contract

Verdict rules

Example: keep only a gain of a fixed size

The contract

Built in

Stop rules

Example: end the campaign after a number of keeps

Example: escalate when a keep costs too much

The contract

Built in

Fact sources

Example: read the spend from a billing endpoint

The contract