> ## Documentation Index > Fetch the complete documentation index at: https://piloop.autoresearch.inc/llms.txt > Use this file to discover all available pages before exploring further. # Verdict rules, stop rules and fact sources > Plugins that decide which candidates to keep and when a campaign stops, and the facts that they read. The verdict rule decides which candidates to keep. The stop rule decides whether a campaign goes on. Both are [pure](https://piloop.autoresearch.inc/plugins/overview.md#host-and-pure-contracts): piloop gives them data, and they return data. The fact sources supply what a stop rule or a section reads. ## Verdict rules After piloop measures a candidate, it asks the verdict rule whether to keep the candidate. piloop gives the rule a `Case`: the measurements of the candidate, the measurements of its bar, and the metrics that decide. The bar is the candidate that this one must beat. The rule returns a `Judgement`: `keep`, `interesting`, `discard` or `crash`, and a reason. piloop stores the verdict and the reason on the candidate. The built-in rule is `bootstrap`. It keeps a candidate when the primary metric beats the bar with confidence and no guardrail metric gets too much worse. ### Example: keep only a gain of a fixed size This rule [calls the built-in rule](https://piloop.autoresearch.inc/plugins/overview.md#call-a-built-in-provider) `bootstrap` and changes one thing: a keep whose gain is below `min_gain` becomes `interesting`. `min_gain` is in the units of the primary metric: ```toml pyproject.toml theme={null} [project.entry-points."piloop.verdict_rules"] min_gain = "piloop_min_gain:MinGain" ``` ```python piloop_min_gain/__init__.py theme={null} """``min_gain``: the built-in verdict, but a keep needs a gain of at least ``min_gain``. No state, no I/O. Use it when a tiny gain is not worth a keep. Each keep moves the head, and every later run must beat the head. The rule is the built-in rule ``bootstrap`` and changes one thing. The confidence, the guardrails and the reasons stay those of ``bootstrap``. """ from piloop.capabilities import Bootstrap, Case, Judgement, Outcome, number class MinGain(Bootstrap): """What ``bootstrap`` decides, but a keep with a gain below ``min_gain`` is ``interesting``.""" min_gain: float """The smallest gain that keeps, in the primary metric's units.""" def decide(self, case: Case) -> Judgement: judged = super().decide(case) name = case.primary.name if judged.verdict is not Outcome.KEEP or name not in case.bar.means: return judged gain = case.primary.direction.gain(case.candidate.means[name], case.bar.means[name]) if gain >= self.min_gain: return judged return Judgement(Outcome.INTERESTING, f'{judged.reason}; the gain is below min_gain {number(self.min_gain)}') ``` [Write a plugin](https://piloop.autoresearch.inc/plugins/overview.md) installs and selects it. ### The contract A verdict rule is a [pure](https://piloop.autoresearch.inc/plugins/overview.md#host-and-pure-contracts) provider, a subclass of `VerdictRule`. Import the types from `piloop.capabilities`. | Member | What it is | | - | - | | `VerdictRule.decide(case: Case) -> Judgement` | Return the verdict on `case.candidate` against `case.bar`, and the reason. | | `VerdictRule.bar(ancestry: Sequence[Node]) -> CandidateId` | Return the bar of a run that starts from `ancestry[0]`. The ancestry goes from it, parent by parent, to its root. | | `VerdictRule.raises_bar(node: Node, lineage: Lineage) -> bool` | Return whether `node`, a commit of a run that keeps on `lineage`, becomes the bar of the run's next commits. | | `VerdictRule.head(trunk: Sequence[Node]) -> CandidateId \| None` | Return the head among the trunk's keeps, oldest first, or None when there is none. | | `VerdictRule.on_moved(moved: Moved) -> Rebase \| Leave` | Return what follows a keep that lacks the head. By default its merge onto the head, and a seed on a conflict. | | `Case.primary: Criterion` | The primary metric of the candidate's track. | | `Case.guardrails: tuple[Criterion, ...]` | The campaign's guardrails, then the primary of every other track. | | `Case.candidate: Side` | The candidate that the rule judges. | | `Case.bar: Side` | What the candidate must beat: its run's bar ([https://piloop.autoresearch.inc/concepts/verdict#the-words](https://piloop.autoresearch.inc/concepts/verdict.md#the-words)). | | `Case.parent: Side \| None` | The candidate's parent, which the candidate changed. None for a candidate without one. | | `Criterion.name: str` | The metric's name, as the evaluate command reports it. | | `Criterion.direction: Direction` | Whether a larger or a smaller value is better. | | `Criterion.margin: float \| None` | The largest regression from the bar that a guardrail may show, in the metric's units. None on a primary. | | `Criterion.floor: float \| None` | The worst value that a candidate may have, whatever the bar, or None. | | `Criterion.fraction: float \| None` | The largest regression from the bar that a guardrail may show, as a fraction of the bar's value. A guardrail has a `margin` or a `fraction`, never both. | | `Criterion.ratio: Ratio \| None` | For a metric that piloop computes: its parts, whose units and samples are in each `Side` by their names. | | `Criterion.allowed(bar: float) -> float` | The largest regression from `bar`, the bar's value, in the metric's units. | | `Criterion.limit() -> str` | How a reason names the regression allowed: `margin 20` or `fraction 0.1`. | | `Criterion.past_floor(value: float) -> bool` | Whether `value` is past the floor, in the metric's direction. | | `Side.name: str` | How a reason names it: `candidate#73`. | | `Side.means: dict[str, float]` | The mean of each metric over its successful measurements. Empty when none succeeded. | | `Side.units: dict[str, dict[str, float]]` | The per-unit results of each metric that has them: unit id → the unit's mean contribution. | | `Side.samples: dict[str, list[float]]` | The samples of each metric that ran in pairs with the other side's: the i-th of the candidate and the i-th of the bar ran one after the other, on one worker. Empty without pairs. | | `Side.stopped: Stopped \| None` | Where its latest climb of the ladder stopped, if it stopped. `means` never include a rung. | | `Side.rungs: dict[RungName, dict[str, float]]` | The mean of each metric at each rung of the ladder that it measured. | | `Judgement.verdict: Outcome` | `keep`, `interesting`, `discard` or `crash`. | | `Judgement.reason: str` | Why, as piloop stores and shows it with the verdict. | | `Node.id: CandidateId` | The candidate's id. | | `Node.verdict: Outcome \| None` | None while it is pending, and for a commit that is never judged. | | `Node.made: Lineage \| Imported` | Where it was made: on the side of the run that made it, or imported. The baseline is on the trunk. | | `Node.means: dict[str, float]` | The mean of each metric over its successful measurements under the current evaluator. | | `Branch.root: CandidateId` | The candidate that the branch run started from. | | `Moved.candidate: Node` | The keep that lacks the head. | | `Moved.head: Node` | The head that it lacks. | | `Rebase.conflicts: Seed \| Leave` | What follows when the merge conflicts. | A rule has no error case: it returns a verdict for every case. The rule also decides the lineage: `bar`, `raises_bar`, `head` and `on_moved` read `Node`s (a candidate's id, verdict, where it was made, `Trunk`, `Branch` or `Imported`, and its means), and each has the built-in behaviour as its default. piloop applies the answers: it moves the head, merges, files the seed, and applies the `Piloop-Note` of a commit. ### Built in `bootstrap` is the rule of a campaign without `[plugins.verdict_rule]`. [The verdict](https://piloop.autoresearch.inc/concepts/verdict.md#the-verdicts) describes it. ## Stop rules On each pass, the daemon asks the stop rule of each active campaign whether the campaign goes on, holds, or stops. piloop gives the rule a `StopCase`: the facts that the rule asked for. The rule returns one of four answers: | Answer | What piloop does | | - | - | | `Go(left)` | The campaign goes on: piloop starts what it needs, within `left`. | | `Hold(reason)` | piloop starts no agent and no evaluation for the campaign until a check answers `Go` again. | | `Stop(reason)` | piloop stops the campaign. | | `Escalate(reason, then)` | piloop posts one escalation at priority 0 for each distinct `reason`, and then acts on `then`. | A hold is not a pause. piloop asks the rule again on each pass, and the hold ends when the rule answers `Go`. piloop does not stop a paused campaign, but it holds one. The built-in rule is `limits`. It holds the campaign while a fact of `hold_at` is unknown or at its limit. Else it stops the campaign at the first of its other limits that the campaign reaches. ### Example: end the campaign after a number of keeps This rule stops the campaign after `keeps` candidates are kept. It [calls the built-in rule](https://piloop.autoresearch.inc/plugins/overview.md#call-a-built-in-provider) `limits` and adds one limit, so the limits of `limits`, such as `max_hours` and `hold_at`, still apply: ```toml pyproject.toml theme={null} [project.entry-points."piloop.stop_rules"] after_keeps = "piloop_stop_after_keeps:AfterKeeps" ``` ```python piloop_stop_after_keeps/__init__.py theme={null} """``after_keeps``: stop the campaign after ``keeps`` keeps, or at a limit of the built-in rule. No state, no I/O. Use it for a campaign that wants a few good changes and no more, for example "make the build pass on the new compiler". It stops when enough candidates are kept. The rule is the built-in rule ``limits`` with one more limit, so ``max_hours``, ``max_tokens``, ``hold_at`` and the others still apply. """ from piloop.capabilities import KEEPS, Escalate, Fact, FactValue, Go, Hold, Limits, Stop, StopCase class AfterKeeps(Limits): """The campaign ends after ``keeps`` keeps. ``limits`` holds it and stops it at its own limits.""" keeps: int """The number of keeps that ends the campaign.""" def limits(self) -> dict[Fact[FactValue], float]: return super().limits() | {KEEPS: self.keeps} def check(self, case: StopCase) -> Go | Hold | Stop | Escalate: answer = super().check(case) keeps = case.facts.get(KEEPS) if not isinstance(answer, Go) or keeps is None or keeps < self.keeps: return answer return Stop(f'after_keeps: {keeps} keeps') ``` ```text theme={null} [plugins.stop_rule] provider = "after_keeps" keeps = 3 max_hours = 12 ``` ### Example: escalate when a keep costs too much This rule compares the fact that `spend` names, since the latest keep, with its value per keep so far. Past `escalate_at` times, it answers `Escalate` once for each keep; past `stop_at` times, it stops the campaign: ```toml pyproject.toml theme={null} [project.entry-points."piloop.stop_rules"] keep_cost = "piloop_keep_cost:KeepCost" ``` ```python piloop_keep_cost/__init__.py theme={null} """``keep_cost``: tell a human when a keep costs too much, and stop at a harder line. No state, no I/O. The rule compares the fact ``spend`` names, since the latest keep, with its value per keep so far. Past ``escalate_at`` times, it asks a human once for each keep; past ``stop_at`` times, it stops the campaign. It is the built-in rule ``limits`` with these two lines, so ``max_hours``, ``hold_at`` and the others still apply. Before the first keep there is no value per keep, and the rule is ``limits``. """ from pydantic import Field from piloop.capabilities import ( KEEPS, Escalate, Fact, FactValue, Go, Hold, Limits, Stop, StopCase, per_keep, reading, since_keep, ) class KeepCost(Limits): """The cost since the latest keep against the cost per keep: escalate at ``escalate_at``×, stop at ``stop_at``×.""" spend: str = 'spend' """The fact that is what the campaign spent so far, which a source of ``[plugins.fact_source] use`` reads.""" escalate_at: float = Field(default=2, gt=0) """Ask a human once the cost since the latest keep is this many times the cost per keep.""" stop_at: float = Field(default=4, gt=0) """Stop once the cost since the latest keep is this many times the cost per keep.""" def reads(self) -> list[Fact[FactValue]]: return list(dict.fromkeys([*super().reads(), reading(self.spend), KEEPS, since_keep(self.spend)])) def check(self, case: StopCase) -> Go | Hold | Stop | Escalate: answer = super().check(case) each, since = per_keep(case.facts, self.spend), case.facts.get(since_keep(self.spend)) if not isinstance(answer, Go) or each is None or since is None: return answer if since >= self.stop_at * each: return Stop( f'keep_cost: {self.spend} {since:g} since the latest keep, {self.stop_at:g} × the {each:g} per keep' ) if since >= self.escalate_at * each: # one escalation for each keep: the reason names the keep, not the amount, which changes on every pass return Escalate( f'keep_cost: since keep {case.facts.get(KEEPS)}, {self.spend} passed {self.escalate_at:g} × its value' f' per keep; it stops at {self.stop_at:g} ×' ) return answer ``` ```text theme={null} [plugins.stop_rule] provider = "keep_cost" spend = "spend" escalate_at = 2 stop_at = 4 ``` ### The contract A stop rule is a [pure](https://piloop.autoresearch.inc/plugins/overview.md#host-and-pure-contracts) provider, a subclass of `StopRule`. Its fields are its limits and other parameters. Import the types from `piloop.capabilities`. Most facts cost database queries on every pass, so piloop gets only those of `reads`. `piloop status` and the briefs join the parts of `describe` into the progress line: piloop formats no fact itself. | Member | What it is | | - | - | | `StopRule.while_held: WhileHeld` | What the closing review does when the campaign stops (`piloop campaign stop`) while the rule holds its launches: `run` it anyway, `skip` it, or `wait` until the hold ends. | | `StopRule.reads() -> list[Fact[FactValue]]` | Return the facts that `check` and `describe` read: piloop gets only those. | | `StopRule.describe(case: StopCase) -> list[str]` | Return where the campaign stands against the rule, in short parts: `run 7/200`, `2.1h/12h`. | | `StopRule.check(case: StopCase) -> Go \| Hold \| Stop \| Escalate` | Return whether the campaign goes on, holds or stops, or tell a human why and then do one of those. | | `StopRule.check_config(shape: Shape) -> str \| None` | Return why the rule cannot serve a campaign of `shape`, or None. By default, it serves every campaign. | | `StopCase.primary: str` | The name of the primary metric of the campaign's first track. | | `StopCase.direction: Direction` | Whether a larger or a smaller value of the primary metric is better. | | `StopCase.facts: Facts` | The facts that the rule reads (`reads`), from the campaign's fact sources. | `Go.left` says how much of each fact is left before the rule stops the campaign. piloop starts no more runs than `RUNS` leaves, and cuts each agent's timeout where `HOURS` runs out, so a run that starts just before the stop ends with it. An agent whose start comes after the rule answers `Stop` never starts: the pass that stops the campaign closes its row. `check_config` refuses a campaign that the rule cannot serve when the file loads: `limits` refuses a `target` on a campaign of more than one track. A fact is a typed key, `Fact[T]`, that a [fact source](#fact-sources) of the campaign supplies. `case.facts.get(KEEPS)` reads one, and `case.facts.get(reading('spend'))` one that a source reads by name. Every fact that a rule reads needs a source in the campaign, or piloop refuses the configuration. The facts of the built-in sources `campaign` and `optimization`, from `piloop.capabilities`, and the ratios over facts that a rule names (`spend`, the fact of what the campaign spent; `cost`, the evaluator's metric of what one evaluation costs): | Fact | Source | Type | What it is | | - | - | - | - | | `HOURS` (`hours`) | `campaign` | `float` | The hours since the first run started, without the time that the campaign was paused. A reopened campaign counts from its first run since the latest reopen. | | `TOKENS` (`tokens`) | `campaign` | `int` | The tokens that the agents of the campaign spent, summed over the agents that ended. A reopened campaign counts the agents since the latest reopen. | | `HEAD` (`head`) | `optimization` | `float` | The primary metric of the first track at the head, under the current evaluator. | | `RUNS` (`runs`) | `optimization` | `int` | The runs that hold a place under a limit on runs: those that count, and the attempts that may still count. A reopened campaign counts those since the latest reopen. | | `UNSETTLED` (`unsettled`) | `optimization` | `int` | The attempts whose outcome can still change: queued, running, or being scored. | | `WITHOUT_KEEP` (`without_keep`) | `optimization` | `int` | The finished runs that count since the latest keep, or since the campaign was last set active again. | | `KEEPS` (`keeps`) | `optimization` | `int` | The candidates of runs that are kept, on the trunk or on a branch. The baseline is not one. | | `EVALUATIONS` (`evaluations`) | `optimization` | `int` | The runs of the evaluate command under every evaluator: each measurement, failed ones and bar samples among them. | | `METRIC_SUMS` (`metric_sums`) | `optimization` | `dict[str, float]` | Each metric summed over every measurement. A rule that reads a cost metric names it. | | `ATTEMPTS` (`attempts`) | `optimization` | `int` | The runs that are an attempt at their idea: the open ones, and those whose end is one. | | `FUNNEL` (`funnel`) | `optimization` | `Funnel` | How far the attempts that ended and are judged got (`Funnel`). | | Ratio | What it is | | - | - | | `reading(name: str) -> Fact[float]` | The fact of a value that a source reads under `name`, such as `spend` in a result of `command`. | | `since_keep(name: str) -> Fact[float]` | How much the value of `reading(name)` grew since the latest keep: `_since_keep`. | | `per_keep(facts: Facts, spend: str) -> float \| None` | The fact `spend` per keep. Reads `reading(spend)` and `KEEPS`. | | `per_evaluation(facts: Facts, cost: str) -> float \| None` | The sum of the evaluator's metric `cost` per evaluation. Reads `EVALUATIONS` and `METRIC_SUMS`. | | `worker_spend(facts: Facts, spend: str, cost: str) -> float \| None` | The fact `spend` less the sum of the evaluator's metric `cost`. Reads `reading(spend)` and `METRIC_SUMS`. | | `per_run(facts: Facts, spend: str, cost: str) -> float \| None` | `worker_spend` per attempt. Reads `reading(spend)`, `METRIC_SUMS` and `ATTEMPTS`. | Each is None while a term is, or when its count is 0. A rule that reads a ratio asks for the facts of its terms. A stop that a person requests (`piloop campaign stop`) stops the campaign whatever the rule answers, also when the rule holds it. The closing review of that stop then follows the rule's `while_held` ([Spend outside piloop](https://piloop.autoresearch.inc/concepts/cost.md#spend-outside-piloop)). ### Built in `limits` is the rule of a campaign without `[plugins.stop_rule]`. [Stop conditions](https://piloop.autoresearch.inc/concepts/cost.md) lists its limits. ## Fact sources A **fact source** supplies the facts that a stop rule or a section reads. ### Example: read the spend from a billing endpoint ```python theme={null} import json import urllib.request from pathlib import Path from piloop.capabilities import FactReadError, Readings class Billing(Readings): """``billing``: what a billing endpoint says that the campaign spent, as ``spend``.""" url: str def names(self) -> tuple[str, ...]: return ('spend',) def read(self, repo: Path) -> dict[str, float]: try: with urllib.request.urlopen(self.url, timeout=30) as answer: return {'spend': float(json.load(answer)['total'])} except (OSError, KeyError, ValueError) as e: raise FactReadError(f'{self.url}: {e}') from e ``` ```toml theme={null} [project.entry-points."piloop.fact_sources"] billing = "my_package:Billing" ``` ```text theme={null} [plugins.fact_source] use = ["campaign", "optimization", "billing"] [plugins.fact_source.billing] url = "https://billing.example.com/campaigns/enwik8" ``` `hold_at = { spend = 1000 }` then holds the campaign at that spend. A fact source is a provider of `FACT_SOURCES`, a subclass of `FactSource`. `[plugins.fact_source] use` lists a campaign's sources; a campaign that does not set it has every installed source that is on by default (`DEFAULT`): `campaign`, the hours and tokens of any campaign, and `optimization`, its research. A list that adds a source lists those too, and a source with a table of its own must be in it. Two sources never supply the same fact. A `Readings` source (`piloop.plugins.fact_source.kit.readings`) is one that the daemon reads every `interval` seconds, outside any transaction. It implements `names` and `read`, and piloop does the rest: each reading is a `fact.read` event on the campaign, its one record. It posts a note at each of its `notices`, and escalates once when its readings start to fail. ### The contract | Member | What it is | | - | - | | `FactSource.DEFAULT: ClassVar[bool]` | Whether a campaign that does not set `[plugins.fact_source] use` has this source. Only the class that sets it is a default: a subclass is not (`is_default`). | | `FactSource.CHANGES: ClassVar[frozenset[type[EventModel]]]` | The events that `record` writes, which wake the campaigns' pass: its stop rule reads the new values. A source whose facts change only with the campaign's state (its ideas, runs, agents, status and config) names none. | | `FactSource.supplies() -> tuple[Fact[FactValue], ...]` | The facts that it supplies. Two sources of a campaign never supply the same one. | | `FactSource.gather(s: Session, campaign: Campaign, wanted: frozenset[Fact[FactValue]], now: datetime) -> dict[Fact[FactValue], FactValue]` | The values of the facts in `wanted` that it supplies, as of `now`. It reads with `s`, never writes. | | `FactSource.due(s: Session, campaign: Campaign) -> datetime \| None` | When the daemon calls `read` next: a time at or before now reads now. | | `FactSource.read(repo: Path) -> dict[str, float]` | Read values by name, at the campaign's repo root. The daemon calls it outside any transaction, when it is due. | | `FactSource.record(s: Session, campaign: Campaign, values: dict[str, float] \| FactReadError) -> None` | Record what `read` returned or raised. The caller commits. By default, nothing. | | `Readings.CHANGES: ClassVar[frozenset[type[EventModel]]]` | Each reading is a `fact.read` event. | | `Readings.interval: float` | The number of seconds between two readings. | | `Readings.notices: dict[str, list[float]]` | Values by the name that it reads. When a value passes one, the daemon posts one note to the inbox, at priority 1. | | `Readings.names() -> tuple[str, ...]` | The names of the values that it reads. | | `Readings.read(repo: Path) -> dict[str, float]` | Read every value of `names`. The daemon calls it outside any transaction, at the campaign's repo root. | | `Readings.supplies() -> tuple[Fact[FactValue], ...]` | `reading(name)` and `since_keep(name)` for each name of `names`. | | `Readings.gather(s: Session, campaign: Campaign, wanted: frozenset[Fact[FactValue]], now: datetime) -> dict[Fact[FactValue], FactValue]` | Each value of the latest reading that has it, None after a failed one, and its growth since the latest keep. | | `Readings.due(s: Session, campaign: Campaign) -> datetime \| None` | When the source reads next: `interval` after its latest reading, or now before the first. | | `Readings.record(s: Session, campaign: Campaign, values: dict[str, float] \| FactReadError) -> None` | Record a reading as a `fact.read` event, with a note for each notice that it passed. The caller commits. | The built-in `command` source runs a command that writes `{"metrics": {"": }}` to `$PILOOP_RESULT_FILE`, as the evaluate command does. [Spend outside piloop](https://piloop.autoresearch.inc/concepts/cost.md#spend-outside-piloop) configures it. piloop gives no fact a meaning: a stop rule or a section names the facts that it reads. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.