> ## Documentation Index
> Fetch the complete documentation index at: https://piloop.autoresearch.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Workers and capacity

> How many jobs each machine runs, and how to run agents and evaluations on more machines, such as a GPU box, with one daemon.

A **worker** runs the daemon's jobs: agent sessions and evaluations. `piloop serve` has one, called `serve`, and
`piloop worker` adds one on another machine. A job's code, inputs and results go through the
[CAS](https://piloop.autoresearch.inc/concepts/archive.md#the-cas), so the worker and the server share one.

## Capacity

A **pool** is the capacity of each worker in it: its slots, and the thresholds that keep jobs off a busy host, as in a
Kubernetes node pool. `piloop serve`'s worker is in the pool `serve`:

```toml theme={null}
[plugins.executor]
provider = "workers"

[plugins.executor.pools.serve]
agents = 2
evaluations = 1
max_load = 1.5      # the 1-minute load per CPU
min_memory_gb = 4
```

Each heartbeat carries a sample of the worker's host. A worker over `max_load` or under `min_memory_gb` takes no job,
and its slots count zero. The daemon launches an agent only into a free slot, and gives it to the campaign with the
fewest live agents. To change a pool's key while the daemon runs, set a **limit**. It wins over `config.toml` from
the daemon's next pass, and `piloop status` shows it with its reason:

```bash theme={null}
piloop limits set executor.pools.serve.evaluations=1 --reason "the GPU trains a model"
piloop limits unset executor.pools.serve.evaluations
```

```text theme={null}
executor.pools.serve.evaluations = 1: the daemon applies it at once
executor.pools.serve.evaluations: the limit is gone
```

`piloop worker list` shows each worker, and why it got no lease:

```text theme={null}
 worker  pool   seen    load     memory   disk      agents  evaluations  no lease
 serve   serve  6s ago  78.1/32  60.8 GB  379.3 GB  0       0            slots
```

## When a worker goes silent

The timing follows the two stages of Kubernetes, with its defaults. For `workers`, each number is a key of
[`[plugins.executor]`](https://piloop.autoresearch.inc/reference/configuration#plugins):

| Key | Then |
| - | - |
| `renew_seconds` | The worker sends a heartbeat. |
| `grace_seconds` | The worker is unreachable, and its slots count zero. |
| `toleration_seconds` | Its leases expire, and their jobs go back to the queue with a new token. |

A worker that hears nothing back stops its own jobs two heartbeats before its leases expire, so a job never runs
twice.

## On the server

Give the worker's [pool](#capacity) its slots in `config.toml`:

```toml theme={null}
[plugins.executor.pools.gpu]
agents = 2
evaluations = 1
```

Set `tls` in `[plugins.network]`: a certificate, `tls = { cert = "<cert.pem>", key = "<key.pem>" }`, or `"proxy"`
when a reverse proxy or an SSH tunnel ends TLS. Only a loopback address may go without it.

Then listen, serve the CAS, and make the worker's token. `piloop init` started a daemon on the socket only, so stop it
first:

```bash theme={null}
piloop stop
piloop serve --detach --listen 0.0.0.0:8443
piloop cas serve --listen 0.0.0.0:7701
piloop token create gpu1
```

`piloop cas serve` serves the server's `[plugins.cas]` until you stop it. When that is a bucket that the worker can
reach, skip it, and give the worker the same `[plugins.cas]`. `token create` prints the token once: piloop keeps only
its hash.

## On the worker

Install the same piloop as on the server, and log in to the harnesses that its agents use. In its `config.toml`:

```toml theme={null}
[plugins.cas]
provider = "cached"
local_max_gb = 50
remote = { provider = "http", url = "https://piloop.example.com:7701" }
```

Start it with the token:

```bash theme={null}
PILOOP_HTTPS_TOKEN=<token> piloop worker --server https://piloop.example.com:8443 --pool gpu
```

The worker's name is the token's name. It calls the server, and the server never calls it. For a certificate that
your system does not trust, set `SSL_CERT_FILE`. On the server, `piloop worker list` shows it:

```text theme={null}
 worker  pool   seen    load     memory    disk      agents  evaluations  no lease
 gpu1    gpu    6s ago  21.8/32  103.8 GB  816.4 GB  1       0            nothing
 serve   serve  6s ago  21.8/32  103.8 GB  816.4 GB  0       0            slots
```

The server refuses a worker with another piloop protocol, an unknown pool, a revoked token, or a `[plugins.cas]` that
is not the server's. The worker then stops its jobs and exits with the reason.

## Workers that the server starts

A pool with a **provisioner** starts its own workers when jobs wait, and stops them when they idle. The pool counts
the slots of its `max` workers, so its jobs wait on leases, and each waiting lease that no free slot takes starts a
worker. One that held no lease for `idle_seconds` is drained: it takes no more jobs, and the server stops it once it
holds none. One that cannot reach its server for `orphan_exit_seconds` exits by itself once it holds no job, so it never
outlives a stopped server.

On Modal, each worker is a sandbox of an image with `piloop` in it:

```toml theme={null}
[plugins.network]
listen = "0.0.0.0:8443"
tls = "proxy"
url = "https://piloop.example.com:8443"

[plugins.executor.pools.gpu]
agents = 0
evaluations = 1
runners = ["process"]

[plugins.executor.pools.gpu.provisioner]
max = 4

[plugins.executor.pools.gpu.provisioner.compute]
provider = "modal"
image = "ghcr.io/acme/piloop-worker:0.3"
gpu = "A100"
secrets = ["piloop-cas"]
```

Each worker gets a token of its own, and dials `url`. Give it the server's `[plugins.cas]` in its image, or as
`PILOOP_PLUGINS` in the provider's `env`.

`secrets` names the Modal Secrets that each sandbox gets as its environment. They are for the worker itself, such as
the credentials of a bucket that the worker reads. A job's keys, such as a harness's API key, are piloop secrets, which
the worker gets when the job starts. For an image in a private registry, name the Modal Secret that pulls it:

```toml theme={null}
[plugins.executor.pools.gpu]
runners = ["process"]

[plugins.executor.pools.gpu.provisioner]
max = 4

[plugins.executor.pools.gpu.provisioner.compute]
provider = "modal"
image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/piloop-worker:0.3"
registry = { kind = "aws-ecr", secret = "piloop-ecr-pull" }
```

`aws-ecr` takes `AWS_ROLE_ARN` and `AWS_REGION` (Modal's OIDC) or an IAM user's keys. `basic` takes
`REGISTRY_USERNAME` and `REGISTRY_PASSWORD`. The pull Secret is not in the sandbox's environment. The server needs Modal's credentials and `piloop[modal]`. To stop every
worker of the pool, drain them:

```bash theme={null}
piloop limits set executor.pools.gpu.provisioner.max=0 --reason "the GPU budget is spent"
```

`piloop worker list` shows each worker that the server started, until it ends.

## Behind an authenticating proxy

When a proxy in front of the server already authenticates every caller and stamps who it is in headers, such as an
Envoy `ext_authz` or an oauth2-proxy, let it be the only check. Name the headers that it stamps as claims, and give
each caller its access by the values of its claims:

```toml theme={null}
[plugins.network]
provider = "proxy"
listen = "0.0.0.0:8443"
url = "https://piloop.example.com"
claims = { org = "x-org-id", kind = "x-subject-kind", user = "x-user-id", principal = "x-principal" }
name_claims = ["user", "principal"]
rules = [
  { when = { org = "acme", principal = "key-0123" }, access = "worker" },
  { when = { org = "acme", kind = "service_account" }, access = "act", caller = "service" },
  { when = { org = "acme" }, access = "act" },
]
```

The first rule that matches wins, and a caller that no rule matches is refused. A worker names itself in
`x-piloop-worker`; anyone else is named by the first of its `name_claims` that is not empty, here its user, else its
principal. A caller signs its writes as `human:<name>`, or as `service:<name>` when its rule says `caller = "service"`.
The proxy must remove any copy of these headers that a caller sends, and only the proxy may reach `listen`:
whoever reaches it can claim anything. The server refuses piloop's own tokens (`plt_`) on this network: the proxy is
the one authentication.

Workers share one credential that the proxy accepts and that the `worker` rule matches. Store it as a secret, which
each provisioned worker gets, with its name:

```bash theme={null}
printf %s "$WORKER_KEY" | piloop secret set PILOOP_WORKER_CREDENTIAL
```

Give a provisioned worker the same network with `PILOOP_PLUGINS='{"network": {"provider": "proxy"}}'` in the
provider's `env`.

## What an agent on a worker gets

* The environment of the worker, and the [secrets](https://piloop.autoresearch.inc/guides/secrets.md) that its job names, which the server sends when
  the job starts.
* A clone of the campaign's repo and a copy of the shared [workspace](https://piloop.autoresearch.inc/concepts/campaigns.md#files). Its commits
  and its workspace writes come back when it ends. The worker keeps its clones under its home: a new clone starts
  from a copy of the whole repo that the server builds once per campaign, then takes only what is newer.
* Its `piloop` commands, which the worker relays to the server while the agent's lease is live.

Until it ends, its log files are in the worker's `$PILOOP_HOME`. Its session goes to the server's CAS a few seconds
after it writes, and at least every 30 seconds. So `piloop attach` and `piloop status` show what it wrote, and a lost
worker loses at most those 30 seconds.

> [!WARNING]
>
> An agent on a worker runs as the worker's user, so it holds the worker's token. It can do whatever the worker can
> do on the server, as an agent of `piloop serve` can open its database. Run a worker as a user that you give that
> trust.

## Faults

| What happens | What piloop does |
| - | - |
| A worker dies during a run | The run ends `worker_lost`. It is not an attempt. |
| A worker dies during an evaluation | The evaluation is queued again. Partial output is never scored. |
| A late report | It is refused, and kept as a `lease.stale_report` event. Its commits and logs stay in the CAS, by address. |
| The same report twice | The second gets the first's answer. |
| A report names blobs that the server's CAS lacks | The job is lost, with the blobs it names. |
| A worker stops or dies before it sends a report | The report waits in `$PILOOP_HOME/spool/<worker>`, and goes out when the worker starts again. |
| The server does not answer | The worker tries each call again. Its log says `worker.unreachable` at the first failure, then once a minute. |
| An agent runs `piloop` after its lease ended | The call is refused. |
| The machine sleeps | A gap longer than the grace period is the daemon's own absence: nothing expires. |
| `piloop serve` restarts | Its worker finds its leases again and re-attaches to each agent whose worktree is there. |
| A started worker does not register within `register_seconds` | It is stopped, and the pool waits 30 s, doubling to 10 minutes, before it starts another. |
| The platform refuses a start: a quota, no GPU | The pool backs off the same way. Three refusals in a row are an escalation in the inbox. |
| The platform loses a worker, or it goes silent | It is stopped. Its jobs come back as for any lost worker, and the pool starts another if they still wait. |
| The platform has a worker that the server has no record of | It is stopped. Two servers on one account never stop each other's: each tags its own. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.