> ## Documentation Index
> Fetch the complete documentation index at: https://piloop.autoresearch.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Workers on other machines

> Run agents and evaluations on more machines, such as a GPU box, with one daemon.

A **worker** runs the daemon's jobs: agent sessions and evaluations. `piloop serve` has one, and `piloop worker` adds
one on another machine. A job's code, inputs and results go through the [CAS](https://piloop.autoresearch.inc/concepts/artifact-store.md), so the
worker and the server share one.

## On the server

Give the worker's [pool](https://piloop.autoresearch.inc/architecture/leases.md#capacity) its slots in `config.toml`:

```toml theme={null}
[plugins.executor.pools.gpu]
agents = 2
evaluations = 1
```

Set `tls` in `[plugins.network]`: a certificate, `tls = { cert = "<cert.pem>", key = "<key.pem>" }`, or `"proxy"`
when a reverse proxy or an SSH tunnel ends TLS. Only a loopback address may go without it.

Then listen, serve the CAS, and make the worker's token:

```bash theme={null}
piloop serve --detach --listen 0.0.0.0:8443
piloop cas serve --listen 0.0.0.0:7701
piloop token create gpu1
```

`piloop cas serve` serves the server's `[plugins.cas]` until you stop it. When that is a bucket that the worker can
reach, skip it, and give the worker the same `[plugins.cas]`. `token create` prints the token once: piloop keeps only
its hash.

## On the worker

Install the same piloop as on the server, and log in to the harnesses that its agents use. In its `config.toml`:

```toml theme={null}
[plugins.cas]
provider = "cached"
local_max_gb = 50
remote = { provider = "http", url = "https://piloop.example.com:7701" }
```

Start it with the token:

```bash theme={null}
PILOOP_HTTPS_TOKEN=<token> piloop worker --server https://piloop.example.com:8443 --pool gpu
```

The worker's name is the token's name. It calls the server, and the server never calls it. For a certificate that
your system does not trust, set `SSL_CERT_FILE`. On the server, `piloop worker list` shows it:

```text theme={null}
 worker  pool   seen    load     memory    disk      agents  evaluations  no lease
 gpu1    gpu    6s ago  21.8/32  103.8 GB  816.4 GB  1       0            nothing
 serve   serve  6s ago  21.8/32  103.8 GB  816.4 GB  0       0            slots
```

The server refuses a worker with another piloop protocol, an unknown pool, a revoked token, or a `[plugins.cas]` that
is not the server's. The worker then stops its jobs and exits with the reason.

## Workers that the server starts

A pool with a provisioner starts its workers when its jobs wait, and stops them when they idle. On Modal, each worker
is a sandbox of an image with `piloop` in it:

```toml theme={null}
[plugins.network]
listen = "0.0.0.0:8443"
tls = "proxy"
url = "https://piloop.example.com:8443"

[plugins.executor.pools.gpu]
agents = 0
evaluations = 1
runners = ["process"]

[plugins.executor.pools.gpu.provisioner]
max = 4

[plugins.executor.pools.gpu.provisioner.compute]
provider = "modal"
image = "ghcr.io/acme/piloop-worker:0.3"
gpu = "A100"
secrets = ["piloop-cas"]
```

Each worker gets a token of its own, and dials `url`. Give it the server's `[plugins.cas]` in its image, or as
`PILOOP_PLUGINS` in the provider's `env`. The server needs Modal's credentials and `piloop[modal]`. To stop every
worker of the pool, drain them:

```bash theme={null}
piloop limits set executor.pools.gpu.provisioner.max=0 --reason "the GPU budget is spent"
```

`piloop worker list` shows each worker that the server started, until it ends.

## What an agent on a worker gets

* The environment of the worker, with the keys that its [MCP servers](https://piloop.autoresearch.inc/guides/mcp-servers.md) need.
* A clone of the campaign's repo and a copy of the shared [workspace](https://piloop.autoresearch.inc/concepts/campaigns.md#shared-files). Its commits
  and its workspace writes come back when it ends.
* Its `piloop` commands, which the worker relays to the server while the agent's lease is live.

Until it ends, its output is in the worker's `$PILOOP_HOME`, and `piloop attach` names the worker.

> [!WARNING]
>
> An agent on a worker runs as the worker's user, so it holds the worker's token. It can do whatever the worker can
> do on the server, as an agent of `piloop serve` can open its database. Run a worker as a user that you give that
> trust.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.