Skip to main content
A worker runs the daemon’s jobs: agent sessions and evaluations. piloop serve has one, called serve, and piloop worker adds one on another machine. A job’s code, inputs and results go through the CAS, so the worker and the server share one.

​
Capacity

A pool is the capacity of each worker in it: its slots, and the thresholds that keep jobs off a busy host, as in a Kubernetes node pool. piloop serve’s worker is in the pool serve:
Each heartbeat carries a sample of the worker’s host. A worker over max_load or under min_memory_gb takes no job, and its slots count zero. The daemon launches an agent only into a free slot, and gives it to the campaign with the fewest live agents. To change a pool’s key while the daemon runs, set a limit. It wins over config.toml from the daemon’s next pass, and piloop status shows it with its reason:
piloop worker list shows each worker, and why it got no lease:

​
When a worker goes silent

The timing follows the two stages of Kubernetes, with its defaults. For workers, each number is a key of [plugins.executor]: A worker that hears nothing back stops its own jobs two heartbeats before its leases expire, so a job never runs twice.

​
On the server

Give the worker’s pool its slots in config.toml:
Set tls in [plugins.network]: a certificate, tls = { cert = "<cert.pem>", key = "<key.pem>" }, or "proxy" when a reverse proxy or an SSH tunnel ends TLS. Only a loopback address may go without it. Then listen, serve the CAS, and make the worker’s token. piloop init started a daemon on the socket only, so stop it first:
piloop cas serve serves the server’s [plugins.cas] until you stop it. When that is a bucket that the worker can reach, skip it, and give the worker the same [plugins.cas]. token create prints the token once: piloop keeps only its hash.

​
On the worker

Install the same piloop as on the server, and log in to the harnesses that its agents use. In its config.toml:
Start it with the token:
The worker’s name is the token’s name. It calls the server, and the server never calls it. For a certificate that your system does not trust, set SSL_CERT_FILE. On the server, piloop worker list shows it:
The server refuses a worker with another piloop protocol, an unknown pool, a revoked token, or a [plugins.cas] that is not the server’s. The worker then stops its jobs and exits with the reason.

​
Workers that the server starts

A pool with a provisioner starts its own workers when jobs wait, and stops them when they idle. The pool counts the slots of its max workers, so its jobs wait on leases, and each waiting lease that no free slot takes starts a worker. One that held no lease for idle_seconds is drained: it takes no more jobs, and the server stops it once it holds none. One that cannot reach its server for orphan_exit_seconds exits by itself once it holds no job, so it never outlives a stopped server. On Modal, each worker is a sandbox of an image with piloop in it:
Each worker gets a token of its own, and dials url. Give it the server’s [plugins.cas] in its image, or as PILOOP_PLUGINS in the provider’s env. secrets names the Modal Secrets that each sandbox gets as its environment. They are for the worker itself, such as the credentials of a bucket that the worker reads. A job’s keys, such as a harness’s API key, are piloop secrets, which the worker gets when the job starts. For an image in a private registry, name the Modal Secret that pulls it:
aws-ecr takes AWS_ROLE_ARN and AWS_REGION (Modal’s OIDC) or an IAM user’s keys. basic takes REGISTRY_USERNAME and REGISTRY_PASSWORD. The pull Secret is not in the sandbox’s environment. The server needs Modal’s credentials and piloop[modal]. To stop every worker of the pool, drain them:
piloop worker list shows each worker that the server started, until it ends.

​
Behind an authenticating proxy

When a proxy in front of the server already authenticates every caller and stamps who it is in headers, such as an Envoy ext_authz or an oauth2-proxy, let it be the only check. Name the headers that it stamps as claims, and give each caller its access by the values of its claims:
The first rule that matches wins, and a caller that no rule matches is refused. A worker names itself in x-piloop-worker; anyone else is named by the first of its name_claims that is not empty, here its user, else its principal. A caller signs its writes as human:<name>, or as service:<name> when its rule says caller = "service". The proxy must remove any copy of these headers that a caller sends, and only the proxy may reach listen: whoever reaches it can claim anything. The server refuses piloop’s own tokens (plt_) on this network: the proxy is the one authentication. Workers share one credential that the proxy accepts and that the worker rule matches. Store it as a secret, which each provisioned worker gets, with its name:
Give a provisioned worker the same network with PILOOP_PLUGINS='{"network": {"provider": "proxy"}}' in the provider’s env.

​
What an agent on a worker gets

  • The environment of the worker, and the secrets that its job names, which the server sends when the job starts.
  • A clone of the campaign’s repo and a copy of the shared workspace. Its commits and its workspace writes come back when it ends. The worker keeps its clones under its home: a new clone starts from a copy of the whole repo that the server builds once per campaign, then takes only what is newer.
  • Its piloop commands, which the worker relays to the server while the agent’s lease is live.
Until it ends, its log files are in the worker’s $PILOOP_HOME. Its session goes to the server’s CAS a few seconds after it writes, and at least every 30 seconds. So piloop attach and piloop status show what it wrote, and a lost worker loses at most those 30 seconds.
An agent on a worker runs as the worker’s user, so it holds the worker’s token. It can do whatever the worker can do on the server, as an agent of piloop serve can open its database. Run a worker as a user that you give that trust.

​
Faults