Skip to main content
Every project has one shared workspace, $PILOOP_HOME/projects/<id>/workspace. It is linked into every agent’s worktree as ./workspace/. Agents leave notes, logs, evaluation outputs and analysis arrays there. On a long campaign it grows to hundreds of gigabytes, much of it copies. piloop stores workspace files in a content-addressed artifact store once nothing needs them any more. It removes the local copies later, and piloop restore brings them back.

​
What is archived, and when

The daemon archives in the background, one job at a time. Its scheduling loop never waits for an archiving.
  • When an idea is done (its last run is judged), the entries that belong to it are archived: workspace/idea<id>… (idea12/, idea12-canary.log). Workers are told to put their output there.
  • When a campaign stops and none of its agents is alive any more, everything left in its workspace is archived.
  • Never archived: NOTES.md and research/ at the top of the workspace. These small knowledge files stay where research sweeps read them.
An archiving stores every file and symlink, stores a manifest that lists them, and checks that the store holds all of it. Only then does it record the archiving in the archives table. It removes nothing: the local copies stay, so an agent or a human that still reads a file, for example to answer an escalation, finds it.

​
When local copies go

The daemon removes the workspace copies of an archiving later, and records when in archives.pruned_at:
  • By age: 7 days after the archiving.
  • Under disk pressure: while the disk of the store has less than min_free_gb ([storage], 20 GB by default) free, the oldest archivings first.
It removes only a file that still holds what was stored (same size and hash), and a symlink that still points where it did. A file that an agent wrote since stays, and so does its directory. It checks age and pressure every hour, before the next archiving, and checks pressure again before it stores anything. Storing a file writes its compressed copy first, so archiving needs room on the disk of the store. A workspace larger than the free space above min_free_gb is stored in batches: before each batch, the daemon removes older local copies while the disk is short, then measures the free space again. When not one more file fits, the rest is not stored, and the daemon writes an anomaly.disk event on the project, with the bytes needed, the bytes free and the reserve. Only the last batch of a job that stored everything marks the job as done (archives.complete). A job that stops short, because of the disk, a failure or a stop of the daemon, stays due, and its next run stores the rest. After a failed or deferred archiving the daemon waits 10 minutes before the next one. Which archivings are due is computed from the database. A daemon that stops in the middle of a backlog picks it up where it was when it starts again. Content that was already stored is found by its hash and is not stored twice. The first daemon start after an upgrade archives every done idea and stopped campaign that is already there. piloop store status shows how many are still due. Workers are told to put their output under workspace/idea<id>/, so it is archived with their idea. They are also told not to copy the code tree into the workspace (every candidate’s tree is in git) and to leave evaluator output in the project’s own cache.

​
Getting files back

piloop restore reads the files from the local store, or downloads them from the remote. A file that is already in the workspace with other content is kept and listed, never overwritten. Agents may run it too.

​
How the store works

One store serves every project of a PILOOP_HOME, in $PILOOP_HOME/store.
  • Content addressing. A blob is addressed by the BLAKE3 hash of its whole content. A file that appears in many places, or in two campaigns, is stored once.
  • Compression. Blobs are kept zstd-compressed. JSON and logs shrink about tenfold; arrays that are already compressed (.npz) barely shrink.
  • Streaming. Files of any size are read and written in 1 MiB pieces, never held in memory.
  • Manifests. A manifest lists the paths of one archiving, each with its blob, size and mode (or a symlink’s target). It is a blob too, and the archives table records its address.
Nothing is ever deleted from the store. There is no garbage collection.

​
A remote

Set [storage.remote] in $PILOOP_HOME/config.toml to mirror the store. Its kind picks the backend, and the backend owns the other keys of the table. s3 and gcs are built in; gcs needs the gcs extra (uv tool install --reinstall 'piloop[gcs]'). A package can add a backend: see storage backends. The keys of [storage] itself hold for every backend. The configuration reference lists every key and shows the same store on S3 and on GCS.
piloop checks [storage.remote] against its backend when it opens the store: piloop serve, piloop restore and piloop store fail on a key the backend does not know.
  • Mirror. With a remote, every blob is uploaded as it is stored. For S3, boto3 finds the credentials (environment, profile or instance role).
  • Storage class. Each object gets Glacier Instant Retrieval (millisecond reads at a fraction of the price of STANDARD), except objects under standard_below: Glacier IR bills each object as at least 128 KB, so small ones are cheaper as STANDARD.
  • Eviction. Every hour, before the next archiving, the daemon drops the local copies of blobs that the remote holds: those older than evict_after_days, then the oldest ones until the local store fits local_max_gb. piloop store evict does the same on demand. It first uploads any blob that was stored before you set the remote.
  • Remote only. With evict_after_days = 0, the local store is only a staging area: each blob’s local copy is dropped as soon as it is uploaded, and piloop restore downloads it.
  • One remote per store. The first upload writes the remote’s URL to $PILOOP_HOME/store/remote.url. A store whose blobs went to one remote refuses to open with another remote, or with none: the blobs it marked as uploaded are only in the first one. To move to a new remote, copy the objects there, then write its URL into that file.
piloop store verify checks that every blob of every archive is still there. It hashes the local copies again and asks the remote for the evicted ones.