The runner protocol

What a machine has to say to take work from this instance, and what it gets back. Everything here is HTTP and JSON: there is no SDK to install, and a runner written in an afternoon in any language is a supported runner.

The core job lifecycle is claim, heartbeat, logs and report. Additional endpoints handle artifacts, caches, orchestration and workspace maintenance. Each uses ordinary HTTP so an operator can inspect a failed exchange with curl.

If you do not want to write one, this instance ships a runner that speaks it: ./buddy build:runner --target linux-x64 compiles the executor into a single file with nothing to install on the machine that runs it. The compiled runner does not load .env or bunfig.toml from its working directory. Supply its URL with --url and its credential through REVIEWOS_RUNNER_TOKEN in the runner service environment. Application preload code must never execute before runner policy. See self-hosting. The rest of this page is for the case where you would rather write your own, which is a supported thing to do.

The shipped runner retries failed log deliveries with the original sequence, including connection failures and temporary proxy errors. It makes at most five attempts. A command whose output cannot be delivered fails and is stopped; output loss never counts as success. After a command exits, background-held output pipes retain a two-second grace. If a delivery is still pending at that boundary, the job fails because buffered command output cannot be verified.

Registering a runner

A runner is a row an operator creates, with a token it holds and a scope that says what it may see:

The token is stored as a SHA-256 hash, so it is shown once at creation and never again. A registration token in a database in plain text is one in every backup.

Labels are one per line, and a job's runs-on must be satisfied by them. They are a matching rule, not a permission: reach is decided by scope first, and a runner registered for one repository being handed another's source is not a scheduling mistake but the instance giving somebody's private code to a machine its owner chose.

Versioning

Every request may carry the version it speaks:

X-Runner-Protocol: 1

Every answer says what this server speaks, on the ordinary responses as well as the refusals:

X-Runner-Protocol-Supported: 1

A runner that sends no version is assumed to be speaking the oldest one this server still accepts, so a fleet written before the header existed keeps working and learns what to send from the response.

A version this server cannot speak is refused with 426 Upgrade Required, in either direction, and the body says which end is behind. Upgrading a hundred machines and upgrading one server are different afternoons, and a runner left to guess produces a job that hangs rather than an error anybody can act on.

The check runs before the credential is looked at: a runner that cannot be spoken to is going to misread whatever it is handed, and telling it the token is fine first only delays the confusion.

The five calls

All five are POST and all five are exempt from CSRF, because a machine holds a bearer token and no cookie - there is no ambient credential for a forged cross-site request to ride. Four of them take and return JSON; the artifact upload takes the file itself as the body, for the reason given there.

POST /api/runner/claim

Authenticated with the registration token. Asks for one job.

Protocol 2 also accepts { "accept_work": false } for a capacity hold. The controller updates the runner's last-seen time and delivers a requested stop, but does not select or lease a job. A held runner also sends hold_reason, the sentence it logged, for example { "accept_work": false, "hold_reason": "Insufficient workspace disk space at /work: 9355 MiB available, 24576 MiB required" }. The controller stores it on the runner, with the time the hold began, and clears both on the first claim that accepts work again. The fleet listing shows it as hold, and a queued job whose only matching runners are holding says so on the run page and in the run API. The field is optional in both directions: an older controller ignores it, and a hold from an older runner that sends none is recorded with a generic reason. Protocol 1 runners remain supported. Deploy the controller before the new runner: a protocol 1 controller refuses version 2 with 426 before it could ignore the hold and lease work.

Set REVIEWOS_RUNNER_MIN_FREE_DISK_MB in the runner's own environment to require that many MiB free before each claim. The portable and Buddy runners use the same policy. Zero, the compatibility default, disables the floor. Invalid explicit settings and failed filesystem measurements hold work. The configured workspace root must already exist when a floor is enabled. The measurement uses blocks available to the runner's user on both its workspace and process temp filesystems.

Choose a floor that includes checkout, installation and build growth before a native tool boots, as well as host headroom. For example, a native pool can set REVIEWOS_RUNNER_MIN_FREE_DISK_MB=12288 after checking its workload. This is admission, not a reservation or disk quota: other host processes can still spend space after the claim. The operator must provide sustained real headroom. A held runner continues safe terminal-workspace recovery and control polls, logs its hold at most once per minute, and resumes ordinary claims when capacity returns.

Set REVIEWOS_RUNNER_MAX_LOAD_PER_CPU to hold admission when the one-minute host load divided by available logical CPUs exceeds that value. For example, 1.5 allows a load of at most 6 on four available CPUs. Zero, the default, disables this additional policy. Invalid settings, unavailable measurements, and Windows hosts with an enabled load policy refuse work. Control polls and automatic recovery remain active during a hold. This guards admission only; operators must still coordinate other host workloads after a job starts.

Restarts, outages and the lease

A runner gives its lease up 20 seconds before the instance would (its clock starts when the heartbeat's answer arrives, after the instance set the expiry), heartbeats every 10 seconds, and stops the job's processes when it does. The instance waits RECLAIM_GRACE_SECONDS (30) past a lapse before another claim or the reclaim sweep may hand the job on, so a runner that was only cut off has stopped its work before the next attempt can start anywhere.

SIGTERM or SIGINT abandons the job in hand: steps, services and every process carrying the job's RUNNER_TRACKING_ID are stopped within 15 seconds (under launchd's 20 before SIGKILL), nothing is reported, and the instance requeues the job when the lease lapses. A second signal exits at once. Each attempt is also written to a ledger (~/.local/share/reviewos/job-attempts/, beside a configured journal root, or inside REVIEWOS_WORKSPACE) until its processes are verified gone; before every claim the runner stops whatever an attempt whose runner process is gone left behind, found by tracking id or by a working directory inside that attempt's workspace, and holds new work if any of it survives SIGKILL.

Step output and conclusions are retried through an unreachable instance - a deploy restarting its gateway - for up to a lease (REVIEWOS_RUNNER_DELIVERY_DEADLINE_MS), and a claim that cannot connect is retried with a growing pause rather than ending the process.

{ "job": null }

No work is a 200, not a 404. A runner polling an idle instance is not making a mistake, and an error status for the common case fills a fleet's logs with red that means nothing.

A job no runner claims within CI_QUEUE_TIMEOUT_HOURS (24 by default, as in Actions) is cancelled by the scheduled lease sweep, with the reason on the job, and its run settles. The wait counts from when the job last entered the queue, so a job a lost runner gave back starts again. A run held in its concurrency group or waiting on an approval is not timed out: it is waiting on a person, not a machine.

When there is work, the answer carries what the job is, the steps to run, and a credential minted for this claim alone. Everything afterwards authenticates with that, not with the registration token: the registration token is installed once and never rotated, and it must not be the thing travelling on every call.

Run commands inherit shell and working directory independently from the pinned workflow and job, then apply explicit step overrides. A called job resolves the nearest call's immutable version, including nested calls. Generated jobs retain their uploaded step values. A missing or inconsistent called definition fails the claimed job before credentials or commands are delivered; that failure is neither retried nor tolerated. A stale claim cannot fail a replacement lease.

Failures before a step starts are retained on the job itself, bounded to 2000 characters and redacted with that job's delivered secrets. Run detail includes this error even when no step attempt exists.

It carries no repository secret. By the threat model an untrusted run - a pull request from a fork - never receives one at all.

POST /api/runner/heartbeat

Authenticated with the job credential. Extends the lease.

A lease is sixty seconds. Short and renewed, rather than long: a job held by a machine that fell over is stuck for as long as its lease, and the machine cannot be asked.

A refused heartbeat is a 409, and it means stop working. The lease lapsed or the run was cancelled, and anything reported afterwards will be refused anyway.

POST /api/runner/logs

Authenticated with the job credential. Appends output.

{ "sequence": 1, "content": "compiling\n", "stream": "stdout" }

The sequence is the runner's own counter, and it is what makes at-least-once delivery safe: sending the same chunk twice stores it once and answers duplicate: true. A runner that did not hear an answer should send it again.

Or as events, when the runner has more than bytes to say:

{
  "sequence": 2,
  "events": [
    { "type": "group", "text": "Install dependencies" },
    { "type": "line", "text": "bun install", "at": "2026-08-13T10:00:01Z", "stream": "stdout" },
    { "type": "endgroup" }
  ]
}

Text is not deprecated and will not be. A runner that sends what its build printed is doing the honest thing; events exist for the four things text cannot carry and that cannot be recovered from it afterwards:

• which lines were grouped, because ::group:: is a marker one CI product uses and a string somebody's build may legitimately print;
• when the job printed each line, because the time this server received a chunk is not that - a runner batching a hundred lines for one round trip would have them all arrive in the same millisecond;
• which stream each line came from, since a chunk carries one stream and a runner interleaving stdout and stderr would otherwise have to split every chunk or lie;
• where colour started and stopped, so a page can render it or ignore it per reader rather than showing escape bytes as text.

Four event types: line, group, endgroup, image. Nesting is flat, like every CI product that has this. A group nobody closed is closed at the end, because a build that fails inside one never gets to close it - and that is the group somebody came to read.

An image is an artifact of this run, named rather than linked:

{ "type": "image", "text": "the failing screen", "artifact": "screenshot.png" }

Upload the bytes as an artifact first, then print the event naming it. There is no URL field and there will not be one: the only thing a build may put on a page a colleague opens is something it uploaded here, under its own run. A URL would be a request the reader's browser makes to somebody else's server every time the log is opened, which is a tracking pixel a build can install in a page other people read.

An image with no artifact is dropped rather than kept as a line - a caption with nothing under it is not an image. text is the alt text, and it is what a plain-text reader gets: the stored text form of the event reads [image: the failing screen - artifact screenshot.png], so curl of a log says what was there instead of leaving a gap.

What the log will actually render in place is decided by the bytes, not by the Content-Type of the upload: PNG, JPEG, GIF and WEBP, at most 8MB. SVG is refused - it is a document that can carry script, and rendering it in place would run it. Anything else, and anything larger, stays downloadable from the run's artifacts and shows in the log as a line saying the image is not available. An artifact that has expired shows the same line: the log outlives the bytes, and a broken image with no explanation is the worst way to say so.

The answer reports how many events were understood. An event type this server has not learned is dropped, not refused: a newer runner mid-fleet-upgrade should not lose the lines around it, and the protocol version is what tells it the server is older.

The text form is stored alongside, derived from the events, so everything that reads a log as text keeps working without knowing any of this exists.

There is a ceiling on a job's total output, enforced on the way in. Past it the append is accepted and dropped rather than refused - a 4xx there makes a correct runner retry a chunk that will never be wanted.

POST /api/runner/report

Authenticated with the job credential. Records what happened.

{ "state": "succeeded" }

A duplicate report is a 200, marked duplicate: true. At-least-once delivery means a correct runner will say it twice, and answering 409 to that is how one retries forever.

cancelled is the one report that survives a revoked lease. When a run is cancelled every lease it holds is expired at that moment - which is what stops a worker that already lost its connection publishing a success over a run somebody stopped - and a runner that heard the cancellation, stopped, and came back to say so is still the holder. "I stopped" cannot fabricate a verdict the way "I succeeded" can, so that one is accepted and nothing else is.

The host runner treats a definitive refusal of its log upload as a lease revocation too. Output can arrive before the next heartbeat, so it stops the process tree and reports cancelled immediately. Recoverable transport errors and proxy backpressure keep their bounded retries.

A run whose runner never acknowledges is ended by the control plane two lease periods after the request. Cooperative first, forceful after a deadline.

POST /api/runner/artifacts

Authenticated with the job credential. Publishes a file for somebody to collect later.

The body is the file. A runner is a program, the thing it has is a stream of bytes, and asking it to build a multipart form around them would mean every implementer pulling in a library to send one file. The name and the retention ride in headers, so nothing needs parsing before the bytes can be written.

curl -X POST "$SERVER/api/runner/artifacts" \
  -H "Authorization: Bearer $JOB_TOKEN" \
  -H "X-Artifact-Name: coverage.lcov" \
  -H "X-Artifact-Retention-Days: 30" \
  --data-binary @coverage.lcov

The answer carries the SHA-256 the bytes were stored under and the date the artifact stops being available. Storage is content-addressed, so the same file published by eight jobs of a matrix costs one copy, and a re-run producing identical output costs nothing.

The same name twice is idempotent when the content matches - a 200 marked duplicate, which is what a runner that did not hear the first answer gets. The same name with different content is a 409: silently replacing an artifact somebody may already have downloaded leaves two people holding different files under one name and no way to tell.

Automatic artifact-paths collection prefixes each name with its producing job-<id>-, so matrix copies retain separate logs. This prefix is reserved: another job’s credential cannot publish there. Long collected paths retain a fingerprint and filename extension inside the 200-character limit.

POST /api/runner/artifacts/fetch accepts { "name": "out/report.txt" } for an explicit run-wide upload or a uniquely collected path. Multiple collected producers return 409; add job_id to select one. That selection is still scoped to the caller’s run. The generated command accepts reviewos-download NAME [output-path] [producer-job-id]; a producer can publish its REVIEWOS_JOB_ID as a step output for a dependent job.

There are two ceilings, per artifact and per run. One without the other is not a ceiling - a per-artifact limit alone is walked around by a matrix of fifty jobs each uploading just under it.

Artifacts are not dependency caches. A cache is an optimisation the instance may drop whenever it likes; an artifact is something a person asks for by name three weeks later, and the two want opposite retention rules.

Upgrading a fleet

1. Upgrade the server first. It keeps speaking to every runner version in its supported range, and a fleet is never upgraded atomically.
2. Upgrade runners at whatever pace suits. Each one that reconnects is checked, and any that is too old is refused with a sentence saying so.
3. Watch X-Runner-Protocol-Supported on ordinary responses. A version about to be retired shows up there long before it stops working.

Retiring a version - raising the minimum - is a breaking change for anybody who has not upgraded, and it belongs in a release note rather than in a patch.

Fetching a repository, cheaply

A fleet's expensive operation is not running the job: it is the clone at the start of it. Ten runners taking the same repository at the same moment is ten upload-pack processes computing ten nearly identical packfiles, and that is the most expensive thing this server does.

Three things make it cheaper, in the order worth trying them.

Let the client use the checkpoint bundle. An instance running the write-ahead log writes a full bundle of each repository to its blob store and advertises it on the repository itself, through git's own bundle-uri mechanism. A client that understands it fetches that bundle over plain HTTP - from the instance, a bucket, or a CDN in front of either - and then asks upload-pack only for what has landed since. Nothing is needed on the runner: git 2.46 and newer will use an advertised bundle by default, and silently falls back to an ordinary clone if it cannot fetch one. This is why GIT_WAL earns its keep on instances that will never run a second node.

Keep a reference clone on the runner. A long-lived machine that builds the same repository repeatedly should not start from nothing each time:

git clone --reference /var/cache/reviewos/acme-app.git https://forge.example/acme/app

or, for a machine that only ever builds one project, a persistent working copy that fetches rather than clones. Either turns a full clone into an incremental one, which is the difference between minutes and seconds on a repository with real history.

Ask for less. Most jobs do not need the whole history:

git clone --depth 1 --single-branch --branch main https://forge.example/acme/app

A shallow single-branch clone is usually a tenth of the objects. It is not free for the server - a shallow clone makes upload-pack compute a custom pack - so it is third rather than first, but it beats a full clone comfortably.

Archives, when history is not wanted at all. A job that needs a tree and not a repository can take /{owner}/{repository}/archive/{ref}.tar.gz, which this instance caches per commit in its blob store. The second runner asking for the same commit is served from storage without git running at all.

What not to do: disabling gc on the runner's caches, or pointing many runners at one shared filesystem checkout. The first grows unbounded; the second turns a clone storm into a lock convoy, which is slower than the storm.

Recovering temporary workspaces

The runner journals temporary workspaces outside the checkout, under ~/.local/share/reviewos/workspace-journals. The namespace binds the controller URL and registered runner ID, so rotating a credential does not strand its journal. A completed live job disposes its own workspace only after stopping its tracked processes and services. A failed shutdown preserves the data.

After a restart, recovery asks the authenticated controller which journaled jobs this runner actually claimed and which are now terminal. Claim history survives reassignment. The storage layer then checks directory identity, ownership, mounts and process references before removing anything. Active jobs, unknown directories, changed identities and unavailable inspection permissions are kept. An explicitly retained REVIEWOS_WORKSPACE is outside temporary recovery.

For operator inspection, use the standalone binary's read-only preview. Set REVIEWOS_RUNNER_TOKEN through the runner's established secret configuration; the maintenance command uses that registered credential and never registers a new runner. It verifies the runner identity even when no journal exists.

reviewos-runner --clean-workspaces --url https://forge.example \
  --runner-id 1 --journal-root /home/runner/.local/share/reviewos/workspace-journals

Add --apply to retire only the verified candidates. Buddy exposes the same operation as buddy runner:workspaces:clean with the same options. A preview does not create a missing journal directory.

On Linux, protected same-user processes can make a runner unable to prove that a recovered workspace is unused. The preview reports preserved candidates in that case. An operator already running as root can inspect the existing owner's journal with --owner-uid 1000 and the explicit journal root; it cannot create workspaces for that user. If the runner used a different temporary directory (including macOS per-user temporary roots), also name that original root with --temporary-root. Do not change process inspection permissions, disable the guard or delete directories by a filename pattern. Old workspaces without a journal require an explicit forensic ownership review.