Remote Runners

traceway-runner executes browser monitors for a Traceway instance running with SYNTHETICS_BROWSER_MODE=remote. The server only queues browser runs; runners claim them over an outbound long-poll, execute the Playwright spec in headless Chromium, and post the result back. HTTP and TCP monitors never need a runner, they run inside every image and every plain-binary deployment.

Use a runner when you want browser checks but:

• your Traceway deployment is a plain binary or a non-browser image and you would rather not put Node and Chromium on that host,
• you do not want user-authored check scripts executing on the box that holds your data,
• you want to probe from a different network vantage point, or scale browser capacity horizontally.

If you run a single host and are fine with checks executing there, embedded mode is simpler: no secret, no extra process.

How runners authenticate

Runners are deployment infrastructure, not dashboard entities. There is nothing to create in the UI and there are no per-runner tokens:

• The server holds one shared credential, SYNTHETICS_RUNNER_SECRET. Every runner presents it as an Authorization: Bearer header on POST /api/runners/poll and POST /api/runners/results/:runId. Treat it like any other deploy secret.
• A runner self-registers a liveness row under its TRACEWAY_RUNNER_NAME (defaults to the hostname) on first poll. The name is also its claim identity, so give each runner a unique name.
• Traffic is outbound only. The runner long-polls the server (each poll is held up to 25 seconds), so it works behind NAT with no inbound ports and no firewall changes.
• Rotation: change the secret on the server, restart it, update the runners, restart them. A runner presenting a stale secret exits immediately with a clear error.
• The runner receives only the check script and the env vars configured on that check. Scripts run with an allowlisted environment, so they never inherit the server's env or the runner's own. Everything else about what a script can reach depends on the sandbox below.
• If a runner dies mid-run, the server reclaims the stale run after 5 minutes. An expired run is recorded as missed, never executed late.

1. Enable remote mode on the server

Add to the Traceway backend's environment and restart it:

SYNTHETICS_BROWSER_MODE=remote
SYNTHETICS_RUNNER_SECRET=$(openssl rand -hex 32)   # generate once, keep it

The backend refuses to start in remote mode without the secret. Two behaviors worth knowing: creating a browser monitor is rejected with a pointer here while the mode is off, and the poll endpoint answers 403 unless the mode is actually remote, so a runner pointed at a misconfigured server logs explicit errors instead of idling silently.

2. Prepare the runner host

Any Linux, macOS, or Windows host with outbound HTTPS works. A small VM (2 vCPU, 2 to 4GB RAM) comfortably runs the default 2 workers; each concurrent Chromium costs roughly 300 to 500MB.

Get the binary. Each backend release attaches prebuilt binaries (traceway-runner_linux_amd64, traceway-runner_linux_arm64, and darwin/windows variants) to the GitHub release. Or build it yourself, it is CGO-free:

cd backend && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o traceway-runner ./cmd/traceway-runner

Set up the Playwright harness. The runner never uses npm at runtime; it spawns node against a directory that has @playwright/test installed. Pinning the browser download inside the harness keeps the service independent of $HOME:

sudo mkdir -p /opt/traceway-runner && cd /opt/traceway-runner
npm init -y && npm i @playwright/test
PLAYWRIGHT_BROWSERS_PATH=/opt/traceway-runner/browsers npx playwright install --with-deps chromium

--with-deps installs Chromium's system libraries on apt-based distros (Ubuntu, Debian). On Amazon Linux or other rpm-based distros, run npx playwright install chromium and install the shared libraries with your package manager instead.

Install bubblewrap. Check scripts are code someone else wrote, and the env allowlist alone does not contain them. Without a sandbox, node runs as the same user as the runner, which means a script can read the runner's own /proc/<pid>/environ and recover the shared secret, overwrite @playwright/test in the harness to backdoor every later run, and read the run directory of a check executing beside it. On a fleet serving more than one team, all three are cross-tenant leaks.

sudo apt install bubblewrap     # or: sudo dnf install bubblewrap

Then set SYNTHETICS_BROWSER_SANDBOX=bwrap. Each run gets its own pid, ipc, uts and user namespace with a fresh /proc, so the runner's process table and environment are not merely unreadable, they are absent. The harness is bind-mounted read-only and only that run's directory is writable, so poisoning fails with EROFS. The other runs' scratch directories are masked behind an empty overlay, so a run cannot read what a check working beside it downloaded or captured. Nothing under /etc is mounted except DNS, CA certificates and locale data, so a secret in the unit file or an EnvironmentFile is out of reach. The network namespace is deliberately shared, since probes exist to reach the internet.

The default is auto, which uses bubblewrap when the host can run it and falls back to unconfined with a startup warning. Set bwrap explicitly in production: a host whose kernel forbids unprivileged user namespaces then fails at startup instead of quietly running scripts unconfined. The runner probes the sandbox at boot by executing node --version through the real bind set, so a broken host is caught before the first check.

3. Run it as a service

Run the binary under a dedicated non-root user: it executes user-authored scripts. A systemd unit:

[Unit]
Description=Traceway synthetic runner
After=network-online.target
Wants=network-online.target
 
[Service]
User=traceway-runner
WorkingDirectory=/opt/traceway-runner
Environment=TRACEWAY_URL=https://traceway.example.com
Environment=TRACEWAY_RUNNER_SECRET=<the server's SYNTHETICS_RUNNER_SECRET>
Environment=TRACEWAY_RUNNER_NAME=prod-runner-1
Environment=RUNNER_WORKERS=2
Environment=SYNTHETICS_PLAYWRIGHT_DIR=/opt/traceway-runner
Environment=PLAYWRIGHT_BROWSERS_PATH=/opt/traceway-runner/browsers
Environment=SYNTHETICS_BROWSER_SANDBOX=bwrap
Restart=always
RestartSec=5
 
[Install]
WantedBy=multi-user.target

sudo systemctl enable --now traceway-runner

Shutdown is graceful: on SIGTERM the runner stops polling, lets in-flight runs finish or be reclaimed, and exits.

4. Verify

• The runner log prints polling https://... (workers: 2, harness: ...) and then run N (check M): executing once browser runs are due.
• GET /api/health/deep (with your HEALTH_DEEP_TOKEN) shows synthetics.runnersOnline: 1 and, over time, executedTotal climbing with missedTotal flat.
• In the dashboard, run history rows for browser monitors show via <runner name>, and failed runs carry their screenshot and logs links.

Scaling and operations

• More capacity is just more runners, or a higher RUNNER_WORKERS. Work distributes itself: an idle runner claims the next queued run, a busy one is not asking.
• Upgrades: replace the binary and restart. The version is reported per runner and visible in its liveness row.
• Same host as the server? It works (point TRACEWAY_URL at http://127.0.0.1:8082), but on a single host embedded mode does the same job with fewer moving parts. The runner earns its keep once scripts should live off the server box.