Capacity & Self-Monitoring

Traceway's backend can monitor itself. When enabled, the server continuously reports ingestion pressure, ClickHouse health, cache sizes, and process resource usage into a regular Traceway project, so you can dashboard them, and more importantly, alert on them before you hit a wall.

This page lists every self-monitoring metric, recommended alert thresholds, and a decision tree for the three scaling questions that matter in production: do I scale ClickHouse, do I scale the backend server, or do I put Kafka in front of ingestion?

Enabling self-monitoring

1. Create a project in your Traceway dashboard (a dedicated "monitoring" project keeps things tidy). For larger setups, point self-monitoring at a separate Traceway instance so an outage doesn't take down its own monitoring.
2. Copy the project token and set:

MONITORING_TRACEWAY_URL="<project_token>@https://your-traceway-host/api/report"

That single variable turns everything on:

• Every HTTP request to the backend is captured as a transaction (so endpoint latency/error-rate alert rules work on POST /api/otel/v1/traces out of the box).
• Runtime metrics are reported every 30s: cpu.used_pcnt, mem.used, mem.total, go.go_routines, go.heap_objects, go.num_gc, go.gc_pause.
• The backend reporters described below start emitting traceway.* metrics.

Note: mem.used is Go heap memory only. The real OOM-relevant number is traceway.proc.rss_mb (resident set size, Linux only).

Metric reference

Ingestion (every batch)

Emitted once per ingest request, tagged with signal and table:

Backend process (every 30s)

ClickHouse health (every 60s, ClickHouse deployments only)

Not emitted in SQLite mode.

Session recording uploader (every 10s)

Source map cache (every 30s)

Source map cache sizing

Symbolication resolves stack frames against object storage at sourcemaps/{projectId}/{filename}, with one in-memory LRU of built resolvers in front of it. A resolver keeps only a flat token table plus interned file and function names; the raw .map and bundle bytes are held only while building. Concurrent requests for the same map share one storage read and one build, so a given map exists in memory at most once. The cache cap is on the estimated in-memory size, not the raw file size.

Lookups that find no map (or fail to load one) are negative-cached per filename with an escalating cooldown: a missing map starts at 1 minute, a transient failure at 15 seconds, and each consecutive failure doubles the cooldown up to a 15 minute cap. Uploading a file clears its cooldown and evicts any cached resolver on the instance that handled the upload; in multi-instance deployments the other instances retry within their remaining cooldown. This keeps JS projects that never upload maps from generating a steady stream of storage reads, and keeps a storage outage from adding read-timeout latency to every exception report.

Upgrading from 1.7? The flat storage layout is new in 1.8 and existing maps are migrated automatically at startup; see Upgrading to 1.8.

In both modes, a memory miss first tries a precompiled .tw resolver from object storage before re-parsing the map and bundle, and rebuilds happen once cluster-wide because the resulting .tw is written back to storage. Optionally, SOURCEMAP_CACHE_TYPE=disk adds a local disk tier of those .tw files between the memory cache and object storage, so a miss becomes a local mmap open instead of a storage download. See the .tw format and cache tiers for the mechanics.

Sizing guidance:

• Small box (1-2 GB RAM): SOURCEMAP_CACHE_MAX_BYTES_MB=100.
• Plenty of RAM, many active source maps: raise SOURCEMAP_CACHE_MAX_BYTES_MB until traceway.sourcemap.misses.delta stays near zero in steady state; every miss is a storage read plus a rebuild, which burns CPU and briefly doubles that map's memory footprint.
• More active maps than the memory cap can hold, or multiple backend instances: enable the disk tier. Watch traceway.sourcemap.builds.delta (full rebuilds) shrink toward zero while disk.hits.delta and store_hits.delta absorb the misses; give SOURCEMAP_DISK_CACHE_MAX_MB enough room that disk.evictions.delta stays near zero.
• Watch traceway.proc.rss_mb against your host RAM; if it climbs during exception ingest with stack traces, the source map cache is the usual suspect. With the disk tier, .tw token tables are memory-mapped and show up as page cache rather than process heap.

Alerting on these metrics

Create rules in the monitoring project under Notifications using the Metric threshold rule type. A rule is defined by: metric name, aggregation (avg, max, min, sum, p95, p99), operator (gt, gte, lt, lte, eq), threshold value, and lookback window in minutes. Rules are evaluated every 60 seconds.

Tag caveat: metric threshold rules aggregate across all tag combinations of a metric name. That is what you want for the metrics on this page: max on traceway.ch.parts.max_per_partition gives you the worst partition across every table, and max on traceway.ch.disk.used_pcnt gives you the fullest disk. Just avoid avg on per-table/per-disk metrics, where mixing series would dilute the signal.

Two more rule types complete the picture:

• Endpoint P95 threshold on POST /api/otel/v1/traces: client-visible ingest latency (the backend's own HTTP requests are transactions in the monitoring project).
• No data with data type metrics, which fires when the backend stops self-reporting entirely (crash, network partition).

Recommended starting alerts

Plus the two non-metric rules: Endpoint P95 threshold on POST /api/otel/v1/traces (e.g. 1000 ms over 10 minutes) and No data on metrics with 10 minutes of silence.

Scale ClickHouse, scale the backend, or add Kafka?

Ingestion in Traceway is synchronous: each ingest request converts the payload and writes it to ClickHouse before responding. That makes the diagnosis straightforward: the metrics tell you which side of that pipeline is saturated, and whether the problem is sustained or bursty.

Scale ClickHouse when the pressure is sustained and CH-side

ClickHouse can't merge or store as fast as you're writing. Signals, in rough order of severity:

• traceway.ch.parts.max_per_partition climbing steadily, traceway.ch.merges.running persistently high
• traceway.ch.delayed_inserts.delta > 0 across multiple intervals (then rejected_inserts if ignored)
• traceway.ch.disk.used_pcnt rising, traceway.ch.memory.tracking_bytes near the host's RAM
• traceway.ingest.batch.insert_ms p95 high while the ClickHouse host's CPU is also high

Fix: more CPU/RAM/disk for ClickHouse, faster disks, or sharding. Kafka won't help here: a buffer in front of a database that's saturated on average just delays the inevitable.

Scale the backend server when the Go process is the bottleneck

• cpu.used_pcnt avg > 80% together with high traceway.ingest.batch.convert_ms; payload conversion (JSON/OTLP decoding, hashing) is CPU work that scales with traffic
• traceway.proc.rss_mb approaching host RAM: before resizing, tune the source map cache variables above; they're the largest tunable consumer
• go.go_routines growing without bound

Fix: a bigger box, or multiple backend instances behind a load balancer (the backend is stateless apart from in-memory caches).

Add Kafka when the load is bursty and ClickHouse averages look healthy

Because inserts happen inside the request, every ClickHouse hiccup or burst becomes client-visible ingest latency. The tell-tale combination:

• traceway.ingest.batch.insert_ms p95 spiking while ClickHouse CPU, disk, and merges look calm on average
• traceway.ch.pool.open pinned at 15 (pool saturated) during peaks
• traceway.recordings.dropped.delta > 0; the one buffered path is already overflowing during bursts
• P95 on POST /api/otel/v1/traces rising even though total traceway.ch.inserted_rows.delta is well within what ClickHouse handles at steady state

That pattern means ClickHouse can absorb your average write rate but not your peaks, and the synchronous path turns peaks into client errors and data loss. A queue (Kafka) between the HTTP handlers and ClickHouse decouples client latency from insert latency, absorbs spikes, and enables larger, better-shaped batches.

Rule of thumb