Symbolicator
OpenTelemetry

OpenTelemetry

Traceway's symbolicator ships as a standalone OpenTelemetry Collector processor, source_map_symbolicator. Add it to your collector pipeline and minified JavaScript stack traces in spans, span events, and log records are rewritten to original files, lines, columns, and function names before they reach any backend, Traceway or otherwise.

It's a drop-in replacement for Honeycomb's source_map_symbolicator processor (opens in a new tab): same component type, same attribute contract, same store layout, same configuration keys. Existing pipelines and instrumentation (for example @honeycombio/opentelemetry-web with GlobalErrorsInstrumentation) work unchanged. What you gain by swapping:

  • A cache bounded by disk, not RAM. Maps and bundles compile once into the .tw resolver format and are memory-mapped from a local cache directory with an LRU byte cap. Resident memory tracks the hot set, restarts warm from disk.
  • Pure Go, no cgo. No glibc base image requirement; runs in scratch.
  • Function names via bundle scope analysis, the same approach Sentry's symbolic uses.

If you want to call the engine from your own Go code instead of a collector, see Library.

The same processor also auto-routes non-symbolic Dart AOT traces and stripped iOS/Swift traces to the Dart and iOS paths. The walkthrough below covers the JavaScript source-map workflow.

Building a collector with the processor

The processor is a public package of the published github.com/tracewayapp/traceway/backend module, exposing the standard NewFactory() entry point, so it plugs into the OpenTelemetry Collector Builder (opens in a new tab) like any other component. Because the package is not at the module root, the manifest entry carries an import line alongside gomod.

The repo's benchmarks/processor (opens in a new tab) is a complete, runnable example of all of this: a builder manifest, a collector config, and a run-local.sh that builds the collector and drives load through it. The steps below mirror it.

Install the builder

Pin the builder to the collector version your manifest targets (this example uses 0.154.0, the version the benchmark pins):

go install go.opentelemetry.io/collector/cmd/builder@v0.154.0

Write the builder manifest

manifest.yaml, the Traceway processor next to whatever receivers and exporters you need. This is benchmarks/processor/manifest-traceway.yaml without its local replaces directive:

dist:
  name: otelcol-symbolicator
  description: OTel Collector with the Traceway symbolicator
  output_path: ./build
 
receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.154.0
 
processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.154.0
  - gomod: github.com/tracewayapp/traceway/backend v1.8.0
    import: github.com/tracewayapp/traceway/backend/app/symbolicator/otelprocessor
 
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.154.0

The processor first ships in v1.8.0; use that tag or any newer one, and keep the collector component versions aligned as OCB manifests require. The benchmark manifest adds a replaces: github.com/tracewayapp/traceway/backend => ../../../backend line because it builds against the working tree; drop it when building against the published tag.

Build it

builder --config manifest.yaml

This produces ./build/otelcol-symbolicator, a self-contained collector binary. With the default goja parser it's pure Go and needs no cgo. For the faster oxc parser, follow benchmarks/processor/run-local.sh: run scripts/build-oxc-shim.sh, generate the sources with builder --config manifest.yaml --skip-compilation, then cd build && CGO_ENABLED=1 go build -tags oxc ..

Configure the pipeline

config.yaml. The benchmark's env-parameterized version is benchmarks/processor/config-traceway.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  source_map_symbolicator:
    source_map_store: file_store
    local_source_maps:
      path: /sourcemaps
    cache_dir: /var/cache/symbolicator
    cache_max_mb: 2048
  batch:
 
exporters:
  otlphttp:
    endpoint: https://otel.example.com
    headers:
      Authorization: Bearer <token>
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [source_map_symbolicator, batch]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [source_map_symbolicator, batch]
      exporters: [otlphttp]

To forward to Traceway, point the exporter at your Traceway server's OTLP base path and use a project token:

exporters:
  otlphttp:
    endpoint: https://traceway.example.com/api/otel
    headers:
      Authorization: Bearer <project token>

Run it

./build/otelcol-symbolicator --config config.yaml

For an end-to-end loop (build the collector, generate a corpus, drive load, sample memory), benchmarks/processor/run-local.sh runs the whole matrix.

Registering the factory programmatically

When you assemble a collector in Go instead of using the builder, register the factory like any other processor factory:

import "github.com/tracewayapp/traceway/backend/app/symbolicator/otelprocessor"
 
factory := otelprocessor.NewFactory()
factories.Processors[factory.Type()] = factory

The component type is source_map_symbolicator, with traces and logs support at alpha stability.

Populating the source map store

The store holds your build output as-is: minified bundles next to their .map files, addressed by URL basename. Upload (or mount) the files your bundler emitted; no manifest or renaming is needed.

  1. A frame URL like https://cdn.example.com/assets/app-3f9c.js:1:13337 is looked up in the store as app-3f9c.js.
  2. The bundle's //# sourceMappingURL= comment is followed to find the map (inline data: URIs are supported).
  3. Both compile into one resolver, cached thereafter.

Content-hashed filenames from different deploys coexist naturally. For stable filenames (app.js), either accept last-upload-wins or set the app.debug.source_map_uuid resource attribute in your web SDK; the processor then prefixes store lookups with that uuid, isolating each build under its own directory:

/sourcemaps/
  9c1d2b3a-.../(app.js, app.js.map)   # build A
  4e5f6a7b-.../(app.js, app.js.map)   # build B

Three store backends are built in:

# Local directory (default)
source_map_store: file_store
local_source_maps:
  path: /sourcemaps
 
# S3, credentials from the default AWS chain
source_map_store: s3_store
s3_source_maps:
  region: eu-central-1
  bucket: my-sourcemaps
  prefix: production
 
# GCS, credentials from Application Default Credentials
source_map_store: gcs_store
gcs_source_maps:
  bucket: my-sourcemaps
  prefix: production

What the processor reads and writes

For each span, span event, or log record carrying an exception:

AttributeDirectionMeaning
exception.stacktraceread + rewrittenRaw stack in, symbolicated stack out
exception.structured_stacktrace.{urls,functions,lines,columns}read + rewrittenStructured parallel arrays, preferred over string parsing when present
exception.type, exception.messagereadUsed for the rewritten header line
app.debug.source_map_uuidread (resource)Optional store key prefix per build
exception.stacktrace.original and .original array variantswrittenOriginals, kept when preserve_stack_trace: true
exception.symbolicator.failedwrittentrue when any frame failed
exception.symbolicator.errorwrittenFirst failure, with a count hint when several frames failed
exception.symbolicator.parsing_methodwrittenstructured_stacktrace_attributes or processor_parsed
traceway.processor_type, traceway.processor_versionwrittenProvenance stamps

Raw string parsing handles the stack formats browsers produce: V8 (including async, new, [as alias], and eval frames) and Firefox.

Configuration reference

KeyDefaultDescription
source_map_storefile_storefile_store, s3_store, or gcs_store
local_source_maps.path.Root directory for file_store
s3_source_maps.region / .bucket / .prefixS3 location
gcs_source_maps.bucket / .prefixGCS location
timeout5sPer-fetch budget for store reads
cache_dir""Directory for the .tw disk cache; empty disables the disk tier
cache_max_mb2048Byte cap for the disk cache, LRU-evicted
cache_max_disk_pct0Cap as a percentage of the cache directory's filesystem; when both caps are set, the smaller wins
source_map_cache_size128Max resolvers held in memory (cheap mmap handles when the disk tier is on)
preserve_stack_tracetrueKeep originals under the .original keys
build_uuid_attribute_keyapp.debug.source_map_uuidResource attribute used as a store key prefix
language_attribute_keytelemetry.sdk.languageAttribute checked against allowed_languages
allowed_languages[]When set, only records with a matching language are processed
dart_default_archarm64Default CPU architecture used when a Dart trace does not carry one
ios_default_archarm64Default CPU architecture used when an iOS trace does not carry one
parsergojaBundle parser for function names: goja or oxc (the latter only in -tags oxc builds)

Every attribute key from the table above is remappable (stack_trace_attribute_key, urls_attribute_key, symbolicator_failure_attribute_key, and so on), with the same configuration keys and defaults as Honeycomb's processor.

Failed fetches are negative-cached for one minute per bundle, so a missing upload cannot turn an error storm into a store-request storm.

Sizing the cache

  • cache_dir unset: every resolver is rebuilt from the store on demand and held only in the in-memory LRU (source_map_cache_size entries). Fine for small corpora.
  • cache_dir set: each map+bundle compiles to a .tw file once; subsequent loads are mmaps measured in microseconds. Budget with cache_max_mb, or cache_max_disk_pct: 50 to track the filesystem size. .tw files are typically a small multiple of the map's mappings field, far smaller than the parsed-in-RAM representation.

For the numbers behind that choice (throughput, memory, and what survives memory pressure), see Performance. For the full engine internals, see Architecture and JavaScript.