Observability

Send OpenTelemetry traces from your own services into 2kw.ai and see every extraction, chat completion, and downstream span in one viewer.

Overview

2kw.ai records traces from two sources and stores them in a single span table so the trace viewer, analytics dashboard, and experiment scoring all read from the same place:

• Self-instrumentation — Every extraction, chat completion, conversion, and transcription handled by 2kw.ai already emits OpenTelemetry spans. Nothing to configure; they show up in the viewer automatically.
• External OTLP ingestion — Your own services (retrieval workers, orchestration glue, customer apps calling the Gateway) can POST standard OTLP to 2kw.ai's /api/v1/traces endpoint. Their spans land in the same trace id as the platform-side spans when you propagate traceparent.

That second path is what this page is about. If you only care about the platform-side spans, there's nothing to do — open Traces in the sidebar.

Quickstart

Send your first span with curl in under a minute. The request body is an OTLP/HTTP JSON payload with one ResourceSpans + one ScopeSpans + one Span. Replace sk_… with an API key from Keys.

curl -X POST https://api.2kw.ai/v1/traces \
  -H "Authorization: Bearer sk_…" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {
        "attributes": [
          { "key": "service.name", "value": { "stringValue": "my-service" } }
        ]
      },
      "scopeSpans": [{
        "scope": { "name": "manual-test" },
        "spans": [{
          "traceId": "5b8efff798038103d269b633813fc60c",
          "spanId": "eee19b7ec3c1b174",
          "name": "hello.world",
          "kind": 1,
          "startTimeUnixNano": "1776600000000000000",
          "endTimeUnixNano": "1776600001000000000",
          "status": { "code": 1 }
        }]
      }]
    }]
  }'

Open Traces in the sidebar — hello.world appears under the service name my-service.

The OTLP endpoint

POST /api/v1/traces

Accepts OTLP over HTTP in the two encodings the OTel spec defines:

Authentication: API key via Authorization: Bearer sk_…. The key's organization becomes the tenant for every span in the request — see Tenant isolation below.

Response: 200 OK with an empty ExportTraceServiceResponse on success. Malformed payloads return 400 Bad Request and no spans from the request are persisted. 401 Unauthorized on a missing or invalid API key.

Tenant isolation

The tenant (organization_id) of every persisted span is taken from the authenticated API key — not from any attribute you send. Even if a span arrives with backbone.organization_id="some-other-org", it will be stored against your org. This is intentional: it means you can't accidentally (or deliberately) write into another tenant's trace store.

Limits

• Single-request payload: 4 MB.
• No hard span-count cap per request — the practical limit is the 4 MB payload ceiling.
• Retention is subject to your plan; contact support if you need a longer window than the default.

Client examples

All examples below assume OTEL_EXPORTER_OTLP_ENDPOINT=https://api.2kw.ai/v1/traces and OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer sk_…. Point your existing OTel SDK at those values and your spans flow into 2kw.ai.

Python

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
    OTLPSpanExporter,
)

provider = TracerProvider(
    resource=Resource.create({"service.name": "my-worker"}),
)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="https://api.2kw.ai/v1/traces",
            headers={"Authorization": "Bearer sk_…"},
        )
    )
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("ingest.file"):
    # your work here
    ...

Node / TypeScript

import { NodeSDK } from '@opentelemetry/sdk-node'
import { Resource } from '@opentelemetry/resources'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'my-worker',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'https://api.2kw.ai/v1/traces',
    headers: { Authorization: 'Bearer sk_…' },
  }),
})
sdk.start()

Java

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;

OpenTelemetrySdk sdk = OpenTelemetrySdk.builder()
    .setTracerProvider(
        SdkTracerProvider.builder()
            .setResource(Resource.create(
                Attributes.of(AttributeKey.stringKey("service.name"), "my-worker")))
            .addSpanProcessor(BatchSpanProcessor.builder(
                OtlpHttpSpanExporter.builder()
                    .setEndpoint("https://api.2kw.ai/v1/traces")
                    .addHeader("Authorization", "Bearer sk_…")
                    .build()
            ).build())
            .build())
    .buildAndRegisterGlobal();

Joining your spans to platform spans

When your service calls the 2kw.ai API, propagate the W3C traceparent header (the OTel SDK does this automatically if you instrument the HTTP client). 2kw.ai reads traceparent on every request, so the span it creates for the extraction / chat / conversion inherits your trace id and becomes a child of your calling span. One trace, your service + 2kw.ai, single view.

Attribute conventions

OTel attributes you send through are stored verbatim on the span and visible in the detail panel. A few keys do more than that:

service.name (resource attribute)

Displayed in the Traces table as the Services column. Use one per logical deployment — the platform's own spans use backbone.

session.id

Follows the OpenTelemetry GenAI semantic convention. 2kw.ai groups every span sharing the same session.id into a trace_session row, which powers the conversation-history views.

backbone.resource_type / backbone.resource_id

Optional. When set, these get promoted from JSONB attributes to dedicated columns on the span table and link the span to a 2kw.ai domain resource (e.g. an extraction id, dataset item id). Useful when you want traces filterable by "show me everything that happened for extraction X".

The backbone.op.* contract — analytics opt-in

2kw.ai's analytics dashboard reads one uniform set of attributes off the root span of every tracked operation. If you stamp these on your own root spans, your operations show up in the Activity / Spend / Health lenses alongside platform-native ones.

All five are optional. Stamp surface alone to get the operation counted; add tokens / cost / duration to show up in Spend and Health. Fewer than all five is fine.

PII and tracing settings

2kw.ai records spans by default, but prompt and completion content are not captured unless you opt in. The defaults lean conservative — raw LLM input/output is PII-sensitive and shouldn't land in the span store until an admin decides the tradeoff is worth it.

Two org-level toggles, both off by default:

• includePrompts — Retain prompt content (gen_ai.content.prompt, gen_ai.input.messages, llm.input_messages, input.value, and equivalents). Also required to persist the extraction / chat input that the platform-side spans stamp.
• includeCompletions — Same, for completion / response content.

Flip them via PUT /api/v1/tracing/settings (admin only) or the UI's Tracing Settings panel. When enabled, the span detail panel's Input / Output sections render the raw content. When disabled, those keys never reach the database — the panel's Input / Output sections are empty, and the rest of the span (timing, tokens, cost, metadata attributes) is unaffected.

Viewing traces

Open Traces from the sidebar. The list aggregates spans by trace_id into a summary row: root span name, worst-case status, duration, span count, services, start time.

• Search matches trace id, span name, source service, or the backbone.operation attribute (a low-cardinality label the platform stamps on root spans, e.g. extraction, chat.completions, conversion). Any span in the trace matching the query surfaces the whole trace.
• Filter by status code (OK / Error / Unset) from the toolbar.
• Click a row to open the detail aside: span tree on the left, selected span's Input / Output / Attributes on the right.
• Deep link — the selected trace id lives in the URL (?traceId=…). Share the link and a teammate lands on the same view.

Troubleshooting

"I sent spans but the viewer is empty"

• Check the API key — an unauthenticated request returns 401 and the span never persists.
• Check the trace date range filter in the viewer.
• Check that the traceparent you're sending is valid — malformed ids (not 32 hex chars for trace id, not 16 for span id) cause the span to be rejected at ingest.

"My span is in the viewer but the Input / Output section is empty"

Your org has includePrompts / includeCompletions off (the default). Prompt- and completion-content keys are stripped at ingest, so the detail panel has no content to render in those sections — only the other attributes remain. Flip the toggles in Tracing Settings if your org is OK with capturing raw content going forward. See PII and tracing settings.

"My custom spans don't appear in the analytics dashboard"

Analytics only aggregates spans carrying backbone.op.surface. See The backbone.op.* contract.

"I get 400 Bad Request with no details"

The OTLP parser accepted the HTTP request but rejected the payload. Common causes: unsupported content type, truncated protobuf, invalid hex ids. Switch the exporter to the JSON encoding (application/json) temporarily — error responses include more context.