OpenTelemetry Trace Format

SkyWalking receives traces in OTLP format through the otlp-traces handler of the OpenTelemetry receiver. The handler stores them in one of two ways, selected by receiver-otel.default.otlpTraceStorage:

Span listeners such as the GenAI analyzer and AI evaluation run in both modes. In otlp mode they read the OTLP span directly instead of its Zipkin conversion.

Supported Protocols

Both OTLP/gRPC and OTLP/HTTP are supported for traces, logs, and metrics:

OTLP/HTTP supports both application/x-protobuf and application/json content types.

Zipkin conversion mode

1. Make sure to enable the otlp-traces handler in the OTLP receiver of application.yml.

receiver-otel:
  selector: default
  default:
    enabledHandlers: otlp-traces
    otlpTraceStorage: zipkin

2. Enable the Zipkin receiver and the Zipkin query in application.yml, they hold the storage and the query of the converted spans. The OAP refuses to start when otlp-traces runs in this mode with receiver-zipkin disabled.

3. Read deploy Lens UI documentation to query the traces, or enable the TraceQL /zipkin datasource for Grafana, see TraceQL Service.

The conversion is lossy, because the Zipkin v2 model is smaller than OTLP. What the Zipkin copy cannot keep:

• Span kind. Zipkin has CLIENT, SERVER, PRODUCER and CONSUMER only. An OTLP SPAN_KIND_INTERNAL span is stored without a kind and reads back as SPAN_KIND_UNSPECIFIED; the receiver adds a span.kind=internal tag so the information is searchable as an ordinary tag, but the TraceQL kind intrinsic cannot match it.
• Attribute types. Every attribute becomes a string tag, so an integer such as http.response.status_code reads back as "200".
• Events and links. An event becomes a Zipkin annotation; when it carries attributes they are flattened into the annotation value as name|{attributes}|droppedCount. A link becomes an otlp.link.<n> tag holding the trace and span ids in decimal, the trace state and the flattened attributes.
• Resource and instrumentation scope. Resource attributes are merged into the span tags, and the scope name and version are kept as otel.library.name and otel.library.version tags.

Choose the native OTLP mode when any of these matters.

Native OTLP mode

receiver-otel:
  selector: default
  default:
    enabledHandlers: otlp-traces
    otlpTraceStorage: otlp
    otlpTraceSearchableTags: resource.deployment.environment.name,resource.deployment.environment,resource.service.namespace,resource.service.version,resource.k8s.namespace.name,resource.k8s.cluster.name,span.http.request.method,span.http.response.status_code,span.url.path,span.rpc.method,span.rpc.grpc.status_code,span.db.system.name,span.db.namespace,span.messaging.system,span.messaging.destination.name,span.peer.service
    otlpTraceSampleRate: 10000
    otlpTraceMaxSpansPerSecond: 0

traceQL:
  selector: default
  default:
    enableDatasourceOTLP: true
    restContextPathOTLP: /otlp
    otlpTracesListResultTags: http.request.method,http.response.status_code

The Zipkin receiver and query modules are not needed in this mode.

Service name

The service name is taken from the resource attributes in this order: service.name, faas.name, k8s.deployment.name, process.executable.name. The attributes themselves are not rewritten. A span whose resource carries none of these keys is rejected and reported in the OTLP partial_success response, it is never stored as an anonymous span.

Service group

otlpServiceGroupKeys (SW_OTEL_RECEIVER_OTLP_SERVICE_GROUP) groups OTLP services in OAP’s group::name convention, the one agent:: names follow, from a resource attribute. It lists attribute keys, comma separated, in order of preference; the first key a resource carries a value for is the group, and the keys after it are not read:

receiver-otel:
  default:
    otlpServiceGroupKeys: service.namespace,k8s.namespace.name

The grouped name is the service everywhere OAP names one: the trace analysis, its metrics, instances, endpoints and topology; the stored span, in both storage modes, so a TraceQL search names the service as the layer does, {resource.service.name="opentelemetry-demo::cart"}; and the OTLP logs, which are filed under it. The resource attributes are stored as sent, service.name and the grouping attribute among them, so {resource.service.namespace="opentelemetry-demo"} still finds the group’s traces; only service.layer is stored as the registered layer it names, UNDEFINED otherwise, as the service catalog files it. A query names each span’s service by the grouped name: a search lists it as the span’s service.name, and a trace fetched by id carries it as its resource’s service.name, the other resource attributes as sent. In the zipkin storage mode the stored service is lower case, since Zipkin lowercases every service name, so a group value with capitals, service.namespace=Shop, gives Shop::cart in the analysis and the logs but shop::cart in the stored span and the query; Kubernetes namespace names are always lower case.

A caller that dials the host cart still reaches the grouped cart: the trace analyzer learns a network address alias from the two spans of a call, keyed by the caller and the address it dialed, so shop::frontend dialing cart:7070 reaches shop::cart and outlet::frontend dialing the same cart:7070 reaches outlet::cart, see Address-named peers. No host is matched against service names or Kubernetes DNS, so the group needs no particular relation to the namespace for this, and a callee is named only once one of its caller’s calls has been paired; the calls before then are drawn to the address-named node, which the topology query folds onto the service once the row lands. To pin which TAL rules analyze the spans, set their resource’s service.layer to a registered layer name and declare TAL rules for that layer. Only those rules run. An unknown layer name is logged and its spans are not analyzed. See Layer. The group has no cluster part either: the namespace shop of two Kubernetes clusters groups both clusters’ cart as one shop::cart.

Pick an attribute that holds one low-cardinality value per service, such as service.namespace, deployment.environment.name or k8s.namespace.name; a value that changes per pod or per request splits a service into many, and a service whose spans carry the attribute only some of the time is split into its grouped and its plain name. A service.name the SDK grouped already, shop::cart, is kept, and a value that holds :: itself groups nothing. Empty, the default, groups nothing.

What is stored

Every span is stored as a single-span ResourceSpans message, so the query returns the exact resource, instrumentation scope, attributes, events, links and status the SDK sent, except that each resource’s service.name is the service the span was stored under: grouped, and through the naming rules below. A few columns are extracted for search: service name and instance (service.instance.id), instrumentation scope name, span name, kind, status code, peer.service, start time and duration. The service, span and peer names go through the core module’s naming rules first, the length limits and the endpoint grouping rules SkyWalking applies to its own services and endpoints, so a TraceQL name matches the grouped name while the stored span keeps the original. Every resource attribute is indexed as a resource.<key>=<value> tag and every span attribute as a span.<key>=<value> tag, so a scoped TraceQL attribute matches its own scope only and an unscoped .<key> matches either.

The BanyanDB storage keeps the spans in the otlpTrace group, see BanyanDB TTL for its default retention and BanyanDB storage for the group settings. The JDBC storages add an otlp_span_tag table for the tag index.

Query

Point a Grafana Tempo datasource at http://<oap-host>:3200/otlp, see Use Grafana As The UI. The supported TraceQL subset and the HTTP API are described in TraceQL Service.

The status module exposes the same queries with the execution trace for debugging under /debugging/query/otlp/api/search and /debugging/query/otlp/api/v2/trace, see Status APIs.

Analysis

Every accepted span also goes through the Trace Analysis Language rules of the trace-analyzer module, in both storage modes and before the head sampling and the rate limit above. The shipped rule file analyzes the spans whose resource carries service.layer=OTLP_GENERAL, and registers their services under that layer with their metrics, topology and slow statements; every other span is only stored. See OpenTelemetry Trace Analysis for how an application enables it, the settings, and what it produces.