> For the complete documentation index, see [llms.txt](https://docs.controltheory.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/traces/sending-traces.md).

# Sending Traces

Send OpenTelemetry traces to Dstl8 through an OpenTelemetry source or the Docker agent, name your services, and connect logs to spans

Dstl8 takes traces over OTLP, the OpenTelemetry protocol. Any OpenTelemetry SDK, auto-instrumentation agent or Collector can send them. This page covers where to send, which settings matter, and what to check when traces don't show up.

## Choose a path

| Your setup                                                                               | Send traces to                                   |
| ---------------------------------------------------------------------------------------- | ------------------------------------------------ |
| Apps or a Collector that can reach the internet (cloud VMs, Kubernetes, ECS, serverless) | An [OpenTelemetry source](#opentelemetry-source) |
| Apps on a Docker host running the Dstl8 agent                                            | The [Docker agent](#docker-agent) on that host   |

On Kubernetes, send traces to an OpenTelemetry source; the Kubernetes agent doesn't take them (see [How traces reach Dstl8](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/traces.md#how-traces-reach-dstl8)). If you already run an OpenTelemetry Collector in the cluster, add the source as an exporter on its `traces` pipeline (see the [Collector example](#collector)).

## OpenTelemetry source

An OpenTelemetry source takes logs, metrics and traces on one endpoint. Create one as described in [OpenTelemetry (OTel)](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/opentelemetry-otel.md). It gives you:

* An **OTLP/HTTP Endpoint**, such as `https://<org_id>.app.dstl8.ai/src-<id>`. Traces go to `/v1/traces` under it.
* A **Bearer Token**, sent in the `Authorization` header of every request.

The source speaks OTLP over HTTP (not gRPC). Set your exporter's protocol to `http/protobuf`.

### SDK environment variables

Most OpenTelemetry SDKs and auto-instrumentation agents read the standard environment variables:

```bash
OTEL_SERVICE_NAME=checkout
OTEL_RESOURCE_ATTRIBUTES=service.instance.id=checkout-7f9c,deployment.environment=prod
OTEL_EXPORTER_OTLP_ENDPOINT=https://<org_id>.app.dstl8.ai/src-<id>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<your-token>
```

* Set the base endpoint, without `/v1/traces`: the SDK adds the path for each signal. If you set only the traces endpoint (`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`), give the full URL ending in `/v1/traces`.
* Write the space in `Bearer <token>` as `%20`. Some SDKs reject a raw space in this variable.
* Set the protocol along with the endpoint. SDKs differ in their default, and an exporter speaking gRPC to an HTTP endpoint fails.

### Collector

If you run an OpenTelemetry Collector, add Dstl8 as an exporter. One `otlphttp` exporter serves all three signals:

```yaml
exporters:
  otlphttp/dstl8:
    endpoint: https://<org_id>.app.dstl8.ai/src-<id>
    headers:
      Authorization: "Bearer <your-token>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/dstl8]   # keep your existing exporters here too
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/dstl8]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/dstl8]
```

Keep the token out of the file in production: most Collector distributions accept `${env:DSTL8_TOKEN}` in its place.

## Docker agent

The Dstl8 agent on a Docker host listens for OTLP from applications on that host:

| Port   | Protocol  |
| ------ | --------- |
| `4317` | OTLP/gRPC |
| `4318` | OTLP/HTTP |

Point the SDK at the host, with no token:

```bash
OTEL_SERVICE_NAME=checkout
OTEL_EXPORTER_OTLP_ENDPOINT=http://<host-address>:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

From inside a container, `localhost` is the container itself: use the host's IP address, the Docker bridge gateway (usually `172.17.0.1`) or `host.docker.internal`. See [Docker](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/docker.md#send-your-own-telemetry-to-the-agent) for the details, and keep these ports off the public internet; the listeners are unauthenticated.

## Name your services

Dstl8 groups spans by `service.name`. It's the name you see on the service map, in the services table and on service pages, and the unit you assign to a workspace.

* **Set `service.name` on every service**, with `OTEL_SERVICE_NAME` or in your SDK's resource. Without it, most SDKs send `unknown_service` (or `unknown_service:<process>`), and every unnamed app lands in one bucket.
* **Use the same name everywhere.** The name on your traces, logs and metrics should match, so a service's page shows all three. `checkout` and `checkout-svc` are two services to Dstl8.
* **Set `service.instance.id` per replica** (pod name, hostname or container ID). On an OpenTelemetry source it names each instance's log stream, so you can tell replicas apart.
* **Add `deployment.environment`** (for example `prod`, `staging`) when one source receives more than one environment. Better still, use a separate OpenTelemetry source per environment.
* **On Amazon ECS with the Docker agent,** a span with no `service.name` is named after its task definition's family, so set `service.name` if you want a different name.

## Sampling

Send every span if you can. Dstl8 works out requests, errors, latency and the service map from every span it receives, and then keeps the traces worth opening for you (see [What Dstl8 keeps](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/traces.md#what-dstl8-keeps)).

If you sample in your SDK, Dstl8's request rates and error counts reflect only what you send, and a rare failure may never arrive. If you must sample:

* **Sample by trace ID,** so every service makes the same keep-or-drop decision and traces stay whole. With the standard variables: `OTEL_TRACES_SAMPLER=parentbased_traceidratio` and `OTEL_TRACES_SAMPLER_ARG=0.25` (keeps 25%).
* **Don't sample spans independently** in each service: you'll get traces with holes in them.

## Health checks

Dstl8 drops health-check and probe spans as they arrive: requests to `/healthz`, `/readyz`, `/livez`, `/health`, `/ready` and `/metrics`, and requests from Kubernetes probes (`kube-probe/`). They would otherwise swamp request rates and latency with traffic that tells you nothing.

Routes match exactly, so `/api/health` or `/healthz/deep` is kept.

## Connecting logs to spans

When you click a span in the waterfall, Dstl8 shows the log lines written while it ran. Lines that carry the span's trace context belong to that exact request: they are marked **●** and listed first. Other lines from the same pod or service during the span are shown too, but may belong to other requests.

To get your log lines matched, send them with OpenTelemetry trace context. Most OpenTelemetry SDKs and log bridges attach the active trace and span to every log record they emit, so logs sent through OpenTelemetry alongside your traces are connected without extra work. Check that your logging library is wired to OpenTelemetry's logs bridge (or the OpenTelemetry agent for your language), and that your logs are sent to Dstl8 too.

## Troubleshooting

| Symptom                                                                     | What to check                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **No traces showing**                                                       | Check the exporter is sending to `/v1/traces` under the source's endpoint (or the base URL with an `otlphttp` exporter), with `Authorization: Bearer <token>`. Check the protocol is `http/protobuf` for an OpenTelemetry source. Then check the service is assigned to the workspace you're looking at (see [Which workspace sees which traces](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/traces.md#which-workspace-sees-which-traces)), and that the workspace takes **Traces** in **Signals this workspace gets** |
| **Traces from Kubernetes pods never arrive**                                | The Kubernetes agent doesn't take traces. Send them to an OpenTelemetry source instead                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| **Docker: no traces from the host**                                         | Check the app reaches `<host-address>:4318` (HTTP) or `:4317` (gRPC) with the matching protocol, and not `localhost` from inside a container. Traces need agent v1.4.0 or later (see [Agent versions](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/collecting.md#agent-versions)): update it as described in [Docker](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/docker.md#update-the-agent)                                                                                 |
| **A service shows as `unknown_service…`**                                   | The SDK has no service name. Set `OTEL_SERVICE_NAME`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| **The same service shows twice**                                            | Its spans carry two different `service.name` values, for example `checkout` and `checkout-svc`. Use one name everywhere                                                                                                                                                                                                                                                                                                                                                                                                                          |
| **Traces show in one workspace but not another**                            | Traces are scoped by service. Assign the service (or the whole source with **All streams**) to the other workspace too                                                                                                                                                                                                                                                                                                                                                                                                                           |
| **"Your plan does not include traces."**                                    | Traces is not part of your current plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| **Most traces are missing**                                                 | Dstl8 keeps the traces worth opening, not every trace (see [What Dstl8 keeps](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/traces.md#what-dstl8-keeps)). Use the **Errors** tile to find failing ones, and the top of the scatter for slow ones. If you also sample in your SDK, see [Sampling](#sampling)                                                                                                                                                                                                              |
| **Traces have missing spans**                                               | Each service is sampling on its own. Sample by trace ID, or not at all                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| **Some services' spans never appear, though the exporter reports no error** | Spans need a start time, and spans with timestamps far in the future or far in the past are dropped. Check the clock on the host that sends them                                                                                                                                                                                                                                                                                                                                                                                                 |
| **A health-check endpoint is missing from the services table**              | Health-check and probe spans are dropped by design (see [Health checks](#health-checks))                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| **Span log lines are never marked ●**                                       | The log lines don't carry trace context. Send them through OpenTelemetry's logs bridge; see [Connecting logs to spans](#connecting-logs-to-spans)                                                                                                                                                                                                                                                                                                                                                                                                |


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.controltheory.com/controltheory-documentation/dstl8-docs/traces/sending-traces.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
