> For the complete documentation index, see [llms.txt](https://docs.controltheory.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/collecting.md).

# Collecting Metrics

Send metrics to an OpenTelemetry source or a Docker agent, or turn on Prometheus metrics, Remote Write and OTLP-from-pods on a Kubernetes source

Metrics reach Dstl8 five ways. Each is opt-in: nothing is collected until you send metrics or turn a path on.

| Path                                                                     | Use it when                                                             | Turn it on                                                     |
| ------------------------------------------------------------------------ | ----------------------------------------------------------------------- | -------------------------------------------------------------- |
| [OpenTelemetry source](#opentelemetry-source)                            | Your apps or an OpenTelemetry Collector can send OTLP over the internet | Point the exporter at the source's URL                         |
| [Docker](#docker)                                                        | Your apps run on a host with the Docker agent                           | Send OTLP to the agent on the host                             |
| [Kubernetes: Prometheus metrics](#kubernetes-collect-prometheus-metrics) | You want node, pod and container metrics from the cluster               | The **Collect Prometheus metrics** switch                      |
| [Kubernetes: Remote Write](#remote-write)                                | You already run Prometheus in the cluster                               | The same switch                                                |
| [Kubernetes: OTLP from pods](#kubernetes-otlp-from-pods)                 | Your pods already push OTLP to the agent on their node                  | `dstl8 sources metrics <source> --prom-metrics --otlp-metrics` |

Your plan must include Metrics: **Organization Settings** → **General** lists it under **Features**.

## Agent versions

Metrics and traces from an agent need a recent agent. An older agent keeps sending logs and reports no error, so check the version first when something is missing. Re-running the install command upgrades the agent to the latest version.

| Path                                            | Needs agent     |
| ----------------------------------------------- | --------------- |
| Docker: metrics                                 | v1.4.1 or later |
| Docker: traces                                  | v1.4.0 or later |
| Kubernetes: Prometheus metrics and Remote Write | v1.4.2 or later |
| Kubernetes: OTLP from pods                      | v1.4.4 or later |

An OpenTelemetry source has no agent, so there is nothing to upgrade.

## OpenTelemetry source

An OpenTelemetry source takes logs, metrics and traces on one endpoint. Create one as in [OpenTelemetry (OTel)](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/opentelemetry-otel.md), then send metrics to the same base URL: the exporter POSTs to `/v1/metrics` under it, with the source's bearer token in the `Authorization` header.

An OTel Collector `otlphttp` exporter pointed at the base URL sends all three signals; add it to your `metrics` pipeline:

```yaml
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlphttp]
```

Set `service.name` on the resource. Dstl8 scopes these metrics to workspaces by service; see [Which workspace sees which metrics](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics.md#which-workspace-sees-which-metrics).

## Docker

The Docker agent listens for OTLP on the host, on ports `4317` (gRPC) and `4318` (HTTP), and forwards metrics as well as logs and traces. Point your apps' OTLP exporter (or a Collector's `otlp` exporter) at the host on one of those ports; no token is needed, since the agent already belongs to its source. See [Docker](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/docker.md) for which address reaches the host from a container.

Like an OpenTelemetry source, Docker metrics are scoped to workspaces by `service.name`, so set it. Metrics from Docker need agent v1.4.1 or later (see [Agent versions](#agent-versions)).

## Kubernetes: Collect Prometheus metrics

On a Kubernetes source, the **Collect Prometheus metrics** switch turns on scraping and Remote Write together. It's on the source's configuration step when you create it, and in **Edit** on the source afterwards.

The switch changes what the agent may do in your cluster, so the install command changes with it: it carries `--prom-metrics`, which gives the agent read-only access to the kubelet's metrics and to the cluster's service endpoints (to find scrape targets), and opens the Remote Write port. **Re-run the install command after turning the switch on or off.** See [Kubernetes: Install options](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/kubernetes.md#install-options).

### What the agent scrapes

Each target is scraped every 60 seconds, and only the metrics listed below are kept: Dstl8 never ingests everything a target exposes.

| Target                     | Scraped by                                                                                                                    | Metrics kept                                                                                                                                                                                                                                                                                                                                    |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **kubelet**                | the agent on each node, its own node's kubelet                                                                                | running pods and containers; volume capacity, available, used and inodes                                                                                                                                                                                                                                                                        |
| **cAdvisor**               | the agent on each node, through its kubelet                                                                                   | container CPU usage and throttling, memory working set and RSS, OOM events, memory limits; pod network receive and transmit                                                                                                                                                                                                                     |
| **node-exporter**          | the agent on each node, the node-exporter pod on that node                                                                    | CPU seconds, memory available and total, filesystem available and size, load (1, 5, 15 min), network receive and transmit, disk I/O time                                                                                                                                                                                                        |
| **kube-state-metrics**     | the cluster agent, once per cluster                                                                                           | pod phase, restarts, waiting and last-terminated reasons, container requests and limits; Deployment, StatefulSet and DaemonSet replicas; node conditions; failed Jobs; HPA replicas                                                                                                                                                             |
| **GPUs and model servers** | the agent on each node, the NVIDIA dcgm-exporter, AMD device-metrics-exporter and vLLM pods on that node                      | GPU activity, memory, power, heat and errors; model-server requests, working memory and tokens. On whenever the switch is, including when you choose targets with `--prom-scrape`, until you turn it off; see [GPUs and Model Servers](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/gpus-and-model-servers.md) |
| **Annotated pods**         | the agent on each node (agent v1.4.4 or later; before that, the cluster agent), pods annotated `prometheus.io/scrape: "true"` | only the metric names you list under **Extra metric names → annotated\_pods**; see [Annotated pods](#annotated-pods)                                                                                                                                                                                                                            |

{% hint style="info" %}
The agent never installs node-exporter or kube-state-metrics. It scrapes the ones your cluster already runs: node-exporter pods labelled `app.kubernetes.io/name` = `node-exporter` or `prometheus-node-exporter` (port 9100), and a kube-state-metrics Service labelled `app.kubernetes.io/name=kube-state-metrics`.
{% endhint %}

To collect only some of these, choose them with the CLI: `dstl8 sources metrics <source> --prom-scrape kubelet,cadvisor` turns on those two, plus **GPUs and model servers**, which stays on until you add `no-gpu`. In the portal, clear the toggles you don't want under the source's **Metrics** settings. With the switch alone, every target above is on, including **Annotated pods**, which keeps nothing until you list metric names for it (see [Annotated pods](#annotated-pods)).

To keep more of a target's metrics than the list above, add their exact names under **Metrics → Extra metric names**, in the entry for that target (for example `cadvisor` or `gpu`). Names must match exactly; patterns aren't accepted.

{% hint style="info" %}
Scraping and Remote Write need agent v1.4.2 or later; an older agent collects no metrics and reports no error. See [Agent versions](#agent-versions).
{% endhint %}

### Annotated pods

To scrape a pod of your own, annotate its pod template with `prometheus.io/scrape: "true"` and `prometheus.io/port: "<port>"` (the port annotation is required), and make sure **Annotated pods** is on under the source's **Metrics** settings (the switch alone turns it on). Annotated pods keeps only the names you list under **Extra metric names → annotated\_pods**; with nothing listed, it keeps nothing.

### Remote Write

With the switch on, the agent on each node also accepts Prometheus **Remote Write** on port **9090**. Point your own Prometheus's `remote_write` at the agent's Service: with the installer's defaults that is `aigent-ds` in the `controltheory` namespace (`aigent-ds.controltheory.svc:9090`). A sender reaches the agent on its own node.

* **Remote Write 2.0 only.** A Remote Write 1.0 request is refused with HTTP 415. Classic histograms and summaries are refused.
* **Port:** change it in the source's **Remote Write port (optional)** field (or `--remote-write-port` in the CLI), from 1024 to 65535, except the OTLP ports and ports the agent already listens on; a conflicting port is refused. Re-run the install command after changing it, and point the sender at the new port.
* **No TLS or auth.** The port is open on the pod network only (never a host port), so any pod in the cluster can reach it. Fence it with a NetworkPolicy if you need to.

Series are filed under the sender's `job` as their service and `instance` as the instance. Remote Write carries no pod of its own, so a series is filed under the sender's node unless the sender also sends `target_info` with Kubernetes attributes; a cluster-wide Prometheus files everything under its own node.

## Kubernetes: OTLP from pods

If your pods already push metrics over OTLP to the agent on their node (ports `1757`/`1758`, see [Kubernetes](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/kubernetes.md)), turn on collecting them with the CLI (agent v1.4.4 or later). Keep the Prometheus switch on beside it by naming both:

```bash
dstl8 sources metrics prod --prom-metrics --otlp-metrics
```

`dstl8 sources metrics` takes the flags you give as the new set, so `--otlp-metrics` on its own collects OTLP from pods and turns Prometheus metrics off. See [Metrics CLI](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/cli.md#dstl8-sources-metrics).

OTLP from pods needs nothing from the cluster, so it doesn't change the install command. It is off by default because pods may already push metrics there that nobody has been collecting, and turning it on unasked could bring a surprising volume.

## When the same metric arrives twice

If a pod's metric comes in by more than one path, Dstl8 keeps one copy, and the source page counts the copies it skipped. See [When the same metric arrives twice](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/series-limits.md#when-the-same-metric-arrives-twice) for which copy wins.

## Limits

Each agent keeps a budget of series, and a new series past it is folded into one overflow series per metric, not dropped. Each source's page shows where it stands. See [Series limits](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/series-limits.md) for the numbers, what folding means and how to fix it. The scrape interval is 60 seconds and can't be changed.

## Not supported

* Metrics from AWS CloudWatch
* A hosted Remote Write endpoint (Remote Write goes to the agent in your cluster)
* Your own `scrape_configs`. To scrape your own pods, annotate them and use [Annotated pods](#annotated-pods)

## Troubleshooting

* **No metrics after turning the switch on.** Re-run the install command: the switch only takes effect once the agent has the access it needs. Then check the agent version; an agent older than v1.4.2 quietly collects logs only (see [Agent versions](#agent-versions)).
* **No node-exporter or kube-state-metrics series.** The agent only scrapes the ones your cluster already runs, found by the labels above. Check their labels, or install them.
* **Remote Write gets HTTP 415.** The sender is using Remote Write 1.0. Switch it to 2.0.
* **OTLP metrics arrive but no workspace shows them.** Check `service.name` is set and that service is assigned to the workspace, with **Metrics** among the workspace's signals. See [Which workspace sees which metrics](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics.md#which-workspace-sees-which-metrics).
* **A source shows Folded in the last hour.** A label with too many values. See [Series limits](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/series-limits.md#how-to-fix-it).

## Tips

* **Start with the switch alone.** It scrapes every target that exists in your cluster (Annotated pods once you list names for it) and accepts Remote Write; narrow it with `--prom-scrape` later if you need to.
* **Prefer OTLP for your own app metrics.** When the same metric also arrives by Remote Write or scraping, the OTLP copy wins anyway.
* **Name your services consistently.** The same `service.name` on metrics, traces and logs keeps a service's signals together on its service page.
* **Check the Series column** in **All metrics** on the [Metrics page](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/explore.md) after adding a new exporter: a metric with thousands of series is a sign of a high-variety label.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.controltheory.com/controltheory-documentation/dstl8-docs/metrics/collecting.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
