> For the complete documentation index, see [llms.txt](https://docs.controltheory.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/dstl8-cli/tap.md). # Tap: Pipe In Any Logs `dstl8 tap` pipes *any* logs into Dstl8 without setting up a dedicated source. It reads log lines from stdin (or runs a local OTLP endpoint with `--listen`), auto-detects the format and vendor, converts each record to OTLP with proper resource attributes, and streams it to your organization's OTLP source — so it renders and groups in the platform exactly like Kubernetes, Docker, and other native sources. ```bash kubectl logs -f deploy/api | dstl8 tap # tail a deployment straight into Dstl8 flyctl logs -j | dstl8 tap # fly.io JSON logs heroku logs --tail | dstl8 tap # heroku logplex docker compose logs -f | dstl8 tap # per-service compose logs cat app.log | dstl8 tap # a plain log file dstl8 tap --listen # receive OTLP locally (SDKs/collectors) ``` {% hint style="info" %} **No OTLP source in your org yet?** Tap auto-creates one named `tap` on first use (the token is generated server-side), waits for it to come online, then starts streaming. Nothing to configure. {% endhint %} A running tap looks like this: ``` Tapping logs → otlp source "tap" (https://acme.dstl8.ai/src-a1b2c3) Detected format: vercel (100) service.name=my-app service.instance.id=my-app.vercel.app tap: 1204 read · 1204 sent · 0 failed · 0 dropped · 512.0 KiB ``` Diagnostics (the banner, detection line, and periodic stats) print to **stderr**; **stdout** is reserved for `--dry-run` OTLP JSON and `--echo` records, so both pipe cleanly into other tools. ### How detection works Tap samples the first several lines of the stream, locks onto the best-matching format, and — for transport wrappers like `kubectl`, `docker`, and `heroku` — strips the wrapper prefix and re-detects the inner payload. A per-line fallback handles mixed streams (for example, a JSON stream with an occasional plain-text panic). Multi-line stack traces (Java, Python tracebacks, Go panics) are joined into a single record, and vendor double-encoding (Vercel's `[fn] {json}`, Fly's inner JSON) is unwrapped automatically. Force a format with `--format ` to skip detection entirely. **Supported formats:** | Category | Formats | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Vendors | `vercel`, `supabase`, `cloudflare` (Logpush), `wrangler` (Workers), `flyio`, `victorialogs`, `heroku`, `docker`, `kubectl`, `cloudwatch` (`aws logs tail`), `gcp` (Cloud Logging) | | Generic | `otlp` (OTLP-JSON passthrough), `json` (zap/logrus/pino/bunyan dialects), `logfmt`, `syslog` (RFC 5424 + 3164), `journald`, `klog`/`glog`, `java` (log4j/logback), `python`, `rails`, `access` (nginx/Apache combined), `nginx-error`, `text` (plain-text fallback) | ### Streams and resource attributes Every record's `service.name` becomes the platform **stream type** and `service.instance.id` becomes the **stream name**. Tap fills these from per-vendor extraction (the Vercel project, Kubernetes pod/container, Fly app, and so on), falling back to the format name and hostname. * Override them with `--service` (stream type) and `--instance` (stream name). * Add arbitrary resource attributes with `--attr key=value` (repeatable). ### Flags | Flag | Description | | ------------------------ | -------------------------------------------------------------------------------------------- | | `--source ` | OTLP source to send to (default: the org's only `otlp` source; auto-created if none) | | `--format ` | Force an input format, skipping auto-detection | | `--service ` | Override `service.name` (the platform stream type) | | `--instance ` | Override `service.instance.id` (the platform stream name) | | `--attr key=value` | Add an extra resource attribute (repeatable) | | `--listen` | Receive OTLP on local ports instead of reading stdin | | `--listen-port ` | Local OTLP HTTP port for `--listen` (default 4318) | | `--listen-grpc-port ` | Local OTLP gRPC port for `--listen` (default 4317) | | `--dry-run` | Print the OTLP JSON to stdout instead of sending (works offline — no source or login needed) | | `--echo` | Print each parsed record as it's sent | | `--quiet` | Suppress the banner and periodic stats | ## Examples ### Kubernetes (kubectl) ```bash kubectl logs -f deploy/api | dstl8 tap kubectl logs -f deploy/api --timestamps --prefix | dstl8 tap # keep pod/container identity kubectl logs -f -l app=checkout --prefix | dstl8 tap # by label selector stern | dstl8 tap --format stern # multi-pod tailing with stern ``` * `--prefix` adds `[pod//]` to each line so tap can set `k8s.*` attributes and derive the service name from the container. * `stern`'s plain output is too ambiguous to auto-detect — pass `--format stern` (strip color with `stern --color never` if needed). {% hint style="info" %} Tapping `kubectl` is great for ad-hoc debugging. For continuous, cluster-wide collection, install the [Kubernetes source](/controltheory-documentation/dstl8-docs/sources/kubernetes.md) instead. {% endhint %} ### Docker ```bash docker compose logs -f | dstl8 tap docker logs -f -t | dstl8 tap ``` Compose's `service-1 |` prefix sets the service and container name automatically. `-t` adds timestamps. ### AWS CloudWatch ```bash aws logs tail --follow | dstl8 tap --service eks-prod aws logs tail --follow --since 1h | dstl8 tap ``` * Keep the default `detailed` output format (it includes the stream name) — don't use `--format short`. * Works across Lambda (`.../[$LATEST]...`), ECS, and EKS control-plane stream shapes; the inner payload (JSON, klog, logfmt) is re-detected automatically. * The log group name isn't part of the output, so the stream type defaults to `cloudwatch` — pass `--service ` to label it. ### Vercel ```bash vercel logs -j -f | dstl8 tap vercel logs -j -f --scope | dstl8 tap # project owned by a team vercel logs -j -n 100 | dstl8 tap # recent history instead of follow ``` * `-j` (JSON Lines) is **required** — the default human-readable output isn't structured enough to detect and map. * `-f` streams only *new* logs; use `-n ` to pull history. * Add `--scope ` if the project belongs to a team (otherwise you'll see "No projects found"), or run `vercel link` once in the project directory and drop `--scope`. ### Fly.io ```bash flyctl logs -j | dstl8 tap ``` `-j` for JSON output. The envelope `level` is always `info`; tap unwraps the inner JSON to recover the real severity. ### Heroku ```bash heroku logs --tail | dstl8 tap ``` The Heroku CLI has no JSON mode; tap parses the logplex text and re-detects the inner payload (router lines are logfmt, app lines vary). ### GCP Cloud Logging ```bash gcloud logging tail 'resource.type=cloud_run_revision' --format=json | jq -c '.[]' | dstl8 tap ``` `gcloud` pretty-prints a JSON *array*, so pipe through `jq -c '.[]'` to get one object per line. ### systemd / journald ```bash journalctl -f -o json | dstl8 tap ``` `-o json` is required for structured fields (`MESSAGE`, `PRIORITY`, `_SYSTEMD_UNIT`, …). ### Any app or file ```bash ./my-server 2>&1 | dstl8 tap # a running process (merge stderr with 2>&1) tail -f /var/log/app.log | dstl8 tap # a growing file cat app.log | dstl8 tap # a static file ``` ### Labeling and previewing ```bash # Force a format and label the stream: flyctl logs -j | dstl8 tap --format fly --service my-app # Preview detection and the OTLP mapping without sending anything: cat app.log | dstl8 tap --dry-run --echo # Add extra resource attributes: kubectl logs -f deploy/api | dstl8 tap --attr env=prod --attr team=payments # Send to a specific source: cat app.log | dstl8 tap --source prod-otlp ``` ## Local OTLP listener (`--listen`) Instead of reading stdin, tap can run a local OTLP endpoint and forward everything it receives to Dstl8. Point any OpenTelemetry SDK, agent, or collector at it — no per-app credentials or endpoint configuration on the Dstl8 side. ```bash dstl8 tap --listen ``` This listens on both standard OTLP ports: **gRPC on 4317** and **HTTP on 4318** (change them with `--listen-grpc-port` / `--listen-port`). Incoming resource attributes are preserved; `service.name` and `service.instance.id` are filled with fallbacks only when absent. ### Point an OTel SDK at it ```bash # gRPC (default port 4317): export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 # or HTTP (default port 4318): export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf ``` Then run your instrumented app as usual — its logs flow through tap into Dstl8. ### Point an OTel Collector at it Add an OTLP exporter to your collector pipeline: ```yaml exporters: otlp: endpoint: localhost:4317 tls: insecure: true service: pipelines: logs: receivers: [otlp] exporters: [otlp] ``` ### Generate test logs with telemetrygen [`telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) is the OpenTelemetry Collector's load-generation tool — handy for trying tap end-to-end without any real infrastructure. ```bash go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest ``` In one terminal, start the listener: ```bash dstl8 tap --listen ``` In another, generate logs: ```bash # 5 logs/second for 30 seconds over gRPC (port 4317): telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 --duration 30s --rate 5 # Same over HTTP (port 4318): telemetrygen logs --otlp-http --otlp-insecure --otlp-endpoint localhost:4318 --duration 30s --rate 5 # A fixed batch of 100 logs with a custom service name, body, and severity: telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 --logs 100 \ --service checkout --body "payment authorized" --severity-text Warn --severity-number 13 # Extra resource attributes (values must be quoted): telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 --duration 30s --rate 5 \ --service checkout --otlp-attributes 'env="staging"' ``` Then confirm arrival: ```bash dstl8 logs tail --source tap ``` ## Requirements and nuances * **Input must be piped.** Running `dstl8 tap` in a bare terminal errors out — pipe logs in, or use `--listen` for the OTLP server mode. * **Prefer a tool's JSON/structured mode** when it has one (`-j`, `--format json`, `-o json`) — you get richer fields and more reliable detection than human-formatted output. * **Follow flags stream only new lines.** `-f` / `--follow` / `--tail` won't replay history; use the tool's count/since flags (`-n`, `--since`) for backlog. * **Detection samples the first several lines.** A very short burst may show `Detected format: mixed` — pass `--format ` to force it. Real streams lock on their own. * **Everything lands in one OTLP source** (auto-created as `tap` if you have none). Use `--service` (stream type) and `--instance` (stream name) to separate and label streams in the dashboard. * **With more than one OTLP source**, tap won't guess — pass `--source `. * **Merge stderr** with `2>&1` when tapping an app that logs to stderr. * **A freshly created source takes a few seconds to provision.** Tap waits for it to come online before sending (you'll see "Waiting for the source endpoint to come online…"); no logs are lost. * **Verify arrival** with `dstl8 logs tail --source tap` (or your `--source`). Tap batches records and retries transient failures with backoff. On `Ctrl+C` (or stdin EOF) it flushes any buffered records and prints a summary. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.controltheory.com/controltheory-documentation/dstl8-docs/dstl8-cli/tap.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.