Tap: Pipe In Any Logs

Pipe logs from anywhere into Dstl8 — format auto-detected, sent as OTLP

dstl8 tap pipes any logs into Dstl8 without setting up a dedicated source. It reads log lines from stdin (or runs a local OTLP endpoint with --listen), auto-detects the format and vendor, converts each record to OTLP with proper resource attributes, and streams it to your organization's OTLP source — so it renders and groups in the platform exactly like Kubernetes, Docker, and other native sources.

kubectl logs -f deploy/api | dstl8 tap          # tail a deployment straight into Dstl8
flyctl logs -j             | dstl8 tap          # fly.io JSON logs
heroku logs --tail         | dstl8 tap          # heroku logplex
docker compose logs -f     | dstl8 tap          # per-service compose logs
cat app.log                | dstl8 tap          # a plain log file
dstl8 tap --listen                              # receive OTLP locally (SDKs/collectors)

No OTLP source in your org yet? Tap auto-creates one named tap on first use (the token is generated server-side), waits for it to come online, then starts streaming. Nothing to configure.

A running tap looks like this:

Tapping logs → otlp source "tap" (https://acme.dstl8.ai/src-a1b2c3)
  Detected format: vercel (100)   service.name=my-app   service.instance.id=my-app.vercel.app
  tap: 1204 read · 1204 sent · 0 failed · 0 dropped · 512.0 KiB

Diagnostics (the banner, detection line, and periodic stats) print to stderr; stdout is reserved for --dry-run OTLP JSON and --echo records, so both pipe cleanly into other tools.

How detection works

Tap samples the first several lines of the stream, locks onto the best-matching format, and — for transport wrappers like kubectl, docker, and heroku — strips the wrapper prefix and re-detects the inner payload. A per-line fallback handles mixed streams (for example, a JSON stream with an occasional plain-text panic). Multi-line stack traces (Java, Python tracebacks, Go panics) are joined into a single record, and vendor double-encoding (Vercel's [fn] {json}, Fly's inner JSON) is unwrapped automatically.

Force a format with --format <name> to skip detection entirely.

Supported formats:

Category	Formats
Vendors	vercel, supabase, cloudflare (Logpush), wrangler (Workers), flyio, victorialogs, heroku, docker, kubectl, cloudwatch (aws logs tail), gcp (Cloud Logging)
Generic	otlp (OTLP-JSON passthrough), json (zap/logrus/pino/bunyan dialects), logfmt, syslog (RFC 5424 + 3164), journald, klog/glog, java (log4j/logback), python, rails, access (nginx/Apache combined), nginx-error, text (plain-text fallback)

Streams and resource attributes

Every record's service.name becomes the platform stream type and service.instance.id becomes the stream name. Tap fills these from per-vendor extraction (the Vercel project, Kubernetes pod/container, Fly app, and so on), falling back to the format name and hostname.

• Override them with --service (stream type) and --instance (stream name).

• Add arbitrary resource attributes with --attr key=value (repeatable).

Flags

Flag	Description
--source <name-or-id>	OTLP source to send to (default: the org's only otlp source; auto-created if none)
--format <name>	Force an input format, skipping auto-detection
--service <name>	Override service.name (the platform stream type)
--instance <name>	Override service.instance.id (the platform stream name)
--attr key=value	Add an extra resource attribute (repeatable)
--listen	Receive OTLP on local ports instead of reading stdin
--listen-port <n>	Local OTLP HTTP port for --listen (default 4318)
--listen-grpc-port <n>	Local OTLP gRPC port for --listen (default 4317)
--dry-run	Print the OTLP JSON to stdout instead of sending (works offline — no source or login needed)
--echo	Print each parsed record as it's sent
--quiet	Suppress the banner and periodic stats

Examples

Kubernetes (kubectl)

kubectl logs -f deploy/api | dstl8 tap
kubectl logs -f deploy/api --timestamps --prefix | dstl8 tap    # keep pod/container identity
kubectl logs -f -l app=checkout --prefix | dstl8 tap            # by label selector
stern <selector> | dstl8 tap --format stern                     # multi-pod tailing with stern

• --prefix adds [pod/<name>/<container>] to each line so tap can set k8s.* attributes and derive the service name from the container.

• stern's plain output is too ambiguous to auto-detect — pass --format stern (strip color with stern --color never if needed).

Tapping kubectl is great for ad-hoc debugging. For continuous, cluster-wide collection, install the Kubernetes source instead.

Docker

docker compose logs -f | dstl8 tap
docker logs -f -t <container> | dstl8 tap

Compose's service-1 | prefix sets the service and container name automatically. -t adds timestamps.

AWS CloudWatch

aws logs tail <log-group> --follow | dstl8 tap --service eks-prod
aws logs tail <log-group> --follow --since 1h | dstl8 tap

• Keep the default detailed output format (it includes the stream name) — don't use --format short.

• Works across Lambda (.../[$LATEST]...), ECS, and EKS control-plane stream shapes; the inner payload (JSON, klog, logfmt) is re-detected automatically.

• The log group name isn't part of the output, so the stream type defaults to cloudwatch — pass --service <name> to label it.

Vercel

vercel logs <deployment-url> -j -f | dstl8 tap
vercel logs <deployment-url> -j -f --scope <team> | dstl8 tap   # project owned by a team
vercel logs <deployment-url> -j -n 100 | dstl8 tap              # recent history instead of follow

• -j (JSON Lines) is required — the default human-readable output isn't structured enough to detect and map.

• -f streams only new logs; use -n <count> to pull history.

• Add --scope <team> if the project belongs to a team (otherwise you'll see "No projects found"), or run vercel link once in the project directory and drop --scope.

Fly.io

flyctl logs -j | dstl8 tap

-j for JSON output. The envelope level is always info; tap unwraps the inner JSON to recover the real severity.

Heroku

heroku logs --tail | dstl8 tap

The Heroku CLI has no JSON mode; tap parses the logplex text and re-detects the inner payload (router lines are logfmt, app lines vary).

GCP Cloud Logging

gcloud logging tail 'resource.type=cloud_run_revision' --format=json | jq -c '.[]' | dstl8 tap

gcloud pretty-prints a JSON array, so pipe through jq -c '.[]' to get one object per line.

systemd / journald

journalctl -f -o json | dstl8 tap

-o json is required for structured fields (MESSAGE, PRIORITY, _SYSTEMD_UNIT, …).

Any app or file

./my-server 2>&1         | dstl8 tap    # a running process (merge stderr with 2>&1)
tail -f /var/log/app.log | dstl8 tap    # a growing file
cat app.log              | dstl8 tap    # a static file

Labeling and previewing

# Force a format and label the stream:
flyctl logs -j | dstl8 tap --format fly --service my-app

# Preview detection and the OTLP mapping without sending anything:
cat app.log | dstl8 tap --dry-run --echo

# Add extra resource attributes:
kubectl logs -f deploy/api | dstl8 tap --attr env=prod --attr team=payments

# Send to a specific source:
cat app.log | dstl8 tap --source prod-otlp

Local OTLP listener (--listen)

Instead of reading stdin, tap can run a local OTLP endpoint and forward everything it receives to Dstl8. Point any OpenTelemetry SDK, agent, or collector at it — no per-app credentials or endpoint configuration on the Dstl8 side.

dstl8 tap --listen

This listens on both standard OTLP ports: gRPC on 4317 and HTTP on 4318 (change them with --listen-grpc-port / --listen-port). Incoming resource attributes are preserved; service.name and service.instance.id are filled with fallbacks only when absent.

Point an OTel SDK at it

# gRPC (default port 4317):
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# or HTTP (default port 4318):
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Then run your instrumented app as usual — its logs flow through tap into Dstl8.

Point an OTel Collector at it

Add an OTLP exporter to your collector pipeline:

exporters:
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlp]

Generate test logs with telemetrygen

telemetrygen is the OpenTelemetry Collector's load-generation tool — handy for trying tap end-to-end without any real infrastructure.

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest

In one terminal, start the listener:

dstl8 tap --listen

In another, generate logs:

# 5 logs/second for 30 seconds over gRPC (port 4317):
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 --duration 30s --rate 5

# Same over HTTP (port 4318):
telemetrygen logs --otlp-http --otlp-insecure --otlp-endpoint localhost:4318 --duration 30s --rate 5

# A fixed batch of 100 logs with a custom service name, body, and severity:
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 --logs 100 \
  --service checkout --body "payment authorized" --severity-text Warn --severity-number 13

# Extra resource attributes (values must be quoted):
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 --duration 30s --rate 5 \
  --service checkout --otlp-attributes 'env="staging"'

Then confirm arrival:

dstl8 logs tail --source tap

Requirements and nuances

• Input must be piped. Running dstl8 tap in a bare terminal errors out — pipe logs in, or use --listen for the OTLP server mode.

• Prefer a tool's JSON/structured mode when it has one (-j, --format json, -o json) — you get richer fields and more reliable detection than human-formatted output.

• Follow flags stream only new lines. -f / --follow / --tail won't replay history; use the tool's count/since flags (-n, --since) for backlog.

• Detection samples the first several lines. A very short burst may show Detected format: mixed — pass --format <name> to force it. Real streams lock on their own.

• Everything lands in one OTLP source (auto-created as tap if you have none). Use --service (stream type) and --instance (stream name) to separate and label streams in the dashboard.

• With more than one OTLP source, tap won't guess — pass --source <name-or-id>.

• Merge stderr with 2>&1 when tapping an app that logs to stderr.

• A freshly created source takes a few seconds to provision. Tap waits for it to come online before sending (you'll see "Waiting for the source endpoint to come online…"); no logs are lost.

• Verify arrival with dstl8 logs tail --source tap (or your --source).

Tap batches records and retries transient failures with backoff. On Ctrl+C (or stdin EOF) it flushes any buffered records and prints a summary.