OTLP Protocol Details

Deep dive into Gonzo's OpenTelemetry Protocol (OTLP) implementation for receiving and processing logs.

Overview

Gonzo implements a complete OTLP logs receiver supporting both gRPC and HTTP transports. This allows Gonzo to receive logs directly from OpenTelemetry SDKs, collectors, and other OTLP-compatible sources.

OTLP Specification

Gonzo follows the OpenTelemetry Protocol Specification for logs:

Version: OTLP 1.0.0+
Transports: gRPC and HTTP
Encoding: Protocol Buffers (gRPC), JSON (HTTP)
Endpoints:
- gRPC: localhost:4317
- HTTP: http://localhost:4318/v1/logs

Architecture

┌─────────────────────────────────────────┐
│         OTLP Client (Sender)            │
│    (App, Collector, SDK, etc)           │
└──────────────┬──────────────────────────┘
               │
               ├──────────┐
               │          │
           gRPC:4317   HTTP:4318
               │          │
    ┌──────────▼────┐  ┌──▼──────────┐
    │  gRPC Server  │  │ HTTP Server │
    │               │  │             │
    └──────┬────────┘  └─┬───────────┘
           │             │
           └─────┬───────┘
                 │
        ┌────────▼─────────┐
        │  OTLP Processor  │
        │  - Decode proto  │
        │  - Extract logs  │
        │  - Map fields    │
        └────────┬─────────┘
                 │
        ┌────────▼─────────┐
        │  Log Converter   │
        │  - Resource attr │
        │  - Log record    │
        │  - Timestamps    │
        └────────┬─────────┘
                 │
        ┌────────▼─────────┐
        │   Analyzer       │
        │   Engine         │
        └──────────────────┘

Transport Details

gRPC Transport

Endpoint: localhost:4317

Protocol: HTTP/2 with Protocol Buffers

Service Definition:

service LogsService {
  rpc Export(ExportLogsServiceRequest) 
    returns (ExportLogsServiceResponse) {}
}

Features:

Bidirectional streaming support
Built-in compression (gzip)
Connection multiplexing
Automatic reconnection

Starting gRPC Receiver:

# Default port 4317
gonzo --otlp-enabled

# Custom port
gonzo --otlp-enabled --otlp-grpc-port=5317

HTTP Transport

Endpoint: http://localhost:4318/v1/logs

Protocol: HTTP/1.1 or HTTP/2

Method: POST

Content-Type: application/json or application/x-protobuf

Features:

Standard HTTP clients work
JSON encoding (more debuggable)
No special libraries required
Works through HTTP proxies

Starting HTTP Receiver:

# Default port 4318
gonzo --otlp-enabled

# Custom port
gonzo --otlp-enabled --otlp-http-port=5318

OTLP Log Data Model

Structure Hierarchy

ExportLogsServiceRequest
└── ResourceLogs[]
    ├── Resource
    │   └── Attributes[]
    │       ├── service.name
    │       ├── host.name
    │       └── ... (resource attributes)
    └── ScopeLogs[]
        ├── Scope
        │   ├── name
        │   └── version
        └── LogRecords[]
            ├── timeUnixNano
            ├── observedTimeUnixNano
            ├── severityNumber
            ├── severityText
            ├── body (AnyValue)
            ├── attributes[]
            └── traceId/spanId (optional)

Resource Attributes

Resource attributes describe the source of logs:

{
  "service.name": "payment-api",
  "service.version": "1.2.3",
  "host.name": "prod-server-01",
  "deployment.environment": "production",
  "cloud.provider": "aws",
  "cloud.region": "us-east-1"
}

How Gonzo Uses Resource Attributes:

Merged with log record attributes
Displayed in Attributes panel
Available for filtering
Shown in log details

Log Record

Individual log entry:

{
  "timeUnixNano": "1705315805000000000",
  "observedTimeUnixNano": "1705315805123456789",
  "severityNumber": 13,  // ERROR
  "severityText": "error",
  "body": {
    "stringValue": "Database connection failed"
  },
  "attributes": [
    {"key": "user_id", "value": {"intValue": "12345"}},
    {"key": "endpoint", "value": {"stringValue": "/api/users"}},
    {"key": "error.type", "value": {"stringValue": "timeout"}}
  ],
  "traceId": "1234567890abcdef",
  "spanId": "1234567890abcdef"
}

Gonzo Mapping:

timeUnixNano        → Timestamp
severityText        → Level (error, warn, info, debug)
body                → Message
attributes          → Extracted to Attributes panel
traceId/spanId      → Available for tracing correlation

Severity Levels

OTLP defines numerical severity levels:

Number

Name

Gonzo Display

1-4

TRACE

⚪ White

5-8

DEBUG

🔵 Blue

9-12

INFO

🟢 Green

13-16

WARN

🟡 Yellow

17-20

ERROR

🔴 Red

21-24

FATAL

🔴 Red

Gonzo reads both severityNumber and severityText fields.

Body Types

The log body can be various types:

String (most common):

{
  "body": {
    "stringValue": "User logged in"
  }
}

Structured:

{
  "body": {
    "kvlistValue": {
      "values": [
        {"key": "event", "value": {"stringValue": "login"}},
        {"key": "user_id", "value": {"intValue": 12345}}
      ]
    }
  }
}

Gonzo handles all OTLP value types and displays them appropriately.

Configuration Examples

OpenTelemetry Collector

Configure collector to send to Gonzo:

# otel-collector-config.yaml
receivers:
  filelog:
    include: [/var/log/app/*.log]
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 100

exporters:
  # Send to Gonzo via gRPC
  otlp/gonzo_grpc:
    endpoint: localhost:4317
    tls:
      insecure: true
  
  # Or via HTTP
  otlphttp/gonzo_http:
    endpoint: http://localhost:4318/v1/logs

service:
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [batch]
      exporters: [otlp/gonzo_grpc]  # or otlphttp/gonzo_http

Start collector and Gonzo:

# Terminal 1: Start Gonzo
gonzo --otlp-enabled

# Terminal 2: Start collector
otelcol --config otel-collector-config.yaml

Application SDK Integration

Python Example:

from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource
import logging

# Create resource with service information
resource = Resource.create({
    "service.name": "my-python-app",
    "service.version": "1.0.0",
    "deployment.environment": "production"
})

# Configure OTLP exporter (gRPC)
exporter = OTLPLogExporter(
    endpoint="localhost:4317",
    insecure=True
)

# Set up logger provider
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(exporter)
)
_logs.set_logger_provider(logger_provider)

# Integrate with standard logging
handler = LoggingHandler(
    level=logging.NOTSET,
    logger_provider=logger_provider
)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Now logs go to Gonzo
logging.info("Application started")
logging.error("Something went wrong", extra={
    "user_id": 12345,
    "endpoint": "/api/users"
})

Go Example:

package main

import (
    "context"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    "go.opentelemetry.io/otel/log"
    "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func main() {
    // Create resource
    res, _ := resource.New(context.Background(),
        resource.WithAttributes(
            semconv.ServiceName("my-go-app"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )

    // Create OTLP exporter
    exporter, _ := otlploggrpc.New(context.Background(),
        otlploggrpc.WithEndpoint("localhost:4317"),
        otlploggrpc.WithInsecure(),
    )

    // Create logger provider
    provider := sdklog.NewLoggerProvider(
        sdklog.WithResource(res),
        sdklog.WithProcessor(sdklog.NewBatchProcessor(exporter)),
    )
    defer provider.Shutdown(context.Background())

    // Get logger
    logger := provider.Logger("app-logger")

    // Send logs
    logger.Emit(context.Background(), log.Record{
        Severity:  log.SeverityInfo,
        Body:      log.StringValue("Application started"),
    })
}

Node.js Example:

const { LoggerProvider } = require('@opentelemetry/sdk-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

// Create resource
const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'my-node-app',
  [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
});

// Create exporter
const exporter = new OTLPLogExporter({
  url: 'grpc://localhost:4317',
});

// Create logger provider
const loggerProvider = new LoggerProvider({ resource });
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(exporter)
);

// Get logger
const logger = loggerProvider.getLogger('app-logger');

// Send logs
logger.emit({
  severityText: 'INFO',
  body: 'Application started',
  attributes: {
    userId: 12345,
    endpoint: '/api/users',
  },
});

HTTP API Reference

POST /v1/logs

Request Format (JSON):

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "my-service" }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my-logger"
          },
          "logRecords": [
            {
              "timeUnixNano": "1705315805000000000",
              "severityNumber": 9,
              "severityText": "info",
              "body": {
                "stringValue": "Log message"
              },
              "attributes": [
                {
                  "key": "user_id",
                  "value": { "intValue": "12345" }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Response (Success):

{
  "partialSuccess": {}
}

Response (Partial Failure):

{
  "partialSuccess": {
    "rejectedLogRecords": 5,
    "errorMessage": "Some records were invalid"
  }
}

Status Codes:

200 OK - All logs accepted
206 Partial Content - Some logs rejected (see partialSuccess)
400 Bad Request - Invalid request format
500 Internal Server Error - Server error

Testing HTTP Endpoint

# Simple test
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [
      {
        "scopeLogs": [
          {
            "logRecords": [
              {
                "timeUnixNano": "1705315805000000000",
                "severityText": "info",
                "body": {"stringValue": "Test log"}
              }
            ]
          }
        ]
      }
    ]
  }'

# Should see log appear in Gonzo

gRPC API Reference

Service: opentelemetry.proto.collector.logs.v1.LogsService

Method: Export

Request: ExportLogsServiceRequest

Response: ExportLogsServiceResponse

Protocol Buffers Definition:

service LogsService {
  rpc Export(ExportLogsServiceRequest) 
    returns (ExportLogsServiceResponse) {}
}

message ExportLogsServiceRequest {
  repeated ResourceLogs resource_logs = 1;
}

message ExportLogsServiceResponse {
  ExportLogsPartialSuccess partial_success = 1;
}

message ExportLogsPartialSuccess {
  int64 rejected_log_records = 1;
  string error_message = 2;
}

Testing gRPC Endpoint

# Using grpcurl
grpcurl -plaintext \
  -d '{
    "resource_logs": [{
      "scope_logs": [{
        "log_records": [{
          "time_unix_nano": "1705315805000000000",
          "severity_text": "info",
          "body": {"string_value": "Test log"}
        }]
      }]
    }]
  }' \
  localhost:4317 \
  opentelemetry.proto.collector.logs.v1.LogsService/Export

Performance & Optimization

Batching

Gonzo processes logs in batches for efficiency:

Incoming Batches:

OTLP senders should batch logs (recommended 100-1000 per batch)
Reduces network overhead
Improves throughput

Internal Processing:

Logs processed individually after receipt
Real-time display without buffering delay

Concurrency

gRPC Handler:

Concurrent request handling
One goroutine per connection
Thread-safe log ingestion

HTTP Handler:

Standard HTTP server concurrency
Multiple simultaneous POST requests supported

Memory Management

Bounded Buffers:

OTLP messages are streamed, not accumulated
Converted logs go to main log buffer (configurable size)
No unlimited memory growth

Large Messages:

Default max receive message size: 4MB (configurable)
Larger messages can be split by sender

# Increase max message size if needed
# (requires code modification currently)

Security Considerations

TLS/SSL

Production Recommendation: Always use TLS in production.

Current Implementation:

gRPC: Insecure mode (no TLS)
HTTP: Plain HTTP

For Production Use:

Run Gonzo behind TLS-terminating proxy (nginx, envoy)
Use OTLP collector with TLS, forward to Gonzo locally

Example with nginx:

server {
    listen 443 ssl http2;
    server_name gonzo.example.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    # gRPC proxy
    location / {
        grpc_pass grpc://localhost:4317;
    }
}

Authentication

Current Implementation: No built-in authentication

Recommendations:

Use in trusted networks only
Deploy behind authenticated proxy
Use firewall rules to restrict access
Consider VPN for remote access

Network Isolation

Best Practice: Don't expose OTLP ports to public internet

# Bind to localhost only (default)
gonzo --otlp-enabled  # Listens on 127.0.0.1

# For containers, use bridge networking
# For Kubernetes, use ClusterIP service

Troubleshooting OTLP Issues

Connection Refused

Symptom: Sender can't connect to Gonzo

Checks:

# Verify Gonzo is running with OTLP enabled
ps aux | grep gonzo

# Check ports are listening
lsof -i :4317  # gRPC
lsof -i :4318  # HTTP

# Test connectivity
telnet localhost 4317
curl http://localhost:4318/v1/logs

Logs Not Appearing

Symptom: Connection works but no logs in Gonzo

Diagnosis:

Check request format matches OTLP spec
Verify severityText or severityNumber is set
Check body field is present
Look for error responses from Gonzo

Test with minimal payload:

curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"body":{"stringValue":"test"}}]}]}]}'

Message Size Exceeded

Symptom: "message size exceeded" errors

Solution:

Reduce batch size in sender
Split large log messages
Currently no config option (fixed at 4MB)

Performance Issues

Symptom: Slow log processing or high CPU

Solutions:

# Reduce update frequency
gonzo --otlp-enabled --update-interval=5s

# Reduce buffer sizes
gonzo --otlp-enabled --log-buffer=500

Compatibility

OTLP Version Support

Specification Version: 1.0.0+
Protocol Version: v1
Backward Compatibility: Follows semantic versioning

OpenTelemetry SDK Compatibility

Tested with:

✅ OpenTelemetry Collector v0.90+
✅ Python SDK v1.20+
✅ Go SDK v1.20+
✅ Node.js SDK v0.45+
✅ Java SDK v1.32+
✅ .NET SDK v1.6+

Transport Support Matrix

Transport

Encoding

Status

gRPC

Protobuf

✅ Supported

HTTP

JSON

✅ Supported

HTTP

Protobuf

✅ Supported

WebSocket

❌ Not supported

Future Enhancements

Planned Features:

TLS support (direct, without proxy)
Authentication (API keys, mTLS)
Compression (gzip, zstd)
Configurable message size limits
Metrics export (self-monitoring)
Multiple OTLP receivers (different ports for different sources)

Resources

Specifications

OpenTelemetry Documentation

Architecture Overview
Common Issues - OTLP
Configuration Reference

Quick Start: gonzo --otlp-enabled starts both gRPC (4317) and HTTP (4318) receivers. Point your OTLP exporter to these endpoints and logs will appear in Gonzo immediately!

Last updated 7 months ago

hashtagOverview

hashtagOTLP Specification

hashtagArchitecture

hashtagTransport Details

hashtaggRPC Transport

hashtagHTTP Transport

hashtagOTLP Log Data Model

hashtagStructure Hierarchy

hashtagResource Attributes

hashtagLog Record

hashtagSeverity Levels

hashtagBody Types

hashtagConfiguration Examples

hashtagOpenTelemetry Collector

hashtagApplication SDK Integration

hashtagHTTP API Reference

hashtagPOST /v1/logs

hashtagTesting HTTP Endpoint

hashtaggRPC API Reference

hashtagService: opentelemetry.proto.collector.logs.v1.LogsService

hashtagTesting gRPC Endpoint

hashtagPerformance & Optimization

hashtagBatching

hashtagConcurrency

hashtagMemory Management

hashtagSecurity Considerations

hashtagTLS/SSL

hashtagAuthentication

hashtagNetwork Isolation

hashtagTroubleshooting OTLP Issues

hashtagConnection Refused

hashtagLogs Not Appearing

hashtagMessage Size Exceeded

hashtagPerformance Issues

hashtagCompatibility

hashtagOTLP Version Support

hashtagOpenTelemetry SDK Compatibility

hashtagTransport Support Matrix

hashtagFuture Enhancements

hashtagResources

hashtagSpecifications

hashtagOpenTelemetry Documentation

hashtagRelated Gonzo Docs

Overview

OTLP Specification

Architecture

Transport Details

gRPC Transport

HTTP Transport

OTLP Log Data Model

Structure Hierarchy

Resource Attributes

Log Record

Severity Levels

Body Types

Configuration Examples

OpenTelemetry Collector

Application SDK Integration

HTTP API Reference

POST /v1/logs

Testing HTTP Endpoint

gRPC API Reference

Service: opentelemetry.proto.collector.logs.v1.LogsService

Testing gRPC Endpoint

Performance & Optimization

Batching

Concurrency

Memory Management

Security Considerations

TLS/SSL

Authentication

Network Isolation

Troubleshooting OTLP Issues

Connection Refused

Logs Not Appearing

Message Size Exceeded

Performance Issues

Compatibility

OTLP Version Support

OpenTelemetry SDK Compatibility

Transport Support Matrix

Future Enhancements

Resources

Specifications

OpenTelemetry Documentation

Related Gonzo Docs