OTLP Protocol Details

Deep dive into Gonzo's OpenTelemetry Protocol (OTLP) implementation for receiving and processing logs.

Overview

Gonzo implements a complete OTLP logs receiver supporting both gRPC and HTTP transports. This allows Gonzo to receive logs directly from OpenTelemetry SDKs, collectors, and other OTLP-compatible sources.

OTLP Specification

Gonzo follows the OpenTelemetry Protocol Specification for logs:

• Version: OTLP 1.0.0+

• Transports: gRPC and HTTP

• Encoding: Protocol Buffers (gRPC), JSON (HTTP)

• Endpoints:

  • gRPC: localhost:4317

  • HTTP: http://localhost:4318/v1/logs

Architecture

┌─────────────────────────────────────────┐
│         OTLP Client (Sender)            │
│    (App, Collector, SDK, etc)           │
└──────────────┬──────────────────────────┘
               │
               ├──────────┐
               │          │
           gRPC:4317   HTTP:4318
               │          │
    ┌──────────▼────┐  ┌──▼──────────┐
    │  gRPC Server  │  │ HTTP Server │
    │               │  │             │
    └──────┬────────┘  └─┬───────────┘
           │             │
           └─────┬───────┘
                 │
        ┌────────▼─────────┐
        │  OTLP Processor  │
        │  - Decode proto  │
        │  - Extract logs  │
        │  - Map fields    │
        └────────┬─────────┘
                 │
        ┌────────▼─────────┐
        │  Log Converter   │
        │  - Resource attr │
        │  - Log record    │
        │  - Timestamps    │
        └────────┬─────────┘
                 │
        ┌────────▼─────────┐
        │   Analyzer       │
        │   Engine         │
        └──────────────────┘

Transport Details

gRPC Transport

Endpoint: localhost:4317

Protocol: HTTP/2 with Protocol Buffers

Service Definition:

service LogsService {
  rpc Export(ExportLogsServiceRequest) 
    returns (ExportLogsServiceResponse) {}
}

Features:

• Bidirectional streaming support

• Built-in compression (gzip)

• Connection multiplexing

• Automatic reconnection

Starting gRPC Receiver:

# Default port 4317
gonzo --otlp-enabled

# Custom port
gonzo --otlp-enabled --otlp-grpc-port=5317

HTTP Transport

Endpoint: http://localhost:4318/v1/logs

Protocol: HTTP/1.1 or HTTP/2

Method: POST

Content-Type: application/json or application/x-protobuf

Features:

• Standard HTTP clients work

• JSON encoding (more debuggable)

• No special libraries required

• Works through HTTP proxies

Starting HTTP Receiver:

# Default port 4318
gonzo --otlp-enabled

# Custom port
gonzo --otlp-enabled --otlp-http-port=5318

OTLP Log Data Model

Structure Hierarchy

ExportLogsServiceRequest
└── ResourceLogs[]
    ├── Resource
    │   └── Attributes[]
    │       ├── service.name
    │       ├── host.name
    │       └── ... (resource attributes)
    └── ScopeLogs[]
        ├── Scope
        │   ├── name
        │   └── version
        └── LogRecords[]
            ├── timeUnixNano
            ├── observedTimeUnixNano
            ├── severityNumber
            ├── severityText
            ├── body (AnyValue)
            ├── attributes[]
            └── traceId/spanId (optional)

Resource Attributes

Resource attributes describe the source of logs:

{
  "service.name": "payment-api",
  "service.version": "1.2.3",
  "host.name": "prod-server-01",
  "deployment.environment": "production",
  "cloud.provider": "aws",
  "cloud.region": "us-east-1"
}

How Gonzo Uses Resource Attributes:

• Merged with log record attributes

• Displayed in Attributes panel

• Available for filtering

• Shown in log details

Log Record

Individual log entry:

{
  "timeUnixNano": "1705315805000000000",
  "observedTimeUnixNano": "1705315805123456789",
  "severityNumber": 13,  // ERROR
  "severityText": "error",
  "body": {
    "stringValue": "Database connection failed"
  },
  "attributes": [
    {"key": "user_id", "value": {"intValue": "12345"}},
    {"key": "endpoint", "value": {"stringValue": "/api/users"}},
    {"key": "error.type", "value": {"stringValue": "timeout"}}
  ],
  "traceId": "1234567890abcdef",
  "spanId": "1234567890abcdef"
}

Gonzo Mapping:

timeUnixNano        → Timestamp
severityText        → Level (error, warn, info, debug)
body                → Message
attributes          → Extracted to Attributes panel
traceId/spanId      → Available for tracing correlation

Severity Levels

OTLP defines numerical severity levels:

Number	Name	Gonzo Display
1-4	TRACE	⚪ White
5-8	DEBUG	🔵 Blue
9-12	INFO	🟢 Green
13-16	WARN	🟡 Yellow
17-20	ERROR	🔴 Red
21-24	FATAL	🔴 Red

Gonzo reads both severityNumber and severityText fields.

Body Types

The log body can be various types:

String (most common):

{
  "body": {
    "stringValue": "User logged in"
  }
}

Structured:

{
  "body": {
    "kvlistValue": {
      "values": [
        {"key": "event", "value": {"stringValue": "login"}},
        {"key": "user_id", "value": {"intValue": 12345}}
      ]
    }
  }
}

Gonzo handles all OTLP value types and displays them appropriately.

Configuration Examples

OpenTelemetry Collector

Configure collector to send to Gonzo:

# otel-collector-config.yaml
receivers:
  filelog:
    include: [/var/log/app/*.log]
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 100

exporters:
  # Send to Gonzo via gRPC
  otlp/gonzo_grpc:
    endpoint: localhost:4317
    tls:
      insecure: true
  
  # Or via HTTP
  otlphttp/gonzo_http:
    endpoint: http://localhost:4318/v1/logs

service:
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [batch]
      exporters: [otlp/gonzo_grpc]  # or otlphttp/gonzo_http

Start collector and Gonzo:

# Terminal 1: Start Gonzo
gonzo --otlp-enabled

# Terminal 2: Start collector
otelcol --config otel-collector-config.yaml

Application SDK Integration

Python Example:

from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource
import logging

# Create resource with service information
resource = Resource.create({
    "service.name": "my-python-app",
    "service.version": "1.0.0",
    "deployment.environment": "production"
})

# Configure OTLP exporter (gRPC)
exporter = OTLPLogExporter(
    endpoint="localhost:4317",
    insecure=True
)

# Set up logger provider
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(exporter)
)
_logs.set_logger_provider(logger_provider)

# Integrate with standard logging
handler = LoggingHandler(
    level=logging.NOTSET,
    logger_provider=logger_provider
)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Now logs go to Gonzo
logging.info("Application started")
logging.error("Something went wrong", extra={
    "user_id": 12345,
    "endpoint": "/api/users"
})

Go Example:

package main

import (
    "context"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    "go.opentelemetry.io/otel/log"
    "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func main() {
    // Create resource
    res, _ := resource.New(context.Background(),
        resource.WithAttributes(
            semconv.ServiceName("my-go-app"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )

    // Create OTLP exporter
    exporter, _ := otlploggrpc.New(context.Background(),
        otlploggrpc.WithEndpoint("localhost:4317"),
        otlploggrpc.WithInsecure(),
    )

    // Create logger provider
    provider := sdklog.NewLoggerProvider(
        sdklog.WithResource(res),
        sdklog.WithProcessor(sdklog.NewBatchProcessor(exporter)),
    )
    defer provider.Shutdown(context.Background())

    // Get logger
    logger := provider.Logger("app-logger")

    // Send logs
    logger.Emit(context.Background(), log.Record{
        Severity:  log.SeverityInfo,
        Body:      log.StringValue("Application started"),
    })
}

Node.js Example:

const { LoggerProvider } = require('@opentelemetry/sdk-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

// Create resource
const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'my-node-app',
  [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
});

// Create exporter
const exporter = new OTLPLogExporter({
  url: 'grpc://localhost:4317',
});

// Create logger provider
const loggerProvider = new LoggerProvider({ resource });
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(exporter)
);

// Get logger
const logger = loggerProvider.getLogger('app-logger');

// Send logs
logger.emit({
  severityText: 'INFO',
  body: 'Application started',
  attributes: {
    userId: 12345,
    endpoint: '/api/users',
  },
});

HTTP API Reference

POST /v1/logs

Request Format (JSON):

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "my-service" }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my-logger"
          },
          "logRecords": [
            {
              "timeUnixNano": "1705315805000000000",
              "severityNumber": 9,
              "severityText": "info",
              "body": {
                "stringValue": "Log message"
              },
              "attributes": [
                {
                  "key": "user_id",
                  "value": { "intValue": "12345" }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Response (Success):

{
  "partialSuccess": {}
}

Response (Partial Failure):

{
  "partialSuccess": {
    "rejectedLogRecords": 5,
    "errorMessage": "Some records were invalid"
  }
}

Status Codes:

• 200 OK - All logs accepted

• 206 Partial Content - Some logs rejected (see partialSuccess)

• 400 Bad Request - Invalid request format

• 500 Internal Server Error - Server error

Testing HTTP Endpoint

# Simple test
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [
      {
        "scopeLogs": [
          {
            "logRecords": [
              {
                "timeUnixNano": "1705315805000000000",
                "severityText": "info",
                "body": {"stringValue": "Test log"}
              }
            ]
          }
        ]
      }
    ]
  }'

# Should see log appear in Gonzo

gRPC API Reference

Service: opentelemetry.proto.collector.logs.v1.LogsService

Method: Export

Request: ExportLogsServiceRequest

Response: ExportLogsServiceResponse

Protocol Buffers Definition:

service LogsService {
  rpc Export(ExportLogsServiceRequest) 
    returns (ExportLogsServiceResponse) {}
}

message ExportLogsServiceRequest {
  repeated ResourceLogs resource_logs = 1;
}

message ExportLogsServiceResponse {
  ExportLogsPartialSuccess partial_success = 1;
}

message ExportLogsPartialSuccess {
  int64 rejected_log_records = 1;
  string error_message = 2;
}

Testing gRPC Endpoint

# Using grpcurl
grpcurl -plaintext \
  -d '{
    "resource_logs": [{
      "scope_logs": [{
        "log_records": [{
          "time_unix_nano": "1705315805000000000",
          "severity_text": "info",
          "body": {"string_value": "Test log"}
        }]
      }]
    }]
  }' \
  localhost:4317 \
  opentelemetry.proto.collector.logs.v1.LogsService/Export

Performance & Optimization

Batching

Gonzo processes logs in batches for efficiency:

Incoming Batches:

• OTLP senders should batch logs (recommended 100-1000 per batch)

• Reduces network overhead

• Improves throughput

Internal Processing:

• Logs processed individually after receipt

• Real-time display without buffering delay

Concurrency

gRPC Handler:

• Concurrent request handling

• One goroutine per connection

• Thread-safe log ingestion

HTTP Handler:

• Standard HTTP server concurrency

• Multiple simultaneous POST requests supported

Memory Management

Bounded Buffers:

• OTLP messages are streamed, not accumulated

• Converted logs go to main log buffer (configurable size)

• No unlimited memory growth

Large Messages:

• Default max receive message size: 4MB (configurable)

• Larger messages can be split by sender

# Increase max message size if needed
# (requires code modification currently)

Security Considerations

TLS/SSL

Production Recommendation: Always use TLS in production.

Current Implementation:

• gRPC: Insecure mode (no TLS)

• HTTP: Plain HTTP

For Production Use:

• Run Gonzo behind TLS-terminating proxy (nginx, envoy)

• Use OTLP collector with TLS, forward to Gonzo locally

Example with nginx:

server {
    listen 443 ssl http2;
    server_name gonzo.example.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    # gRPC proxy
    location / {
        grpc_pass grpc://localhost:4317;
    }
}

Authentication

Current Implementation: No built-in authentication

Recommendations:

• Use in trusted networks only

• Deploy behind authenticated proxy

• Use firewall rules to restrict access

• Consider VPN for remote access

Network Isolation

Best Practice: Don't expose OTLP ports to public internet

# Bind to localhost only (default)
gonzo --otlp-enabled  # Listens on 127.0.0.1

# For containers, use bridge networking
# For Kubernetes, use ClusterIP service

Troubleshooting OTLP Issues

Connection Refused

Symptom: Sender can't connect to Gonzo

Checks:

# Verify Gonzo is running with OTLP enabled
ps aux | grep gonzo

# Check ports are listening
lsof -i :4317  # gRPC
lsof -i :4318  # HTTP

# Test connectivity
telnet localhost 4317
curl http://localhost:4318/v1/logs

Logs Not Appearing

Symptom: Connection works but no logs in Gonzo

Diagnosis:

1. Check request format matches OTLP spec

2. Verify severityText or severityNumber is set

3. Check body field is present

4. Look for error responses from Gonzo

Test with minimal payload:

curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"body":{"stringValue":"test"}}]}]}]}'

Message Size Exceeded

Symptom: "message size exceeded" errors

Solution:

• Reduce batch size in sender

• Split large log messages

• Currently no config option (fixed at 4MB)

Performance Issues

Symptom: Slow log processing or high CPU

Solutions:

# Reduce update frequency
gonzo --otlp-enabled --update-interval=5s

# Reduce buffer sizes
gonzo --otlp-enabled --log-buffer=500

Compatibility

OTLP Version Support

• Specification Version: 1.0.0+

• Protocol Version: v1

• Backward Compatibility: Follows semantic versioning

OpenTelemetry SDK Compatibility

Tested with:

• ✅ OpenTelemetry Collector v0.90+

• ✅ Python SDK v1.20+

• ✅ Go SDK v1.20+

• ✅ Node.js SDK v0.45+

• ✅ Java SDK v1.32+

• ✅ .NET SDK v1.6+

Transport Support Matrix

Transport	Encoding	Status
gRPC	Protobuf	✅ Supported
HTTP	JSON	✅ Supported
HTTP	Protobuf	✅ Supported
WebSocket	-	❌ Not supported

Future Enhancements

Planned Features:

• TLS support (direct, without proxy)

• Authentication (API keys, mTLS)

• Compression (gzip, zstd)

• Configurable message size limits

• Metrics export (self-monitoring)

• Multiple OTLP receivers (different ports for different sources)

Resources

Specifications

• OTLP Specification

• OTLP Log Data Model

• Protocol Buffers

OpenTelemetry Documentation

• OpenTelemetry Logs

• Collector Configuration

• SDK Documentation

Related Gonzo Docs

• Architecture Overview

• Common Issues - OTLP

• Configuration Reference

Quick Start: gonzo --otlp-enabled starts both gRPC (4317) and HTTP (4318) receivers. Point your OTLP exporter to these endpoints and logs will appear in Gonzo immediately!