# OTLP Protocol Details

Deep dive into Gonzo's OpenTelemetry Protocol (OTLP) implementation for receiving and processing logs.

### Overview

Gonzo implements a complete OTLP logs receiver supporting both gRPC and HTTP transports. This allows Gonzo to receive logs directly from OpenTelemetry SDKs, collectors, and other OTLP-compatible sources.

### OTLP Specification

Gonzo follows the [OpenTelemetry Protocol Specification](https://opentelemetry.io/docs/specs/otlp/) for logs:

* **Version**: OTLP 1.0.0+
* **Transports**: gRPC and HTTP
* **Encoding**: Protocol Buffers (gRPC), JSON (HTTP)
* **Endpoints**:
  * gRPC: `localhost:4317`
  * HTTP: `http://localhost:4318/v1/logs`

### Architecture

```
┌─────────────────────────────────────────┐
│         OTLP Client (Sender)            │
│    (App, Collector, SDK, etc)           │
└──────────────┬──────────────────────────┘
               │
               ├──────────┐
               │          │
           gRPC:4317   HTTP:4318
               │          │
    ┌──────────▼────┐  ┌──▼──────────┐
    │  gRPC Server  │  │ HTTP Server │
    │               │  │             │
    └──────┬────────┘  └─┬───────────┘
           │             │
           └─────┬───────┘
                 │
        ┌────────▼─────────┐
        │  OTLP Processor  │
        │  - Decode proto  │
        │  - Extract logs  │
        │  - Map fields    │
        └────────┬─────────┘
                 │
        ┌────────▼─────────┐
        │  Log Converter   │
        │  - Resource attr │
        │  - Log record    │
        │  - Timestamps    │
        └────────┬─────────┘
                 │
        ┌────────▼─────────┐
        │   Analyzer       │
        │   Engine         │
        └──────────────────┘
```

### Transport Details

#### gRPC Transport

**Endpoint**: `localhost:4317`

**Protocol**: HTTP/2 with Protocol Buffers

**Service Definition**:

```protobuf
service LogsService {
  rpc Export(ExportLogsServiceRequest) 
    returns (ExportLogsServiceResponse) {}
}
```

**Features**:

* Bidirectional streaming support
* Built-in compression (gzip)
* Connection multiplexing
* Automatic reconnection

**Starting gRPC Receiver**:

```bash
# Default port 4317
gonzo --otlp-enabled

# Custom port
gonzo --otlp-enabled --otlp-grpc-port=5317
```

#### HTTP Transport

**Endpoint**: `http://localhost:4318/v1/logs`

**Protocol**: HTTP/1.1 or HTTP/2

**Method**: POST

**Content-Type**: `application/json` or `application/x-protobuf`

**Features**:

* Standard HTTP clients work
* JSON encoding (more debuggable)
* No special libraries required
* Works through HTTP proxies

**Starting HTTP Receiver**:

```bash
# Default port 4318
gonzo --otlp-enabled

# Custom port
gonzo --otlp-enabled --otlp-http-port=5318
```

### OTLP Log Data Model

#### Structure Hierarchy

```
ExportLogsServiceRequest
└── ResourceLogs[]
    ├── Resource
    │   └── Attributes[]
    │       ├── service.name
    │       ├── host.name
    │       └── ... (resource attributes)
    └── ScopeLogs[]
        ├── Scope
        │   ├── name
        │   └── version
        └── LogRecords[]
            ├── timeUnixNano
            ├── observedTimeUnixNano
            ├── severityNumber
            ├── severityText
            ├── body (AnyValue)
            ├── attributes[]
            └── traceId/spanId (optional)
```

#### Resource Attributes

Resource attributes describe the source of logs:

```json
{
  "service.name": "payment-api",
  "service.version": "1.2.3",
  "host.name": "prod-server-01",
  "deployment.environment": "production",
  "cloud.provider": "aws",
  "cloud.region": "us-east-1"
}
```

**How Gonzo Uses Resource Attributes**:

* Merged with log record attributes
* Displayed in Attributes panel
* Available for filtering
* Shown in log details

#### Log Record

Individual log entry:

```json
{
  "timeUnixNano": "1705315805000000000",
  "observedTimeUnixNano": "1705315805123456789",
  "severityNumber": 13,  // ERROR
  "severityText": "error",
  "body": {
    "stringValue": "Database connection failed"
  },
  "attributes": [
    {"key": "user_id", "value": {"intValue": "12345"}},
    {"key": "endpoint", "value": {"stringValue": "/api/users"}},
    {"key": "error.type", "value": {"stringValue": "timeout"}}
  ],
  "traceId": "1234567890abcdef",
  "spanId": "1234567890abcdef"
}
```

**Gonzo Mapping**:

```
timeUnixNano        → Timestamp
severityText        → Level (error, warn, info, debug)
body                → Message
attributes          → Extracted to Attributes panel
traceId/spanId      → Available for tracing correlation
```

#### Severity Levels

OTLP defines numerical severity levels:

| Number | Name  | Gonzo Display |
| ------ | ----- | ------------- |
| 1-4    | TRACE | ⚪ White       |
| 5-8    | DEBUG | 🔵 Blue       |
| 9-12   | INFO  | 🟢 Green      |
| 13-16  | WARN  | 🟡 Yellow     |
| 17-20  | ERROR | 🔴 Red        |
| 21-24  | FATAL | 🔴 Red        |

Gonzo reads both `severityNumber` and `severityText` fields.

#### Body Types

The log body can be various types:

**String** (most common):

```json
{
  "body": {
    "stringValue": "User logged in"
  }
}
```

**Structured**:

```json
{
  "body": {
    "kvlistValue": {
      "values": [
        {"key": "event", "value": {"stringValue": "login"}},
        {"key": "user_id", "value": {"intValue": 12345}}
      ]
    }
  }
}
```

Gonzo handles all OTLP value types and displays them appropriately.

### Configuration Examples

#### OpenTelemetry Collector

**Configure collector to send to Gonzo**:

```yaml
# otel-collector-config.yaml
receivers:
  filelog:
    include: [/var/log/app/*.log]
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 100

exporters:
  # Send to Gonzo via gRPC
  otlp/gonzo_grpc:
    endpoint: localhost:4317
    tls:
      insecure: true
  
  # Or via HTTP
  otlphttp/gonzo_http:
    endpoint: http://localhost:4318/v1/logs

service:
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [batch]
      exporters: [otlp/gonzo_grpc]  # or otlphttp/gonzo_http
```

**Start collector and Gonzo**:

```bash
# Terminal 1: Start Gonzo
gonzo --otlp-enabled

# Terminal 2: Start collector
otelcol --config otel-collector-config.yaml
```

#### Application SDK Integration

**Python Example**:

```python
from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource
import logging

# Create resource with service information
resource = Resource.create({
    "service.name": "my-python-app",
    "service.version": "1.0.0",
    "deployment.environment": "production"
})

# Configure OTLP exporter (gRPC)
exporter = OTLPLogExporter(
    endpoint="localhost:4317",
    insecure=True
)

# Set up logger provider
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(exporter)
)
_logs.set_logger_provider(logger_provider)

# Integrate with standard logging
handler = LoggingHandler(
    level=logging.NOTSET,
    logger_provider=logger_provider
)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Now logs go to Gonzo
logging.info("Application started")
logging.error("Something went wrong", extra={
    "user_id": 12345,
    "endpoint": "/api/users"
})
```

**Go Example**:

```go
package main

import (
    "context"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    "go.opentelemetry.io/otel/log"
    "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func main() {
    // Create resource
    res, _ := resource.New(context.Background(),
        resource.WithAttributes(
            semconv.ServiceName("my-go-app"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )

    // Create OTLP exporter
    exporter, _ := otlploggrpc.New(context.Background(),
        otlploggrpc.WithEndpoint("localhost:4317"),
        otlploggrpc.WithInsecure(),
    )

    // Create logger provider
    provider := sdklog.NewLoggerProvider(
        sdklog.WithResource(res),
        sdklog.WithProcessor(sdklog.NewBatchProcessor(exporter)),
    )
    defer provider.Shutdown(context.Background())

    // Get logger
    logger := provider.Logger("app-logger")

    // Send logs
    logger.Emit(context.Background(), log.Record{
        Severity:  log.SeverityInfo,
        Body:      log.StringValue("Application started"),
    })
}
```

**Node.js Example**:

```javascript
const { LoggerProvider } = require('@opentelemetry/sdk-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

// Create resource
const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'my-node-app',
  [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
});

// Create exporter
const exporter = new OTLPLogExporter({
  url: 'grpc://localhost:4317',
});

// Create logger provider
const loggerProvider = new LoggerProvider({ resource });
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(exporter)
);

// Get logger
const logger = loggerProvider.getLogger('app-logger');

// Send logs
logger.emit({
  severityText: 'INFO',
  body: 'Application started',
  attributes: {
    userId: 12345,
    endpoint: '/api/users',
  },
});
```

### HTTP API Reference

#### POST /v1/logs

**Request Format** (JSON):

```json
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "my-service" }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {
            "name": "my-logger"
          },
          "logRecords": [
            {
              "timeUnixNano": "1705315805000000000",
              "severityNumber": 9,
              "severityText": "info",
              "body": {
                "stringValue": "Log message"
              },
              "attributes": [
                {
                  "key": "user_id",
                  "value": { "intValue": "12345" }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}
```

**Response** (Success):

```json
{
  "partialSuccess": {}
}
```

**Response** (Partial Failure):

```json
{
  "partialSuccess": {
    "rejectedLogRecords": 5,
    "errorMessage": "Some records were invalid"
  }
}
```

**Status Codes**:

* `200 OK` - All logs accepted
* `206 Partial Content` - Some logs rejected (see partialSuccess)
* `400 Bad Request` - Invalid request format
* `500 Internal Server Error` - Server error

#### Testing HTTP Endpoint

```bash
# Simple test
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [
      {
        "scopeLogs": [
          {
            "logRecords": [
              {
                "timeUnixNano": "1705315805000000000",
                "severityText": "info",
                "body": {"stringValue": "Test log"}
              }
            ]
          }
        ]
      }
    ]
  }'

# Should see log appear in Gonzo
```

### gRPC API Reference

#### Service: opentelemetry.proto.collector.logs.v1.LogsService

**Method**: Export

**Request**: ExportLogsServiceRequest

**Response**: ExportLogsServiceResponse

**Protocol Buffers Definition**:

```protobuf
service LogsService {
  rpc Export(ExportLogsServiceRequest) 
    returns (ExportLogsServiceResponse) {}
}

message ExportLogsServiceRequest {
  repeated ResourceLogs resource_logs = 1;
}

message ExportLogsServiceResponse {
  ExportLogsPartialSuccess partial_success = 1;
}

message ExportLogsPartialSuccess {
  int64 rejected_log_records = 1;
  string error_message = 2;
}
```

#### Testing gRPC Endpoint

```bash
# Using grpcurl
grpcurl -plaintext \
  -d '{
    "resource_logs": [{
      "scope_logs": [{
        "log_records": [{
          "time_unix_nano": "1705315805000000000",
          "severity_text": "info",
          "body": {"string_value": "Test log"}
        }]
      }]
    }]
  }' \
  localhost:4317 \
  opentelemetry.proto.collector.logs.v1.LogsService/Export
```

### Performance & Optimization

#### Batching

Gonzo processes logs in batches for efficiency:

**Incoming Batches**:

* OTLP senders should batch logs (recommended 100-1000 per batch)
* Reduces network overhead
* Improves throughput

**Internal Processing**:

* Logs processed individually after receipt
* Real-time display without buffering delay

#### Concurrency

**gRPC Handler**:

* Concurrent request handling
* One goroutine per connection
* Thread-safe log ingestion

**HTTP Handler**:

* Standard HTTP server concurrency
* Multiple simultaneous POST requests supported

#### Memory Management

**Bounded Buffers**:

* OTLP messages are streamed, not accumulated
* Converted logs go to main log buffer (configurable size)
* No unlimited memory growth

**Large Messages**:

* Default max receive message size: 4MB (configurable)
* Larger messages can be split by sender

```bash
# Increase max message size if needed
# (requires code modification currently)
```

### Security Considerations

#### TLS/SSL

**Production Recommendation**: Always use TLS in production.

**Current Implementation**:

* gRPC: Insecure mode (no TLS)
* HTTP: Plain HTTP

**For Production Use**:

* Run Gonzo behind TLS-terminating proxy (nginx, envoy)
* Use OTLP collector with TLS, forward to Gonzo locally

**Example with nginx**:

```nginx
server {
    listen 443 ssl http2;
    server_name gonzo.example.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    # gRPC proxy
    location / {
        grpc_pass grpc://localhost:4317;
    }
}
```

#### Authentication

**Current Implementation**: No built-in authentication

**Recommendations**:

* Use in trusted networks only
* Deploy behind authenticated proxy
* Use firewall rules to restrict access
* Consider VPN for remote access

#### Network Isolation

**Best Practice**: Don't expose OTLP ports to public internet

```bash
# Bind to localhost only (default)
gonzo --otlp-enabled  # Listens on 127.0.0.1

# For containers, use bridge networking
# For Kubernetes, use ClusterIP service
```

### Troubleshooting OTLP Issues

#### Connection Refused

**Symptom**: Sender can't connect to Gonzo

**Checks**:

```bash
# Verify Gonzo is running with OTLP enabled
ps aux | grep gonzo

# Check ports are listening
lsof -i :4317  # gRPC
lsof -i :4318  # HTTP

# Test connectivity
telnet localhost 4317
curl http://localhost:4318/v1/logs
```

#### Logs Not Appearing

**Symptom**: Connection works but no logs in Gonzo

**Diagnosis**:

1. Check request format matches OTLP spec
2. Verify severityText or severityNumber is set
3. Check body field is present
4. Look for error responses from Gonzo

**Test with minimal payload**:

```bash
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{"resourceLogs":[{"scopeLogs":[{"logRecords":[{"body":{"stringValue":"test"}}]}]}]}'
```

#### Message Size Exceeded

**Symptom**: "message size exceeded" errors

**Solution**:

* Reduce batch size in sender
* Split large log messages
* Currently no config option (fixed at 4MB)

#### Performance Issues

**Symptom**: Slow log processing or high CPU

**Solutions**:

```bash
# Reduce update frequency
gonzo --otlp-enabled --update-interval=5s

# Reduce buffer sizes
gonzo --otlp-enabled --log-buffer=500
```

### Compatibility

#### OTLP Version Support

* **Specification Version**: 1.0.0+
* **Protocol Version**: v1
* **Backward Compatibility**: Follows semantic versioning

#### OpenTelemetry SDK Compatibility

Tested with:

* ✅ OpenTelemetry Collector v0.90+
* ✅ Python SDK v1.20+
* ✅ Go SDK v1.20+
* ✅ Node.js SDK v0.45+
* ✅ Java SDK v1.32+
* ✅ .NET SDK v1.6+

#### Transport Support Matrix

| Transport | Encoding | Status          |
| --------- | -------- | --------------- |
| gRPC      | Protobuf | ✅ Supported     |
| HTTP      | JSON     | ✅ Supported     |
| HTTP      | Protobuf | ✅ Supported     |
| WebSocket | -        | ❌ Not supported |

### Future Enhancements

**Planned Features**:

* TLS support (direct, without proxy)
* Authentication (API keys, mTLS)
* Compression (gzip, zstd)
* Configurable message size limits
* Metrics export (self-monitoring)
* Multiple OTLP receivers (different ports for different sources)

### Resources

#### Specifications

* [OTLP Specification](https://opentelemetry.io/docs/specs/otlp/)
* [OTLP Log Data Model](https://opentelemetry.io/docs/specs/otel/logs/data-model/)
* [Protocol Buffers](https://protobuf.dev/)

#### OpenTelemetry Documentation

* [OpenTelemetry Logs](https://opentelemetry.io/docs/concepts/signals/logs/)
* [Collector Configuration](https://opentelemetry.io/docs/collector/configuration/)
* [SDK Documentation](https://opentelemetry.io/docs/instrumentation/)

#### Related Gonzo Docs

* Architecture Overview
* Common Issues - OTLP
* Configuration Reference

{% hint style="success" %}
**Quick Start**: `gonzo --otlp-enabled` starts both gRPC (4317) and HTTP (4318) receivers. Point your OTLP exporter to these endpoints and logs will appear in Gonzo immediately!
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.controltheory.com/backup/api-and-architecture/otlp-protocol-details.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.