ControlTheory Website

VictoriaLogs 🪵

Stream and analyze logs directly from VictoriaLogs with Gonzo's native integration. Leverage VictoriaLogs' high-performance log storage with Gonzo's powerful analysis capabilities, including AI insights and pattern detection.

Native Integration: Gonzo has built-in VictoriaLogs support with custom field parsing that properly maps VictoriaLogs format to Gonzo's internal log representation.

Why Use Gonzo with VictoriaLogs?

VictoriaLogs is a high-performance, cost-effective log database. Combining it with Gonzo provides:

  • 🚀 High-Performance Streaming - Direct streaming from VictoriaLogs without intermediaries

  • 🤖 AI-Powered Analysis - Intelligent insights on your stored logs

  • 📊 Pattern Detection - Automatic clustering of similar log entries

  • 🔍 LogsQL Queries - Powerful query language for precise log filtering

  • 🏷️ Smart Field Mapping - Automatic mapping of VictoriaLogs fields to Gonzo attributes

  • ⚡ Real-Time Analysis - Live streaming and analysis of logs as they arrive

Features

Custom Field Parsing

Gonzo automatically maps VictoriaLogs fields to provide optimal display and analysis:

VictoriaLogs Field
Gonzo Mapping
Description

_msg

Log Body

Displayed as main message (not raw JSON)

_time

Timestamp

Parsed as log timestamp

_stream

Attribute

Stream identifier

_stream_id

Attribute

Stream ID

All other fields

Attributes

Available in Attributes panel

Special Field Mappings

Host Detection - Checked in priority order:

  1. k8s.node.name (Kubernetes node)

  2. kubernetes.pod_node_name (Alternative K8s format)

  3. kubernetes_pod_node_name (Underscore format)

Severity Detection - Checked in priority order:

  • level

  • severity

  • log.level

  • log_level

  • loglevel

  • levelname

Format Transformation Example

VictoriaLogs JSON Input:

{
  "_msg": "Database connection established",
  "_stream": "app-logs",
  "_stream_id": "stream-001",
  "_time": "2024-01-15T10:30:46.456Z",
  "level": "INFO",
  "k8s.node.name": "node-02",
  "service": "database",
  "pool_size": 10
}

Gonzo Display:

  • Message: "Database connection established" ✅ (clean message, not raw JSON)

  • Severity: INFO (with appropriate color coding)

  • Timestamp: 2024-01-15T10:30:46.456Z

  • Host: node-02 (mapped from k8s.node.name)

  • Attributes Panel:

    • service = "database"

    • pool_size = 10

    • _stream = "app-logs"

    • _stream_id = "stream-001"

Configuration

Command Line Options

# Required
--vmlogs-url        # Victoria Logs URL endpoint

# Optional
--vmlogs-user       # Basic auth username
--vmlogs-password   # Basic auth password
--vmlogs-query      # LogsQL query (default: "*")

Environment Variables

# Set VictoriaLogs connection parameters
export GONZO_VMLOGS_URL="http://localhost:9428"
export GONZO_VMLOGS_USER="myuser"
export GONZO_VMLOGS_PASSWORD="mypass"
export GONZO_VMLOGS_QUERY="service:'my-app'"

# Then run Gonzo
gonzo

Configuration File

# ~/.config/gonzo/victorialogs.yml
vmlogs-url: "http://localhost:9428"
vmlogs-user: "myuser"
vmlogs-password: "mypass"
vmlogs-query: "*"

# Performance settings for VictoriaLogs
update-interval: 2s
log-buffer: 20000
memory-size: 100000

# AI configuration
ai-model: "gpt-4"

Usage:

gonzo --config ~/.config/gonzo/victorialogs.yml

Usage Examples

Basic Streaming

# Stream all logs from VictoriaLogs
gonzo --vmlogs-url="http://localhost:9428" --vmlogs-query="*"

# Stream with AI analysis
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query="*" \
      --ai-model="gpt-4"

# Stream with custom configuration
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query="*" \
      --log-buffer=20000 \
      --update-interval=2s

Authenticated Access

# Basic authentication
gonzo --vmlogs-url="https://vmlogs.example.com" \
      --vmlogs-user="myuser" \
      --vmlogs-password="mypass" \
      --vmlogs-query="*"

# With specific query
gonzo --vmlogs-url="https://vmlogs.example.com" \
      --vmlogs-user="myuser" \
      --vmlogs-password="mypass" \
      --vmlogs-query='level:error'

# Using environment variables for credentials
export GONZO_VMLOGS_USER="myuser"
export GONZO_VMLOGS_PASSWORD="mypass"
gonzo --vmlogs-url="https://vmlogs.example.com" \
      --vmlogs-query='level:error'

LogsQL Query Examples

# Specific service
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"payment-processor"'

# Service with severity
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"payment-processor" AND level:error'

# Multiple services
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:("api" OR "worker")'

Real-World Integration Patterns

Production Monitoring

# Monitor production errors with AI
gonzo --vmlogs-url="http://vmlogs.prod.company.com" \
      --vmlogs-user="$PROD_USER" \
      --vmlogs-password="$PROD_PASS" \
      --vmlogs-query='kubernetes.namespace:"production" AND level:error' \
      --ai-model="gpt-4"

# Multi-service monitoring
gonzo --vmlogs-url="http://vmlogs.prod.company.com" \
      --vmlogs-query='service:("api" OR "worker" OR "scheduler") AND level:(error OR warning)' \
      --log-buffer=20000

Kubernetes Log Analysis

# Specific namespace
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='kubernetes.namespace:"production"'

# Pod pattern matching
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='kubernetes.pod_name:*backend*'

# Node-specific logs
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='k8s.node.name:"node-01" AND level:error'

# Container-specific analysis
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='kubernetes.container_name:"nginx" AND level:warning'

Time-Based Analysis

# Recent logs (VictoriaLogs handles time filtering)
# Use VictoriaLogs Web UI or API to export specific time ranges
# Then analyze with Gonzo

# Stream recent logs
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='_time:>now-1h'

# Specific time range (if supported by your VictoriaLogs version)
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='_time:>2024-01-15T10:00:00Z AND _time:<2024-01-15T11:00:00Z'

Performance Troubleshooting

# High-duration operations
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='duration:>1000'  # Duration > 1000ms

# Slow queries
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='_msg:*slow* AND service:"database"'

# Memory warnings
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='_msg:(*memory* OR *oom*) AND level:warning'

Implementation Details

VictoriaLogs Client Architecture

Gonzo's VictoriaLogs integration includes:

  1. Streaming Client (internal/vmlogs/client.go)

    • Connects to VictoriaLogs /select/logsql/tail endpoint

    • Maintains persistent streaming connection

    • Handles reconnection on failures

  2. Format Converter

    • Transforms VictoriaLogs JSON to OTLP format

    • Preserves all field information

    • Optimizes for Gonzo display

  3. Field Mapper

    • Handles special fields (_msg, _time, etc.)

    • Maps host fields in priority order

    • Extracts all attributes automatically

  4. Severity Detector

    • Identifies log levels from various field names

    • Provides consistent severity detection

    • Enables proper color coding

Streaming Endpoint

Gonzo uses the VictoriaLogs streaming tail endpoint:

http://victorialogs:9428/select/logsql/tail?query=<logsql_query>

This provides:

  • Real-time log streaming

  • Efficient resource usage

  • Native LogsQL query support

  • Automatic format conversion

Configuration Optimization

High-Performance Setup

# ~/.config/gonzo/vmlogs-production.yml
vmlogs-url: "http://vmlogs.company.com:9428"
vmlogs-user: "production-reader"
vmlogs-password: "${VMLOGS_PASSWORD}"  # From environment
vmlogs-query: "kubernetes.namespace:production AND level:(error OR warning)"

# Optimized for high volume
update-interval: 5s
log-buffer: 50000
memory-size: 200000

# Best AI for production
ai-model: "gpt-4"
ai-context-size: 8000

Development Setup

# ~/.config/gonzo/vmlogs-dev.yml
vmlogs-url: "http://localhost:9428"
vmlogs-query: "*"  # All logs in development

# Fast updates for development
update-interval: 1s
log-buffer: 5000
memory-size: 20000

# Cost-effective AI
ai-model: "gpt-3.5-turbo"

Best Practices

🎯 Efficient Querying

# ✅ Good: Specific queries
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"api" AND level:error'

# ❌ Avoid: Overly broad queries
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='*'  # All logs (high volume)

# ✅ Good: Use field filters
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='kubernetes.namespace:"production"'

# ✅ Good: Combine multiple filters
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"api" AND level:error AND kubernetes.namespace:"production"'

🔒 Security Best Practices

# Use environment variables for credentials
export GONZO_VMLOGS_USER="readonly-user"
export GONZO_VMLOGS_PASSWORD="secure-password"

# Don't hardcode passwords in commands
gonzo --vmlogs-url="https://vmlogs.company.com"  # Credentials from env

# Use HTTPS for remote connections
gonzo --vmlogs-url="https://vmlogs.company.com" \
      --vmlogs-query='level:error'

# Least privilege: Use read-only credentials
# Create VictoriaLogs user with SELECT-only permissions

⚡ Performance Optimization

# For high-volume logs, adjust buffers
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='kubernetes.namespace:"production"' \
      --log-buffer=50000 \
      --update-interval=10s

# Use specific queries to reduce volume
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='level:(error OR warning)'  # Less volume

# Filter at source when possible
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"critical-api" AND level:error'

Troubleshooting

Connection Issues

# Test VictoriaLogs connectivity
curl "http://localhost:9428/select/logsql/query?query=*&limit=10"

# Verify authentication
curl -u myuser:mypass "http://localhost:9428/select/logsql/query?query=*&limit=10"

# Check VictoriaLogs is running
curl "http://localhost:9428/health"

# Test with verbose output
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query="*" \
      --verbose

Query Issues

# Test LogsQL query directly
curl "http://localhost:9428/select/logsql/query?query=level:error&limit=10"

# Verify field names
curl "http://localhost:9428/select/logsql/query?query=*&limit=1" | jq .

# Check query syntax in VictoriaLogs docs
# https://docs.victoriametrics.com/VictoriaLogs/LogsQL.html

# Start with simple query
gonzo --vmlogs-url="http://localhost:9428" --vmlogs-query="*"

# Then add complexity
gonzo --vmlogs-url="http://localhost:9428" --vmlogs-query="level:error"

Performance Issues

# Reduce query scope
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"specific-service"'

# Increase update interval
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query="*" \
      --update-interval=10s

# Increase buffer sizes
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query="*" \
      --log-buffer=50000 \
      --memory-size=100000

Field Mapping Issues

# Check raw VictoriaLogs output
curl "http://localhost:9428/select/logsql/query?query=*&limit=5" | jq .

# Verify field names in Gonzo
# Press Tab to navigate to Attributes panel
# Check if expected fields are present

# Ensure using JSON output (default for VictoriaLogs)
# Gonzo automatically handles VictoriaLogs JSON format

What's Next?

Now that you've mastered VictoriaLogs integration, explore related topics:

  • Configuration - Optimize for VictoriaLogs workflows

  • AI Integration - Enhanced log analysis with AI

  • Advanced Features - Pattern detection in VictoriaLogs data

  • Kubernetes Integration - Combine with K8s log analysis

Or try these advanced VictoriaLogs patterns:

# Multi-environment monitoring
gonzo --vmlogs-url="http://vmlogs.company.com" \
      --vmlogs-query='kubernetes.namespace:("production" OR "staging") AND level:error' \
      --ai-model="gpt-4"

# Service mesh analysis
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='service:"istio-proxy" AND level:warning'

# Custom field analysis
gonzo --vmlogs-url="http://localhost:9428" \
      --vmlogs-query='response_time:>1000 AND service:"api"'

Unlock powerful log analysis with VictoriaLogs and Gonzo! 📊 Native integration, intelligent field mapping, and AI-powered insights make VictoriaLogs logs easy to analyze and understand.

PreviousVercel Logs Integration ▲NextBlogs, Resources & Videos

Last updated