Grafana Loki Integration 🔥

Integrate Gonzo with Grafana Loki for powerful live log tailing and analysis. Use Loki as your log store and Gonzo as your real-time terminal lens.

Overview

Grafana Loki is a powerful log aggregation system designed for storing and querying logs at scale. While Loki excels at long-term retention and powerful queries via LogQL, live tailing logs for debugging and troubleshooting can be challenging. Gonzo fills this gap by providing a real-time, terminal-based interface for Loki logs.

Why use Gonzo with Loki?

• Debug in real time: Watch logs as code changes roll out

• Validate pipelines: Ensure OpenTelemetry agents ship logs correctly to Loki

• Shorten feedback loops: Catch issues instantly without switching tools

• Stay in-flow: Operate inside your terminal alongside kubectl, stern, and k9s

• AI-powered insights: Optional AI analysis of Loki logs

Integration Methods

Gonzo supports two primary methods for accessing Loki logs:

1. logcli - Using Loki's official CLI tool

2. Live Tail API - Direct connection to Loki's live tailing endpoint

Method 1: Using logcli

The easiest way to get started is using Loki's official logcli tool.

Install logcli

# macOS via Homebrew
brew install logcli

# Linux - download binary
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.0/logcli-linux-amd64.zip"
unzip "logcli-linux-amd64.zip"
chmod a+x logcli-linux-amd64
sudo mv logcli-linux-amd64 /usr/local/bin/logcli

# Verify installation
logcli --version

Configure logcli

Set up your Loki connection:

# Local Loki instance
export LOKI_ADDR=http://localhost:3100

# Remote Loki (with auth)
export LOKI_ADDR=https://loki.example.com
export LOKI_USERNAME=your-username
export LOKI_PASSWORD=your-password

# Or use Grafana Cloud
export LOKI_ADDR=https://logs-prod-us-central1.grafana.net
export LOKI_USERNAME=your-user-id
export LOKI_PASSWORD=your-api-key

Query and Tail with Gonzo

# Basic query with follow
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Filter by namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Multiple labels
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo

# With LogQL filters
logcli query --follow '{job="varlogs"} |= "error"' --output=jsonl | gonzo

# Time-based query
logcli query --since=1h '{app="myapp"}' --output=jsonl | gonzo

Common LogQL Queries

By application:

logcli query --follow '{app="nginx"}' --output=jsonl | gonzo

By severity:

logcli query --follow '{job="app"} |= "ERROR"' --output=jsonl | gonzo

Exclude patterns:

logcli query --follow '{app="api"} != "healthcheck"' --output=jsonl | gonzo

JSON parsing:

logcli query --follow '{app="api"} | json' --output=jsonl | gonzo

Regular expressions:

logcli query --follow '{app="web"} |~ "user.*login"' --output=jsonl | gonzo

Method 2: Live Tail API

For more direct integration, use Loki's Live Tail API with Gonzo's custom format support.

Set Up Custom Format

Gonzo includes a built-in Loki format. Download the format file to your config directory:

# Create formats directory
mkdir -p ~/.config/gonzo/formats

# Download Loki format (if not included)
# Or create custom format based on Loki's JSON structure

Live Tail API Connection

# Direct connection to Loki's live tail endpoint
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# With authentication
curl -s -u "$LOKI_USERNAME:$LOKI_PASSWORD" \
  "https://loki.example.com/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# Multiple labels
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"api\",env=\"prod\"}" | gonzo --format=loki-stream

Using a Shell Function

Create a convenient shell function for repeated use:

# Add to ~/.bashrc or ~/.zshrc
loki-tail() {
  local query="${1:-{app=\"myapp\"}}"
  logcli query --follow "$query" --output=jsonl | gonzo
}

# Usage
loki-tail '{namespace="production"}'
loki-tail '{app="api",level="error"}'

Loki Log Format

Loki outputs logs in a specific JSON structure. Gonzo's Loki format handles:

{
  "stream": {
    "app": "my-application",
    "namespace": "production",
    "pod": "my-app-abc123",
    "container": "app"
  },
  "values": [
    [
      "1705315805000000000",
      "2024-01-15T10:30:05Z ERROR Failed to connect to database"
    ]
  ]
}

Gonzo extracts:

• Labels (stream) → Attributes panel

• Timestamp → Log timestamp

• Log line → Message

• Severity → Detected from message

Kubernetes + Loki Workflow

Combine Loki with Kubernetes for powerful log analysis:

1. Query Kubernetes Logs in Loki

# All pods in namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Specific deployment
logcli query --follow '{namespace="prod",app="api"}' --output=jsonl | gonzo

# With error filter
logcli query --follow '{namespace="prod"} |= "ERROR"' --output=jsonl | gonzo

2. Compare with kubectl

Use both tools side by side:

# Terminal 1: Live kubectl logs
kubectl logs -f deployment/my-app | gonzo

# Terminal 2: Historical Loki logs
logcli query --since=1h '{app="my-app"}' --output=jsonl | gonzo

3. Integration with Stern

For multi-pod analysis:

# Stern for live
stern --namespace production api --output json | gonzo

# Loki for historical
logcli query --since=1h '{namespace="production",app="api"}' --output=jsonl | gonzo

Advanced Use Cases

Pipeline Validation

Verify that logs are flowing correctly into Loki:

# Check recent logs
logcli query --since=5m '{app="myapp"}' --output=jsonl | gonzo

# Verify all expected labels exist
logcli labels | grep app
logcli label app | grep myapp

Pattern Detection

Use Gonzo's pattern detection on Loki logs:

# Load logs and analyze patterns
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Press Enter on "Counts" panel to see:
# - Pattern extraction
# - Error clustering
# - Service distribution

AI Analysis

Combine Loki queries with AI analysis:

# Set up AI
export OPENAI_API_KEY="sk-your-key"

# Query and analyze
logcli query --follow '{app="api"} |= "ERROR"' --output=jsonl | gonzo --ai-model="gpt-4"

# In Gonzo:
# 1. Navigate to an error
# 2. Press Enter for details
# 3. Press 'i' for AI analysis

Time-Range Analysis

Analyze specific time periods:

# Last hour
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Specific time range
logcli query --from="2024-01-15T10:00:00Z" --to="2024-01-15T11:00:00Z" \
  '{app="api"}' --output=jsonl | gonzo

# Today's errors
logcli query --since=0h '{app="api"} |= "ERROR"' --output=jsonl | gonzo

Configuration Examples

Persistent Configuration

Create a config for Loki integration:

# ~/.config/gonzo/loki-config.yml
update-interval: 1s
log-buffer: 5000
memory-size: 20000
skin: dracula

# AI analysis for errors
ai-model: "gpt-4"

Use with:

logcli query --follow '{app="api"}' --output=jsonl | gonzo --config ~/.config/gonzo/loki-config.yml

Shell Aliases

Add to ~/.bashrc or ~/.zshrc:

# Loki + Gonzo aliases
alias loki-prod='logcli query --follow '"'"'{env="production"}'"'"' --output=jsonl | gonzo'
alias loki-errors='logcli query --follow '"'"'{level="error"}'"'"' --output=jsonl | gonzo'
alias loki-api='logcli query --follow '"'"'{app="api"}'"'"' --output=jsonl | gonzo'

# With AI
alias loki-ai='logcli query --follow --output=jsonl | gonzo --ai-model="gpt-4"'

Troubleshooting

logcli Not Connecting

Check Loki address:

echo $LOKI_ADDR
curl $LOKI_ADDR/ready

Test connection:

logcli labels
logcli query --limit=5 '{job="varlogs"}'

No Logs Appearing in Gonzo

Verify logcli output:

# Test without Gonzo first
logcli query --follow '{app="myapp"}' --output=jsonl

# Check format
logcli query --limit=1 '{app="myapp"}' --output=jsonl | jq .

Check Gonzo format:

# Ensure using jsonl output
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Not json (wrong format)
logcli query --follow '{app="myapp"}' --output=json | gonzo

Authentication Issues

Grafana Cloud:

# Ensure credentials are set
echo $LOKI_USERNAME
echo $LOKI_PASSWORD

# Test auth
logcli labels

TLS Issues:

# Skip TLS verify for testing (not for production)
export LOKI_TLS_SKIP_VERIFY=true

Performance Issues

Reduce query scope:

# Add more specific labels
logcli query --follow '{app="api",namespace="prod"}' --output=jsonl | gonzo

# Limit throughput
logcli query --follow --limit=100 '{app="api"}' --output=jsonl | gonzo

Adjust Gonzo buffers:

logcli query --follow '{app="api"}' --output=jsonl | \
  gonzo --log-buffer=10000 --update-interval=2s

Best Practices

1. Use Specific Label Selectors

# ❌ Too broad
logcli query --follow '{}' --output=jsonl | gonzo

# ✅ Specific
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo

2. Combine with Filters

# Filter in LogQL for efficiency
logcli query --follow '{app="api"} |= "ERROR" != "healthcheck"' --output=jsonl | gonzo

3. Leverage Gonzo's Features

• Pause (Space) during investigation

• Filter (/) to narrow further

• AI Analysis (i) for complex errors

• Patterns (Counts modal) for trends

4. Save Common Queries

# Create query scripts
cat > ~/loki-queries/prod-errors.sh << 'EOF'
#!/bin/bash
logcli query --follow '{env="production"} |= "ERROR"' --output=jsonl | gonzo
EOF

chmod +x ~/loki-queries/prod-errors.sh

Complete Tutorial

For a comprehensive walkthrough of using Gonzo with Loki, including:

• Detailed setup instructions

• Real-world examples

• Advanced LogQL patterns

• Custom format configuration

• Integration patterns

Read the full guide: Live Tailing Grafana Loki Logs with Gonzo

Related Resources

Documentation

• Custom Formats - Creating format parsers

• Kubernetes Integration - K8s log tailing

• AI Integration - AI-powered analysis

External Links

• Grafana Loki Documentation

• logcli Documentation

• LogQL Guide

Blog Posts

• Live Tailing Grafana Loki Logs with Gonzo

• Using Gonzo with k9s and Stern

Support

Having issues with Loki integration?

• Check Troubleshooting Guide

• Ask in GitHub Discussions

• Report bugs in GitHub Issues

Pro Tip: Use logcli with --output=jsonl (JSON Lines) for best compatibility with Gonzo. Each log entry is a separate JSON object on its own line.

Learn More: Read the complete tutorial on Live Tailing Grafana Loki Logs with Gonzo for advanced patterns and real-world examples.