Grafana Loki Integration 🔥

Integrate Gonzo with Grafana Loki for powerful live log tailing and analysis. Use Loki as your log store and Gonzo as your real-time terminal lens.

Overview

Grafana Loki is a powerful log aggregation system designed for storing and querying logs at scale. While Loki excels at long-term retention and powerful queries via LogQL, live tailing logs for debugging and troubleshooting can be challenging. Gonzo fills this gap by providing a real-time, terminal-based interface for Loki logs.

Why use Gonzo with Loki?

Debug in real time: Watch logs as code changes roll out
Validate pipelines: Ensure OpenTelemetry agents ship logs correctly to Loki
Shorten feedback loops: Catch issues instantly without switching tools
Stay in-flow: Operate inside your terminal alongside kubectl, stern, and k9s
AI-powered insights: Optional AI analysis of Loki logs

Integration Methods

Gonzo supports two primary methods for accessing Loki logs:

logcli - Using Loki's official CLI tool
Live Tail API - Direct connection to Loki's live tailing endpoint

Method 1: Using logcli

The easiest way to get started is using Loki's official logcli tool.

Install logcli

# macOS via Homebrew
brew install logcli

# Linux - download binary
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.0/logcli-linux-amd64.zip"
unzip "logcli-linux-amd64.zip"
chmod a+x logcli-linux-amd64
sudo mv logcli-linux-amd64 /usr/local/bin/logcli

# Verify installation
logcli --version

Configure logcli

Set up your Loki connection:

# Local Loki instance
export LOKI_ADDR=http://localhost:3100

# Remote Loki (with auth)
export LOKI_ADDR=https://loki.example.com
export LOKI_USERNAME=your-username
export LOKI_PASSWORD=your-password

# Or use Grafana Cloud
export LOKI_ADDR=https://logs-prod-us-central1.grafana.net
export LOKI_USERNAME=your-user-id
export LOKI_PASSWORD=your-api-key

Query and Tail with Gonzo

# Basic query with follow
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Filter by namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Multiple labels
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo

# With LogQL filters
logcli query --follow '{job="varlogs"} |= "error"' --output=jsonl | gonzo

# Time-based query
logcli query --since=1h '{app="myapp"}' --output=jsonl | gonzo

Common LogQL Queries

By application:

logcli query --follow '{app="nginx"}' --output=jsonl | gonzo

By severity:

logcli query --follow '{job="app"} |= "ERROR"' --output=jsonl | gonzo

Exclude patterns:

logcli query --follow '{app="api"} != "healthcheck"' --output=jsonl | gonzo

JSON parsing:

logcli query --follow '{app="api"} | json' --output=jsonl | gonzo

Regular expressions:

logcli query --follow '{app="web"} |~ "user.*login"' --output=jsonl | gonzo

Method 2: Live Tail API

For more direct integration, use Loki's Live Tail API with Gonzo's custom format support.

Set Up Custom Format

Gonzo includes a built-in Loki format. Download the format file to your config directory:

# Create formats directory
mkdir -p ~/.config/gonzo/formats

# Download Loki format (if not included)
# Or create custom format based on Loki's JSON structure

Live Tail API Connection

# Direct connection to Loki's live tail endpoint
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# With authentication
curl -s -u "$LOKI_USERNAME:$LOKI_PASSWORD" \
  "https://loki.example.com/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# Multiple labels
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"api\",env=\"prod\"}" | gonzo --format=loki-stream

Using a Shell Function

Create a convenient shell function for repeated use:

# Add to ~/.bashrc or ~/.zshrc
loki-tail() {
  local query="${1:-{app=\"myapp\"}}"
  logcli query --follow "$query" --output=jsonl | gonzo
}

# Usage
loki-tail '{namespace="production"}'
loki-tail '{app="api",level="error"}'

Loki Log Format

Loki outputs logs in a specific JSON structure. Gonzo's Loki format handles:

{
  "stream": {
    "app": "my-application",
    "namespace": "production",
    "pod": "my-app-abc123",
    "container": "app"
  },
  "values": [
    [
      "1705315805000000000",
      "2024-01-15T10:30:05Z ERROR Failed to connect to database"
    ]
  ]
}

Gonzo extracts:

Labels (stream) → Attributes panel
Timestamp → Log timestamp
Log line → Message
Severity → Detected from message

Kubernetes + Loki Workflow

Combine Loki with Kubernetes for powerful log analysis:

1. Query Kubernetes Logs in Loki

# All pods in namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Specific deployment
logcli query --follow '{namespace="prod",app="api"}' --output=jsonl | gonzo

# With error filter
logcli query --follow '{namespace="prod"} |= "ERROR"' --output=jsonl | gonzo

2. Compare with kubectl

Use both tools side by side:

# Terminal 1: Live kubectl logs
kubectl logs -f deployment/my-app | gonzo

# Terminal 2: Historical Loki logs
logcli query --since=1h '{app="my-app"}' --output=jsonl | gonzo

3. Integration with Stern

For multi-pod analysis:

# Stern for live
stern --namespace production api --output json | gonzo

# Loki for historical
logcli query --since=1h '{namespace="production",app="api"}' --output=jsonl | gonzo

Advanced Use Cases

Pipeline Validation

Verify that logs are flowing correctly into Loki:

# Check recent logs
logcli query --since=5m '{app="myapp"}' --output=jsonl | gonzo

# Verify all expected labels exist
logcli labels | grep app
logcli label app | grep myapp

Pattern Detection

Use Gonzo's pattern detection on Loki logs:

# Load logs and analyze patterns
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Press Enter on "Counts" panel to see:
# - Pattern extraction
# - Error clustering
# - Service distribution

AI Analysis

Combine Loki queries with AI analysis:

# Set up AI
export OPENAI_API_KEY="sk-your-key"

# Query and analyze
logcli query --follow '{app="api"} |= "ERROR"' --output=jsonl | gonzo --ai-model="gpt-4"

# In Gonzo:
# 1. Navigate to an error
# 2. Press Enter for details
# 3. Press 'i' for AI analysis

Time-Range Analysis

Analyze specific time periods:

# Last hour
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Specific time range
logcli query --from="2024-01-15T10:00:00Z" --to="2024-01-15T11:00:00Z" \
  '{app="api"}' --output=jsonl | gonzo

# Today's errors
logcli query --since=0h '{app="api"} |= "ERROR"' --output=jsonl | gonzo

Configuration Examples

Persistent Configuration

Create a config for Loki integration:

# ~/.config/gonzo/loki-config.yml
update-interval: 1s
log-buffer: 5000
memory-size: 20000
skin: dracula

# AI analysis for errors
ai-model: "gpt-4"

Use with:

logcli query --follow '{app="api"}' --output=jsonl | gonzo --config ~/.config/gonzo/loki-config.yml

Shell Aliases

Add to ~/.bashrc or ~/.zshrc:

# Loki + Gonzo aliases
alias loki-prod='logcli query --follow '"'"'{env="production"}'"'"' --output=jsonl | gonzo'
alias loki-errors='logcli query --follow '"'"'{level="error"}'"'"' --output=jsonl | gonzo'
alias loki-api='logcli query --follow '"'"'{app="api"}'"'"' --output=jsonl | gonzo'

# With AI
alias loki-ai='logcli query --follow --output=jsonl | gonzo --ai-model="gpt-4"'

Troubleshooting

logcli Not Connecting

Check Loki address:

echo $LOKI_ADDR
curl $LOKI_ADDR/ready

Test connection:

logcli labels
logcli query --limit=5 '{job="varlogs"}'

No Logs Appearing in Gonzo

Verify logcli output:

# Test without Gonzo first
logcli query --follow '{app="myapp"}' --output=jsonl

# Check format
logcli query --limit=1 '{app="myapp"}' --output=jsonl | jq .

Check Gonzo format:

# Ensure using jsonl output
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Not json (wrong format)
logcli query --follow '{app="myapp"}' --output=json | gonzo

Authentication Issues

Grafana Cloud:

# Ensure credentials are set
echo $LOKI_USERNAME
echo $LOKI_PASSWORD

# Test auth
logcli labels

TLS Issues:

# Skip TLS verify for testing (not for production)
export LOKI_TLS_SKIP_VERIFY=true

Performance Issues

Reduce query scope:

# Add more specific labels
logcli query --follow '{app="api",namespace="prod"}' --output=jsonl | gonzo

# Limit throughput
logcli query --follow --limit=100 '{app="api"}' --output=jsonl | gonzo

Adjust Gonzo buffers:

logcli query --follow '{app="api"}' --output=jsonl | \
  gonzo --log-buffer=10000 --update-interval=2s

Best Practices

1. Use Specific Label Selectors

# ❌ Too broad
logcli query --follow '{}' --output=jsonl | gonzo

# ✅ Specific
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo

2. Combine with Filters

# Filter in LogQL for efficiency
logcli query --follow '{app="api"} |= "ERROR" != "healthcheck"' --output=jsonl | gonzo

3. Leverage Gonzo's Features

Pause (Space) during investigation
Filter (/) to narrow further
AI Analysis (i) for complex errors
Patterns (Counts modal) for trends

4. Save Common Queries

# Create query scripts
cat > ~/loki-queries/prod-errors.sh << 'EOF'
#!/bin/bash
logcli query --follow '{env="production"} |= "ERROR"' --output=jsonl | gonzo
EOF

chmod +x ~/loki-queries/prod-errors.sh

Complete Tutorial

For a comprehensive walkthrough of using Gonzo with Loki, including:

Detailed setup instructions
Real-world examples
Advanced LogQL patterns
Custom format configuration
Integration patterns

Read the full guide: Live Tailing Grafana Loki Logs with Gonzo

Documentation

Custom Formats - Creating format parsers
Kubernetes Integration - K8s log tailing
AI Integration - AI-powered analysis

External Links

Blog Posts

Support

Having issues with Loki integration?

Check Troubleshooting Guide
Ask in GitHub Discussions
Report bugs in GitHub Issues

Pro Tip: Use logcli with --output=jsonl (JSON Lines) for best compatibility with Gonzo. Each log entry is a separate JSON object on its own line.

Learn More: Read the complete tutorial on Live Tailing Grafana Loki Logs with Gonzo for advanced patterns and real-world examples.

PreviousAWS CloudWatch ☁️NextVercel Logs Integration ▲

Last updated 2 months ago

Overview

Integration Methods

Method 1: Using logcli

Install logcli

Configure logcli

Query and Tail with Gonzo

Common LogQL Queries

Method 2: Live Tail API

Set Up Custom Format

Live Tail API Connection

Using a Shell Function

Loki Log Format

Kubernetes + Loki Workflow

1. Query Kubernetes Logs in Loki

2. Compare with kubectl

3. Integration with Stern

Advanced Use Cases

Pipeline Validation

Pattern Detection

AI Analysis

Time-Range Analysis

Configuration Examples

Persistent Configuration

Shell Aliases

Troubleshooting

logcli Not Connecting

No Logs Appearing in Gonzo

Authentication Issues

Performance Issues

Best Practices

1. Use Specific Label Selectors

2. Combine with Filters

3. Leverage Gonzo's Features

4. Save Common Queries

Complete Tutorial

Related Resources

Documentation

External Links

Blog Posts

Support