# Grafana Loki Integration 🔥

Integrate Gonzo with Grafana Loki for powerful live log tailing and analysis. Use Loki as your log store and Gonzo as your real-time terminal lens.

### Overview

Grafana Loki is a powerful log aggregation system designed for storing and querying logs at scale. While Loki excels at long-term retention and powerful queries via LogQL, live tailing logs for debugging and troubleshooting can be challenging. Gonzo fills this gap by providing a real-time, terminal-based interface for Loki logs.

**Why use Gonzo with Loki?**

* **Debug in real time**: Watch logs as code changes roll out
* **Validate pipelines**: Ensure OpenTelemetry agents ship logs correctly to Loki
* **Shorten feedback loops**: Catch issues instantly without switching tools
* **Stay in-flow**: Operate inside your terminal alongside kubectl, stern, and k9s
* **AI-powered insights**: Optional AI analysis of Loki logs

### Integration Methods

Gonzo supports two primary methods for accessing Loki logs:

1. **logcli** - Using Loki's official CLI tool
2. **Live Tail API** - Direct connection to Loki's live tailing endpoint

### Method 1: Using logcli

The easiest way to get started is using Loki's official `logcli` tool.

#### Install logcli

```bash
# macOS via Homebrew
brew install logcli

# Linux - download binary
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.0/logcli-linux-amd64.zip"
unzip "logcli-linux-amd64.zip"
chmod a+x logcli-linux-amd64
sudo mv logcli-linux-amd64 /usr/local/bin/logcli

# Verify installation
logcli --version
```

#### Configure logcli

Set up your Loki connection:

```bash
# Local Loki instance
export LOKI_ADDR=http://localhost:3100

# Remote Loki (with auth)
export LOKI_ADDR=https://loki.example.com
export LOKI_USERNAME=your-username
export LOKI_PASSWORD=your-password

# Or use Grafana Cloud
export LOKI_ADDR=https://logs-prod-us-central1.grafana.net
export LOKI_USERNAME=your-user-id
export LOKI_PASSWORD=your-api-key
```

#### Query and Tail with Gonzo

```bash
# Basic query with follow
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Filter by namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Multiple labels
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo

# With LogQL filters
logcli query --follow '{job="varlogs"} |= "error"' --output=jsonl | gonzo

# Time-based query
logcli query --since=1h '{app="myapp"}' --output=jsonl | gonzo
```

#### Common LogQL Queries

**By application**:

```bash
logcli query --follow '{app="nginx"}' --output=jsonl | gonzo
```

**By severity**:

```bash
logcli query --follow '{job="app"} |= "ERROR"' --output=jsonl | gonzo
```

**Exclude patterns**:

```bash
logcli query --follow '{app="api"} != "healthcheck"' --output=jsonl | gonzo
```

**JSON parsing**:

```bash
logcli query --follow '{app="api"} | json' --output=jsonl | gonzo
```

**Regular expressions**:

```bash
logcli query --follow '{app="web"} |~ "user.*login"' --output=jsonl | gonzo
```

### Method 2: Live Tail API

For more direct integration, use Loki's Live Tail API with Gonzo's custom format support.

#### Set Up Custom Format

Gonzo includes a built-in Loki format. Download the format file to your config directory:

```bash
# Create formats directory
mkdir -p ~/.config/gonzo/formats

# Download Loki format (if not included)
# Or create custom format based on Loki's JSON structure
```

#### Live Tail API Connection

```bash
# Direct connection to Loki's live tail endpoint
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# With authentication
curl -s -u "$LOKI_USERNAME:$LOKI_PASSWORD" \
  "https://loki.example.com/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# Multiple labels
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"api\",env=\"prod\"}" | gonzo --format=loki-stream
```

#### Using a Shell Function

Create a convenient shell function for repeated use:

```bash
# Add to ~/.bashrc or ~/.zshrc
loki-tail() {
  local query="${1:-{app=\"myapp\"}}"
  logcli query --follow "$query" --output=jsonl | gonzo
}

# Usage
loki-tail '{namespace="production"}'
loki-tail '{app="api",level="error"}'
```

### Loki Log Format

Loki outputs logs in a specific JSON structure. Gonzo's Loki format handles:

```json
{
  "stream": {
    "app": "my-application",
    "namespace": "production",
    "pod": "my-app-abc123",
    "container": "app"
  },
  "values": [
    [
      "1705315805000000000",
      "2024-01-15T10:30:05Z ERROR Failed to connect to database"
    ]
  ]
}
```

**Gonzo extracts**:

* **Labels** (`stream`) → Attributes panel
* **Timestamp** → Log timestamp
* **Log line** → Message
* **Severity** → Detected from message

### Kubernetes + Loki Workflow

Combine Loki with Kubernetes for powerful log analysis:

#### 1. Query Kubernetes Logs in Loki

```bash
# All pods in namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Specific deployment
logcli query --follow '{namespace="prod",app="api"}' --output=jsonl | gonzo

# With error filter
logcli query --follow '{namespace="prod"} |= "ERROR"' --output=jsonl | gonzo
```

#### 2. Compare with kubectl

Use both tools side by side:

```bash
# Terminal 1: Live kubectl logs
kubectl logs -f deployment/my-app | gonzo

# Terminal 2: Historical Loki logs
logcli query --since=1h '{app="my-app"}' --output=jsonl | gonzo
```

#### 3. Integration with Stern

For multi-pod analysis:

```bash
# Stern for live
stern --namespace production api --output json | gonzo

# Loki for historical
logcli query --since=1h '{namespace="production",app="api"}' --output=jsonl | gonzo
```

### Advanced Use Cases

#### Pipeline Validation

Verify that logs are flowing correctly into Loki:

```bash
# Check recent logs
logcli query --since=5m '{app="myapp"}' --output=jsonl | gonzo

# Verify all expected labels exist
logcli labels | grep app
logcli label app | grep myapp
```

#### Pattern Detection

Use Gonzo's pattern detection on Loki logs:

```bash
# Load logs and analyze patterns
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Press Enter on "Counts" panel to see:
# - Pattern extraction
# - Error clustering
# - Service distribution
```

#### AI Analysis

Combine Loki queries with AI analysis:

```bash
# Set up AI
export OPENAI_API_KEY="sk-your-key"

# Query and analyze
logcli query --follow '{app="api"} |= "ERROR"' --output=jsonl | gonzo --ai-model="gpt-4"

# In Gonzo:
# 1. Navigate to an error
# 2. Press Enter for details
# 3. Press 'i' for AI analysis
```

#### Time-Range Analysis

Analyze specific time periods:

```bash
# Last hour
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Specific time range
logcli query --from="2024-01-15T10:00:00Z" --to="2024-01-15T11:00:00Z" \
  '{app="api"}' --output=jsonl | gonzo

# Today's errors
logcli query --since=0h '{app="api"} |= "ERROR"' --output=jsonl | gonzo
```

### Configuration Examples

#### Persistent Configuration

Create a config for Loki integration:

```yaml
# ~/.config/gonzo/loki-config.yml
update-interval: 1s
log-buffer: 5000
memory-size: 20000
skin: dracula

# AI analysis for errors
ai-model: "gpt-4"
```

Use with:

```bash
logcli query --follow '{app="api"}' --output=jsonl | gonzo --config ~/.config/gonzo/loki-config.yml
```

#### Shell Aliases

Add to `~/.bashrc` or `~/.zshrc`:

```bash
# Loki + Gonzo aliases
alias loki-prod='logcli query --follow '"'"'{env="production"}'"'"' --output=jsonl | gonzo'
alias loki-errors='logcli query --follow '"'"'{level="error"}'"'"' --output=jsonl | gonzo'
alias loki-api='logcli query --follow '"'"'{app="api"}'"'"' --output=jsonl | gonzo'

# With AI
alias loki-ai='logcli query --follow --output=jsonl | gonzo --ai-model="gpt-4"'
```

### Troubleshooting

#### logcli Not Connecting

**Check Loki address**:

```bash
echo $LOKI_ADDR
curl $LOKI_ADDR/ready
```

**Test connection**:

```bash
logcli labels
logcli query --limit=5 '{job="varlogs"}'
```

#### No Logs Appearing in Gonzo

**Verify logcli output**:

```bash
# Test without Gonzo first
logcli query --follow '{app="myapp"}' --output=jsonl

# Check format
logcli query --limit=1 '{app="myapp"}' --output=jsonl | jq .
```

**Check Gonzo format**:

```bash
# Ensure using jsonl output
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Not json (wrong format)
logcli query --follow '{app="myapp"}' --output=json | gonzo
```

#### Authentication Issues

**Grafana Cloud**:

```bash
# Ensure credentials are set
echo $LOKI_USERNAME
echo $LOKI_PASSWORD

# Test auth
logcli labels
```

**TLS Issues**:

```bash
# Skip TLS verify for testing (not for production)
export LOKI_TLS_SKIP_VERIFY=true
```

#### Performance Issues

**Reduce query scope**:

```bash
# Add more specific labels
logcli query --follow '{app="api",namespace="prod"}' --output=jsonl | gonzo

# Limit throughput
logcli query --follow --limit=100 '{app="api"}' --output=jsonl | gonzo
```

**Adjust Gonzo buffers**:

```bash
logcli query --follow '{app="api"}' --output=jsonl | \
  gonzo --log-buffer=10000 --update-interval=2s
```

### Best Practices

#### 1. Use Specific Label Selectors

```bash
# ❌ Too broad
logcli query --follow '{}' --output=jsonl | gonzo

# ✅ Specific
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo
```

#### 2. Combine with Filters

```bash
# Filter in LogQL for efficiency
logcli query --follow '{app="api"} |= "ERROR" != "healthcheck"' --output=jsonl | gonzo
```

#### 3. Leverage Gonzo's Features

* **Pause** (`Space`) during investigation
* **Filter** (`/`) to narrow further
* **AI Analysis** (`i`) for complex errors
* **Patterns** (Counts modal) for trends

#### 4. Save Common Queries

```bash
# Create query scripts
cat > ~/loki-queries/prod-errors.sh << 'EOF'
#!/bin/bash
logcli query --follow '{env="production"} |= "ERROR"' --output=jsonl | gonzo
EOF

chmod +x ~/loki-queries/prod-errors.sh
```

### Complete Tutorial

For a comprehensive walkthrough of using Gonzo with Loki, including:

* Detailed setup instructions
* Real-world examples
* Advanced LogQL patterns
* Custom format configuration
* Integration patterns

**Read the full guide**: [Live Tailing Grafana Loki Logs with Gonzo](https://www.controltheory.com/blog/live-tailing-grafana-loki-logs-with-gonzo/)

### Related Resources

#### Documentation

* Custom Formats - Creating format parsers
* Kubernetes Integration - K8s log tailing
* AI Integration - AI-powered analysis

#### External Links

* [Grafana Loki Documentation](https://grafana.com/docs/loki/)
* [logcli Documentation](https://grafana.com/docs/loki/latest/tools/logcli/)
* [LogQL Guide](https://grafana.com/docs/loki/latest/logql/)

#### Blog Posts

* [Live Tailing Grafana Loki Logs with Gonzo](https://www.controltheory.com/blog/live-tailing-grafana-loki-logs-with-gonzo/)
* [Using Gonzo with k9s and Stern](https://www.controltheory.com/blog/using-gonzo-with-k9s-and-stern-supercharge-your-kubernetes-log-tailing/)

### Support

Having issues with Loki integration?

* Check Troubleshooting Guide
* Ask in [GitHub Discussions](https://github.com/control-theory/gonzo/discussions)
* Report bugs in [GitHub Issues](https://github.com/control-theory/gonzo/issues)

{% hint style="success" %}
**Pro Tip**: Use `logcli` with `--output=jsonl` (JSON Lines) for best compatibility with Gonzo. Each log entry is a separate JSON object on its own line.
{% endhint %}

{% hint style="info" %}
**Learn More**: Read the complete tutorial on [Live Tailing Grafana Loki Logs with Gonzo](https://www.controltheory.com/blog/live-tailing-grafana-loki-logs-with-gonzo/) for advanced patterns and real-world examples.
{% endhint %}
