# Grafana Loki Integration 🔥

Integrate Gonzo with Grafana Loki for powerful live log tailing and analysis. Use Loki as your log store and Gonzo as your real-time terminal lens.

### Overview

Grafana Loki is a powerful log aggregation system designed for storing and querying logs at scale. While Loki excels at long-term retention and powerful queries via LogQL, live tailing logs for debugging and troubleshooting can be challenging. Gonzo fills this gap by providing a real-time, terminal-based interface for Loki logs.

**Why use Gonzo with Loki?**

* **Debug in real time**: Watch logs as code changes roll out
* **Validate pipelines**: Ensure OpenTelemetry agents ship logs correctly to Loki
* **Shorten feedback loops**: Catch issues instantly without switching tools
* **Stay in-flow**: Operate inside your terminal alongside kubectl, stern, and k9s
* **AI-powered insights**: Optional AI analysis of Loki logs

### Integration Methods

Gonzo supports two primary methods for accessing Loki logs:

1. **logcli** - Using Loki's official CLI tool
2. **Live Tail API** - Direct connection to Loki's live tailing endpoint

### Method 1: Using logcli

The easiest way to get started is using Loki's official `logcli` tool.

#### Install logcli

```bash
# macOS via Homebrew
brew install logcli

# Linux - download binary
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.0/logcli-linux-amd64.zip"
unzip "logcli-linux-amd64.zip"
chmod a+x logcli-linux-amd64
sudo mv logcli-linux-amd64 /usr/local/bin/logcli

# Verify installation
logcli --version
```

#### Configure logcli

Set up your Loki connection:

```bash
# Local Loki instance
export LOKI_ADDR=http://localhost:3100

# Remote Loki (with auth)
export LOKI_ADDR=https://loki.example.com
export LOKI_USERNAME=your-username
export LOKI_PASSWORD=your-password

# Or use Grafana Cloud
export LOKI_ADDR=https://logs-prod-us-central1.grafana.net
export LOKI_USERNAME=your-user-id
export LOKI_PASSWORD=your-api-key
```

#### Query and Tail with Gonzo

```bash
# Basic query with follow
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Filter by namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Multiple labels
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo

# With LogQL filters
logcli query --follow '{job="varlogs"} |= "error"' --output=jsonl | gonzo

# Time-based query
logcli query --since=1h '{app="myapp"}' --output=jsonl | gonzo
```

#### Common LogQL Queries

**By application**:

```bash
logcli query --follow '{app="nginx"}' --output=jsonl | gonzo
```

**By severity**:

```bash
logcli query --follow '{job="app"} |= "ERROR"' --output=jsonl | gonzo
```

**Exclude patterns**:

```bash
logcli query --follow '{app="api"} != "healthcheck"' --output=jsonl | gonzo
```

**JSON parsing**:

```bash
logcli query --follow '{app="api"} | json' --output=jsonl | gonzo
```

**Regular expressions**:

```bash
logcli query --follow '{app="web"} |~ "user.*login"' --output=jsonl | gonzo
```

### Method 2: Live Tail API

For more direct integration, use Loki's Live Tail API with Gonzo's custom format support.

#### Set Up Custom Format

Gonzo includes a built-in Loki format. Download the format file to your config directory:

```bash
# Create formats directory
mkdir -p ~/.config/gonzo/formats

# Download Loki format (if not included)
# Or create custom format based on Loki's JSON structure
```

#### Live Tail API Connection

```bash
# Direct connection to Loki's live tail endpoint
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# With authentication
curl -s -u "$LOKI_USERNAME:$LOKI_PASSWORD" \
  "https://loki.example.com/loki/api/v1/tail?query={app=\"myapp\"}" | gonzo --format=loki-stream

# Multiple labels
curl -s "http://localhost:3100/loki/api/v1/tail?query={app=\"api\",env=\"prod\"}" | gonzo --format=loki-stream
```

#### Using a Shell Function

Create a convenient shell function for repeated use:

```bash
# Add to ~/.bashrc or ~/.zshrc
loki-tail() {
  local query="${1:-{app=\"myapp\"}}"
  logcli query --follow "$query" --output=jsonl | gonzo
}

# Usage
loki-tail '{namespace="production"}'
loki-tail '{app="api",level="error"}'
```

### Loki Log Format

Loki outputs logs in a specific JSON structure. Gonzo's Loki format handles:

```json
{
  "stream": {
    "app": "my-application",
    "namespace": "production",
    "pod": "my-app-abc123",
    "container": "app"
  },
  "values": [
    [
      "1705315805000000000",
      "2024-01-15T10:30:05Z ERROR Failed to connect to database"
    ]
  ]
}
```

**Gonzo extracts**:

* **Labels** (`stream`) → Attributes panel
* **Timestamp** → Log timestamp
* **Log line** → Message
* **Severity** → Detected from message

### Kubernetes + Loki Workflow

Combine Loki with Kubernetes for powerful log analysis:

#### 1. Query Kubernetes Logs in Loki

```bash
# All pods in namespace
logcli query --follow '{namespace="production"}' --output=jsonl | gonzo

# Specific deployment
logcli query --follow '{namespace="prod",app="api"}' --output=jsonl | gonzo

# With error filter
logcli query --follow '{namespace="prod"} |= "ERROR"' --output=jsonl | gonzo
```

#### 2. Compare with kubectl

Use both tools side by side:

```bash
# Terminal 1: Live kubectl logs
kubectl logs -f deployment/my-app | gonzo

# Terminal 2: Historical Loki logs
logcli query --since=1h '{app="my-app"}' --output=jsonl | gonzo
```

#### 3. Integration with Stern

For multi-pod analysis:

```bash
# Stern for live
stern --namespace production api --output json | gonzo

# Loki for historical
logcli query --since=1h '{namespace="production",app="api"}' --output=jsonl | gonzo
```

### Advanced Use Cases

#### Pipeline Validation

Verify that logs are flowing correctly into Loki:

```bash
# Check recent logs
logcli query --since=5m '{app="myapp"}' --output=jsonl | gonzo

# Verify all expected labels exist
logcli labels | grep app
logcli label app | grep myapp
```

#### Pattern Detection

Use Gonzo's pattern detection on Loki logs:

```bash
# Load logs and analyze patterns
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Press Enter on "Counts" panel to see:
# - Pattern extraction
# - Error clustering
# - Service distribution
```

#### AI Analysis

Combine Loki queries with AI analysis:

```bash
# Set up AI
export OPENAI_API_KEY="sk-your-key"

# Query and analyze
logcli query --follow '{app="api"} |= "ERROR"' --output=jsonl | gonzo --ai-model="gpt-4"

# In Gonzo:
# 1. Navigate to an error
# 2. Press Enter for details
# 3. Press 'i' for AI analysis
```

#### Time-Range Analysis

Analyze specific time periods:

```bash
# Last hour
logcli query --since=1h '{app="api"}' --output=jsonl | gonzo

# Specific time range
logcli query --from="2024-01-15T10:00:00Z" --to="2024-01-15T11:00:00Z" \
  '{app="api"}' --output=jsonl | gonzo

# Today's errors
logcli query --since=0h '{app="api"} |= "ERROR"' --output=jsonl | gonzo
```

### Configuration Examples

#### Persistent Configuration

Create a config for Loki integration:

```yaml
# ~/.config/gonzo/loki-config.yml
update-interval: 1s
log-buffer: 5000
memory-size: 20000
skin: dracula

# AI analysis for errors
ai-model: "gpt-4"
```

Use with:

```bash
logcli query --follow '{app="api"}' --output=jsonl | gonzo --config ~/.config/gonzo/loki-config.yml
```

#### Shell Aliases

Add to `~/.bashrc` or `~/.zshrc`:

```bash
# Loki + Gonzo aliases
alias loki-prod='logcli query --follow '"'"'{env="production"}'"'"' --output=jsonl | gonzo'
alias loki-errors='logcli query --follow '"'"'{level="error"}'"'"' --output=jsonl | gonzo'
alias loki-api='logcli query --follow '"'"'{app="api"}'"'"' --output=jsonl | gonzo'

# With AI
alias loki-ai='logcli query --follow --output=jsonl | gonzo --ai-model="gpt-4"'
```

### Troubleshooting

#### logcli Not Connecting

**Check Loki address**:

```bash
echo $LOKI_ADDR
curl $LOKI_ADDR/ready
```

**Test connection**:

```bash
logcli labels
logcli query --limit=5 '{job="varlogs"}'
```

#### No Logs Appearing in Gonzo

**Verify logcli output**:

```bash
# Test without Gonzo first
logcli query --follow '{app="myapp"}' --output=jsonl

# Check format
logcli query --limit=1 '{app="myapp"}' --output=jsonl | jq .
```

**Check Gonzo format**:

```bash
# Ensure using jsonl output
logcli query --follow '{app="myapp"}' --output=jsonl | gonzo

# Not json (wrong format)
logcli query --follow '{app="myapp"}' --output=json | gonzo
```

#### Authentication Issues

**Grafana Cloud**:

```bash
# Ensure credentials are set
echo $LOKI_USERNAME
echo $LOKI_PASSWORD

# Test auth
logcli labels
```

**TLS Issues**:

```bash
# Skip TLS verify for testing (not for production)
export LOKI_TLS_SKIP_VERIFY=true
```

#### Performance Issues

**Reduce query scope**:

```bash
# Add more specific labels
logcli query --follow '{app="api",namespace="prod"}' --output=jsonl | gonzo

# Limit throughput
logcli query --follow --limit=100 '{app="api"}' --output=jsonl | gonzo
```

**Adjust Gonzo buffers**:

```bash
logcli query --follow '{app="api"}' --output=jsonl | \
  gonzo --log-buffer=10000 --update-interval=2s
```

### Best Practices

#### 1. Use Specific Label Selectors

```bash
# ❌ Too broad
logcli query --follow '{}' --output=jsonl | gonzo

# ✅ Specific
logcli query --follow '{app="api",env="prod"}' --output=jsonl | gonzo
```

#### 2. Combine with Filters

```bash
# Filter in LogQL for efficiency
logcli query --follow '{app="api"} |= "ERROR" != "healthcheck"' --output=jsonl | gonzo
```

#### 3. Leverage Gonzo's Features

* **Pause** (`Space`) during investigation
* **Filter** (`/`) to narrow further
* **AI Analysis** (`i`) for complex errors
* **Patterns** (Counts modal) for trends

#### 4. Save Common Queries

```bash
# Create query scripts
cat > ~/loki-queries/prod-errors.sh << 'EOF'
#!/bin/bash
logcli query --follow '{env="production"} |= "ERROR"' --output=jsonl | gonzo
EOF

chmod +x ~/loki-queries/prod-errors.sh
```

### Complete Tutorial

For a comprehensive walkthrough of using Gonzo with Loki, including:

* Detailed setup instructions
* Real-world examples
* Advanced LogQL patterns
* Custom format configuration
* Integration patterns

**Read the full guide**: [Live Tailing Grafana Loki Logs with Gonzo](https://www.controltheory.com/blog/live-tailing-grafana-loki-logs-with-gonzo/)

### Related Resources

#### Documentation

* Custom Formats - Creating format parsers
* Kubernetes Integration - K8s log tailing
* AI Integration - AI-powered analysis

#### External Links

* [Grafana Loki Documentation](https://grafana.com/docs/loki/)
* [logcli Documentation](https://grafana.com/docs/loki/latest/tools/logcli/)
* [LogQL Guide](https://grafana.com/docs/loki/latest/logql/)

#### Blog Posts

* [Live Tailing Grafana Loki Logs with Gonzo](https://www.controltheory.com/blog/live-tailing-grafana-loki-logs-with-gonzo/)
* [Using Gonzo with k9s and Stern](https://www.controltheory.com/blog/using-gonzo-with-k9s-and-stern-supercharge-your-kubernetes-log-tailing/)

### Support

Having issues with Loki integration?

* Check Troubleshooting Guide
* Ask in [GitHub Discussions](https://github.com/control-theory/gonzo/discussions)
* Report bugs in [GitHub Issues](https://github.com/control-theory/gonzo/issues)

{% hint style="success" %}
**Pro Tip**: Use `logcli` with `--output=jsonl` (JSON Lines) for best compatibility with Gonzo. Each log entry is a separate JSON object on its own line.
{% endhint %}

{% hint style="info" %}
**Learn More**: Read the complete tutorial on [Live Tailing Grafana Loki Logs with Gonzo](https://www.controltheory.com/blog/live-tailing-grafana-loki-logs-with-gonzo/) for advanced patterns and real-world examples.
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.controltheory.com/controltheory-documentation/gonzo-docs/integration-examples/grafana-loki-integration.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.