# Detailed Usage Guide

Master advanced Gonzo workflows, real-world scenarios, and power-user techniques. This comprehensive guide covers everything from complex multi-source analysis to performance optimization and troubleshooting strategies.

{% hint style="success" %}
**Prerequisites:** Complete the other User Guide sections first - this builds on Interface Overview, Navigation & Controls, Log Input Methods, and Filtering & Search.
{% endhint %}

### Advanced Analysis Workflows

#### Multi-Service Investigation

Analyze logs from multiple services to understand system-wide issues:

{% tabs %}
{% tab title="Microservices Debugging" %}
**Scenario:** API gateway timeouts affecting user experience

```bash
# 1. Start with all related services
gonzo -f api-gateway.log -f user-service.log -f auth-service.log --follow

# 2. Filter for error timeframe
/2024-01-15.*1[0-2]:[0-5][0-9]  # Focus on specific time window

# 3. Look for correlation patterns
# Tab to Word Frequency - look for "timeout", "503", "error"

# 4. Progressive filtering
/timeout.*[0-9]+     # Find timeout events
/trace_id.*abc123    # Follow specific request trace

# 5. Cross-service correlation
# Use Attributes panel to click trace_id values
# Follow the request path across services
```

**Key Techniques:**

* Start broad with all services, narrow by time and error type
* Use trace IDs or correlation IDs to follow requests
* Compare Word Frequency across different time periods
* Leverage Attributes panel for structured data correlation
  {% endtab %}

{% tab title="Infrastructure Problem Analysis" %}
**Scenario:** Database slowdown impacting multiple applications

```bash
# 1. Monitor infrastructure and applications
gonzo -f /var/log/postgresql/postgresql.log \
      -f /var/log/nginx/access.log \
      -f /var/log/app/*.log --follow

# 2. Look for performance indicators
/(slow|timeout|high|lag|delay)

# 3. Correlate timing patterns
# Press Enter on Counts panel for time-series analysis
# Look for spikes in error frequency

# 4. Identify root cause service
# Filter by service: service.*database
# Look for connection pool exhaustion, slow queries

# 5. Impact assessment
# Remove database filter, look for cascade effects
/(503|timeout|unavailable)
```

**Analysis Strategy:**

* Monitor both infrastructure and application layers
* Use time-series analysis in Counts modal
* Identify the root cause service first
* Assess cascade effects on dependent services
  {% endtab %}
  {% endtabs %}

#### Performance Analysis Workflows

**Response Time Investigation**

```bash
# 1. Identify slow operations
gonzo -f app.log --follow
/duration.*[5-9][0-9]{3}   # Duration > 5000ms

# 2. Categorize slow operations  
# Tab to Attributes panel, look for:
# - endpoint patterns
# - user_id patterns  
# - method types
# - database query patterns

# 3. Pattern analysis
# Press Enter on Counts panel
# Look for:
# - Time correlation (when do slowdowns occur?)
# - Pattern frequency (which operations are consistently slow?)
# - Service distribution (which services are affected?)

# 4. Deep dive investigation
/endpoint.*\/api\/users.*duration.*[5-9][0-9]{3}
# Focus on specific slow endpoints

# 5. AI-powered insights (if configured)
# Press 'i' in log details for AI analysis
# Ask: "Why are these operations slow?"
```

**Resource Utilization Analysis**

```bash
# Monitor resource-related logs
gonzo -f app.log -f /var/log/syslog --follow

# Filter for resource indicators
/(memory|cpu|disk|network).*high
/(out.*of.*memory|heap.*exceeded|disk.*full)

# Correlate with application performance
# Look for patterns like:
# - Memory pressure -> GC pauses -> slow responses
# - CPU spikes -> request queueing -> timeouts
# - Disk I/O -> database slowdown -> cascade failures

# Use time-series analysis
# Press Enter on Counts panel
# Look for resource usage spikes correlating with error spikes
```

#### Security Monitoring Workflows

**Authentication Analysis**

```bash
# Monitor authentication-related logs
sudo gonzo -f /var/log/auth.log -f /var/log/app/auth.log --follow

# Look for suspicious patterns
/(fail|invalid|denied|brute.*force)

# Analyze attack patterns
# Tab to Word Frequency - look for:
# - High frequency of specific usernames
# - Repeated IP addresses
# - Unusual user agents

# Geographic analysis (if available)
/country.*CN.*fail   # Failed logins from specific countries
/ip.*192\.168       # Internal vs external attempts

# Time-based analysis
# Press Enter on Counts panel
# Look for attack timing patterns
# - Off-hours activity
# - Burst patterns suggesting automation
```

**Access Control Investigation**

```bash
# Monitor access control events
gonzo -f /var/log/nginx/access.log -f app.log --follow

# Filter for access violations
/status.*(403|401|405)
/(unauthorized|forbidden|access.*denied)

# Analyze access patterns
# Look for:
# - Unusual endpoint access attempts
# - Privilege escalation attempts
# - Data exfiltration patterns

# Use structured filtering
/method.*POST.*status.*403   # POST attempts that failed
/endpoint.*admin.*status.*401 # Admin access attempts
```

### Advanced Configuration Strategies

#### Environment-Specific Configurations

**Development Environment Setup**

```bash
# Create ~/.config/gonzo/dev-config.yml
files:
  - "logs/*.log"
  - "debug/*.log"
follow: true
update-interval: 1s
log-buffer: 2000
ai-model: "gpt-3.5-turbo"  # Faster model for dev

# Usage
gonzo --config ~/.config/gonzo/dev-config.yml
```

**Production Monitoring Setup**

```bash
# Create ~/.config/gonzo/prod-config.yml  
files:
  - "/var/log/app/*.log"
  - "/var/log/nginx/*.log"
follow: true
update-interval: 5s       # Less frequent updates
log-buffer: 10000        # Larger buffer for high volume
memory-size: 50000       # More memory for complex analysis
ai-model: "gpt-4"        # More capable model for critical analysis

# Usage with additional monitoring
gonzo --config ~/.config/gonzo/prod-config.yml \
      -f /var/log/security.log
```

**CI/CD Integration Setup**

```bash
# Create build-analysis.sh
#!/bin/bash
# Analyze build logs with Gonzo

if [ $# -eq 0 ]; then
    echo "Usage: $0 <build-log-file>"
    exit 1
fi

export GONZO_LOG_BUFFER=5000
export GONZO_UPDATE_INTERVAL=2s

gonzo -f "$1" \
      --ai-model="gpt-3.5-turbo" \
      --log-buffer=5000

# Integration with CI
# - Run after failed builds
# - Pipe build output directly: build-command | gonzo
# - Generate failure reports with AI analysis
```

#### Performance Optimization Strategies

**High-Volume Log Handling**

```bash
# For logs with >1000 entries/second
gonzo -f high-volume.log \
      --follow \
      --log-buffer=20000 \
      --memory-size=100000 \
      --update-interval=10s

# Pre-filtering for performance
tail -f /var/log/high-volume.log | grep -E "(ERROR|WARN)" | gonzo

# Distributed analysis approach
# Terminal 1: Current errors
tail -f /var/log/app.log | grep ERROR | gonzo

# Terminal 2: Historical analysis  
gonzo -f /var/log/app.log.1

# Terminal 3: Performance monitoring
tail -f /var/log/app.log | grep -E "(slow|timeout|duration.*[5-9])" | gonzo
```

**Memory-Constrained Environments**

```bash
# Minimal memory configuration
gonzo -f app.log \
      --log-buffer=500 \
      --memory-size=2000 \
      --update-interval=5s

# Streaming analysis (minimal buffering)
tail -f /var/log/app.log | gonzo --log-buffer=100

# Batch processing approach
# Process logs in chunks
split -l 10000 huge.log chunk_
for chunk in chunk_*; do
    echo "Analyzing $chunk..."
    gonzo -f "$chunk"
    read -p "Press enter for next chunk..."
done
```

### Real-World Scenarios

#### Incident Response Workflows

**Production Outage Investigation**

```bash
# Step 1: Rapid triage
gonzo -f /var/log/app/*.log -f /var/log/nginx/*.log --follow

# Step 2: Focus on errors during outage timeframe  
/2024-01-15.*09:[1-3][0-9]   # 9:10-9:39 AM timeframe

# Step 3: Identify error categories
# Tab to Word Frequency
# Look for spike in: "500", "timeout", "connection", "database"

# Step 4: Follow the cascade
/(connection.*refused|timeout|circuit.*breaker)

# Step 5: Find root cause
# Press Enter on Counts panel
# Look at time-series heatmap
# Identify when errors started vs when they peaked

# Step 6: AI-assisted analysis
# Press 'i' on critical error logs
# Ask: "What caused this outage?"

# Step 7: Document timeline
# Use Home/End to build timeline of events
# Note: First error at 09:12, peak at 09:15, resolution at 09:35
```

**Performance Degradation Investigation**

```bash
# Monitor performance indicators
gonzo -f app.log --follow
/(duration|response.*time|elapsed).*[5-9][0-9]{3}

# Correlate with resource usage
gonzo -f app.log -f /var/log/syslog --follow
/(memory|cpu|load).*high

# Identify affected operations
# Use Attributes panel to group by:
# - endpoint
# - user_id  
# - operation_type

# Pattern analysis workflow:
# 1. Filter by slow operations
# 2. Press Enter on Counts for pattern analysis
# 3. Look for common attributes in slow requests
# 4. Correlate with infrastructure metrics
# 5. Identify optimization opportunities
```

#### Development Workflows

**Debugging Application Issues**

```bash
# Development debugging session
gonzo -f logs/app.log -f logs/debug.log --follow

# Focus on current development
/user_id.*12345   # Your test user

# Feature-specific debugging
/feature.*new_checkout.*error

# Exception tracing
/exception.*NullPointer
# Press Enter on specific exception
# Use 'i' for AI analysis: "Explain this exception"

# Performance profiling
/profile.*slow.*query
/(query|sql).*[1-9][0-9]{3}ms   # Queries > 1000ms
```

**Integration Testing Analysis**

```bash
# Monitor integration test logs
gonzo -f test-results.log --follow

# Filter test failures
/(FAIL|ERROR|Assert.*fail)

# API integration analysis
/http.*[45][0-9]{2}   # HTTP 4xx/5xx responses

# Database integration issues
/(connection.*fail|query.*timeout|constraint.*violation)

# Use AI for test failure analysis
# Press 'i' on failed test logs
# Ask: "Why did this integration test fail?"
```

### Advanced Integration Patterns

#### Container Orchestration

**Kubernetes Advanced Monitoring**

```bash
# Multi-namespace monitoring
kubectl logs -f -l app=backend --all-namespaces | gonzo

# Pod lifecycle analysis
kubectl get events --watch | grep -E "(Warning|Error)" | gonzo

# Resource constraint investigation
kubectl top pods --containers | gonzo

# Custom resource monitoring
kubectl logs -f -l tier=database | grep -E "(slow|timeout|error)" | gonzo

# Helm release monitoring
helm status myapp --output json | jq '.info.notes' | gonzo
```

**Docker Swarm Integration**

```bash
# Service log aggregation
docker service logs -f $(docker service ls -q) | gonzo

# Node-specific analysis
docker node ls --format "table {{.ID}}\t{{.Hostname}}\t{{.Status}}"
docker logs -f $(docker ps -q --filter node=worker1) | gonzo

# Stack monitoring
docker stack ps mystack --format "table {{.Name}}\t{{.Image}}\t{{.CurrentState}}"
docker service logs -f mystack_web | gonzo
```

#### Cloud Platform Integration

**AWS CloudWatch Integration Pattern**

```bash
# CloudWatch Logs streaming (requires aws-cli)
aws logs tail /aws/lambda/my-function --follow | gonzo

# ECS task monitoring
aws ecs describe-tasks --cluster my-cluster --tasks $(aws ecs list-tasks --cluster my-cluster --query 'taskArns[]' --output text) | gonzo

# Application Load Balancer logs
aws s3 cp s3://my-alb-logs/ . --recursive
gunzip -c *.gz | gonzo
```

**Azure Monitor Integration**

```bash
# Azure Log Analytics streaming
az monitor log-analytics query --workspace "my-workspace" \
  --analytics-query "AppTraces | where TimeGenerated > ago(1h)" \
  --output tsv | gonzo

# Container Instances monitoring  
az container logs --resource-group mygroup --name mycontainer --follow | gonzo
```

**Google Cloud Logging**

```bash
# Cloud Logging streaming
gcloud logging tail "resource.type=cloud_function" --format="value(textPayload)" | gonzo

# GKE cluster monitoring
gcloud container clusters get-credentials my-cluster
kubectl logs -f deployment/my-app | gonzo
```

### Power User Techniques

#### Multi-Terminal Analysis

**Distributed Investigation Setup**

```bash
# Terminal 1: Real-time error monitoring
gonzo -f /var/log/app.log --follow | grep ERROR | gonzo

# Terminal 2: Performance monitoring  
gonzo -f /var/log/app.log --follow | grep -E "(slow|timeout|duration.*[5-9])" | gonzo

# Terminal 3: Security monitoring
sudo gonzo -f /var/log/auth.log --follow

# Terminal 4: Infrastructure monitoring
gonzo -f /var/log/syslog --follow | grep -E "(error|warning|critical)" | gonzo
```

**Comparative Analysis**

```bash
# Compare current vs previous period
# Terminal 1: Current hour
tail -f /var/log/app.log | grep "$(date +%H):" | gonzo

# Terminal 2: Previous hour  
grep "$(date -d '1 hour ago' +%H):" /var/log/app.log | gonzo

# Terminal 3: Same time yesterday
grep "$(date -d '1 day ago' +%Y-%m-%d)" /var/log/app.log.1 | gonzo
```

#### Automation and Scripting

**Automated Monitoring Scripts**

```bash
#!/bin/bash
# smart-monitor.sh - Intelligent log monitoring

LOG_FILE="/var/log/app.log"
ERROR_THRESHOLD=10
ALERT_EMAIL="admin@company.com"

# Function to check error rate
check_error_rate() {
    local error_count=$(tail -n 100 "$LOG_FILE" | grep -c ERROR)
    if [ "$error_count" -gt "$ERROR_THRESHOLD" ]; then
        echo "High error rate detected: $error_count errors in last 100 lines"
        # Launch Gonzo with error focus
        tail -f "$LOG_FILE" | grep ERROR | gonzo --ai-model="gpt-4" &
        
        # Send alert
        echo "High error rate: $error_count errors" | mail -s "Alert: High Error Rate" "$ALERT_EMAIL"
    fi
}

# Continuous monitoring
while true; do
    check_error_rate
    sleep 300  # Check every 5 minutes
done
```

**Performance Analysis Automation**

```bash
#!/bin/bash  
# performance-analyzer.sh

analyze_performance() {
    local log_file="$1"
    local output_file="/tmp/perf-analysis-$(date +%Y%m%d-%H%M%S).txt"
    
    echo "Performance Analysis Report" > "$output_file"
    echo "Generated: $(date)" >> "$output_file"
    echo "=========================" >> "$output_file"
    
    # Extract slow operations
    grep -E "duration.*[5-9][0-9]{3}" "$log_file" | \
        gonzo --test-mode > /tmp/gonzo-perf.out
    
    # Summary statistics
    echo "Slow Operations Summary:" >> "$output_file"
    grep -c "duration.*[5-9][0-9]{3}" "$log_file" >> "$output_file"
    
    # Top slow endpoints
    echo "Top Slow Endpoints:" >> "$output_file"
    grep -oE "endpoint.*[\"'][^\"']*[\"'].*duration.*[5-9][0-9]{3}" "$log_file" | \
        sort | uniq -c | sort -nr | head -10 >> "$output_file"
    
    echo "Analysis saved to: $output_file"
}

# Usage: ./performance-analyzer.sh /var/log/app.log
analyze_performance "$1"
```

#### AI-Enhanced Analysis Workflows

**Intelligent Problem Detection**

```bash
# Set up AI-enhanced monitoring
export OPENAI_API_KEY="your-key-here"
gonzo -f /var/log/app.log --follow --ai-model="gpt-4"

# AI-guided investigation workflow:
# 1. Let Gonzo run for 10-15 minutes to collect data
# 2. Press Enter on Counts panel for pattern analysis
# 3. Identify suspicious patterns in the heatmap
# 4. Filter for specific pattern: /pattern.*from.*analysis
# 5. Press 'i' on representative log entry
# 6. Ask AI: "Is this a serious issue? What should I investigate?"
# 7. Follow AI recommendations for deeper analysis
```

**Automated Incident Analysis**

```bash
#!/bin/bash
# ai-incident-analyzer.sh

analyze_incident() {
    local start_time="$1"
    local end_time="$2"
    local log_file="$3"
    
    # Extract incident timeframe
    sed -n "/$start_time/,/$end_time/p" "$log_file" > /tmp/incident-logs.txt
    
    # Analyze with Gonzo + AI
    echo "Analyzing incident from $start_time to $end_time..."
    gonzo -f /tmp/incident-logs.txt --ai-model="gpt-4" &
    
    sleep 30  # Let analysis run
    
    # Generate AI report
    echo "Incident Analysis Report" > /tmp/incident-report.txt
    echo "Timeframe: $start_time to $end_time" >> /tmp/incident-report.txt
    
    # Add manual AI queries here based on your findings
    echo "Review the Gonzo analysis and use 'i' key for AI insights"
}

# Usage: ./ai-incident-analyzer.sh "2024-01-15 09:10" "2024-01-15 09:40" /var/log/app.log
analyze_incident "$1" "$2" "$3"
```

### Troubleshooting Advanced Scenarios

#### Performance Issues

**High CPU Usage**

```bash
# Symptoms: Gonzo consuming high CPU
# Solutions:

# 1. Reduce update frequency
gonzo -f high-volume.log --follow --update-interval=10s

# 2. Increase buffer size (fewer refreshes)
gonzo -f busy.log --follow --log-buffer=10000

# 3. Pre-filter data
tail -f /var/log/busy.log | grep -E "(ERROR|WARN)" | gonzo

# 4. Use simpler regex patterns
# Instead of: .*error.*database.*timeout.*
# Use: error.*database
```

**Memory Issues**

```bash
# Symptoms: Gonzo using too much memory
# Solutions:

# 1. Reduce memory size
gonzo -f app.log --memory-size=5000

# 2. Reduce log buffer
gonzo -f app.log --log-buffer=1000

# 3. Reset data periodically
# Press 'r' to reset data when memory usage gets high

# 4. Process logs in batches
split -l 5000 huge.log batch_
for batch in batch_*; do
    gonzo -f "$batch"
done
```

#### Complex Log Format Issues

**Mixed Format Handling**

```bash
# When logs contain multiple formats:
# JSON + plain text + logfmt

# Gonzo handles this automatically, but for optimization:

# 1. Separate by format if possible
grep "^{" mixed.log | gonzo   # JSON lines only
grep -v "^{" mixed.log | gonzo # Non-JSON lines

# 2. Use format-specific filtering
/"level":"error"              # JSON format
/level=error                  # Logfmt format
/\[ERROR\]                   # Plain text format
```

**Custom Timestamp Formats**

```bash
# For unusual timestamp formats
# Gonzo auto-detects most formats, but for custom formats:

# Use filtering to normalize timestamps
sed 's/CustomTimestamp:\([0-9-]*\)/\1/' custom.log | gonzo

# Or filter by custom timestamp patterns
/CustomTimestamp:.*2024-01-15
```

#### Integration Troubleshooting

**OTLP Connection Issues**

```bash
# Debug OTLP connection problems

# 1. Verify Gonzo is listening
netstat -ln | grep 4317
lsof -i :4317

# 2. Test with curl
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{"logs": []}'

# 3. Check firewall settings
sudo ufw status | grep 4317

# 4. Verify OpenTelemetry configuration
# Check collector config for correct endpoints
```

**Container Integration Issues**

```bash
# Docker logs not appearing

# 1. Verify container is running
docker ps | grep my-container

# 2. Check log driver
docker inspect my-container | grep LogDriver

# 3. Test manual log access
docker logs my-container | head -10

# 4. Verify piping works
docker logs -f my-container 2>&1 | cat

# Kubernetes logs not appearing

# 1. Verify pod status
kubectl get pods -l app=my-app

# 2. Check RBAC permissions
kubectl auth can-i get pods --as=system:serviceaccount:default:default

# 3. Test manual access
kubectl logs pod/my-app | head -10
```

### Best Practices Summary

#### 🎯 **Investigation Methodology**

1. **Start broad, narrow progressively** - Begin with all relevant logs, filter by time, then by issue type
2. **Use structured data** - Leverage Attributes panel for correlation
3. **Combine multiple panels** - Word Frequency for discovery, Counts for patterns, Log Viewer for details
4. **Document timeline** - Use Home/End navigation to build incident timelines

#### ⚡ **Performance Optimization**

1. **Right-size buffers** - Balance memory usage with analysis depth
2. **Filter early** - Pre-filter at source when possible
3. **Use appropriate update intervals** - Slower for high-volume, faster for real-time
4. **Reset periodically** - Use 'r' to clear memory in long sessions

#### 🔧 **Configuration Management**

1. **Environment-specific configs** - Different settings for dev vs prod
2. **Save common patterns** - Document frequently used regex filters
3. **Automate repetitive tasks** - Create scripts for common analysis workflows
4. **Version control configs** - Track configuration changes

#### 🤖 **AI Integration**

1. **Use AI for complex patterns** - Let AI identify anomalies you might miss
2. **Ask specific questions** - Guide AI analysis with targeted queries
3. **Validate AI insights** - Cross-check AI conclusions with data
4. **Document AI findings** - Save useful AI analysis for future reference

***

**You are now a Gonzo power user!** 🚀 These advanced techniques will help you conduct sophisticated log analysis, handle complex scenarios, and integrate Gonzo into any workflow. The combination of systematic methodology, performance optimization, and AI assistance makes you capable of tackling any log analysis challenge.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.controltheory.com/backup/user-guide/detailed-usage-guide.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.