Log Analysis

Discover patterns, trends, and anomalies in your logs with Gonzo's advanced analytical capabilities. From automatic pattern detection to time-series analysis, these features reveal insights that would be impossible to find manually.

Access Point: Most log analysis features are accessed through the Counts panel (bottom-right). Press Enter on the Counts panel to open the detailed analysis modal.

Analysis Overview

Gonzo's log analysis combines multiple sophisticated algorithms to provide comprehensive insights:

Feature

Algorithm

What It Reveals

Best For

Pattern Detection

Drain3 clustering

Recurring log templates

Finding common issues

Time-Series Analysis

60-minute rolling window

Trends over time

Understanding incident timing

Heatmap Visualization

ASCII intensity mapping

Activity patterns by severity

Visual pattern recognition

Service Distribution

Real-time aggregation

Which services log what

Multi-service debugging

Anomaly Detection

Statistical analysis

Unusual patterns

Proactive problem identification

The Analysis Dashboard

Counts Panel Overview

The Counts panel (bottom-right) provides your gateway to advanced analysis:

┌─ COUNTS ─────────────────────────────┐
│ Severity Distribution:               │
│ ERROR  █████████████████████   (45%) │
│ WARN   ██████████████          (30%) │
│ INFO   ███████                 (20%) │
│ DEBUG  ██                       (5%) │
│                                      │
│ Total Entries: 2,847                 │
│ Time Span: 2h 15m                    │
│ Entries/min: 21.2                    │
│ Pattern Count: 23                    │
│                                      │
│ Press Enter for detailed analysis... │
└─────────────────────────────────────┘

Key Metrics Explained:

Severity Distribution - Percentage breakdown by log level with visual bars
Total Entries - Count of all processed log entries in current session
Time Span - Duration from first to last log entry
Entries/min - Average logging frequency (useful for capacity planning)
Pattern Count - Number of unique patterns detected by drain3 algorithm

Press Enter on the Counts panel to access the comprehensive analysis modal:

┌─ LOG ANALYSIS (Press ESC to close) ──────────────────────────────┐
│                                                                  │
│ Time-Series Heatmap (60-minute rolling window):                  │
│ ┌────────────────────────────────────────────────────────────┐   │
│ │Time: 60  50  40  30  20  10  0 (minutes ago)              │   │  
│ │ERROR ████░░██████░░░░████████████████ High intensity      │   │
│ │WARN  ░░██████░░████░░░░██████░░░░░░░░ Medium intensity    │   │
│ │INFO  ░░░░░░░░░░░░░░░░████░░░░░░░░░░░░ Low intensity       │   │
│ │DEBUG ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ Minimal activity    │   │
│ └────────────────────────────────────────────────────────────┘   │
│                                                                  │
│ Top 3 Patterns by Severity:                                     │
│ ┌────────────────────────────────────────────────────────────┐   │
│ │ ERROR:                                                     │   │
│ │ 1. Database connection timeout (247 occurrences)          │   │
│ │ 2. User authentication failed (156 occurrences)           │   │
│ │ 3. API rate limit exceeded (89 occurrences)               │   │
│ │                                                            │   │
│ │ WARN:                                                      │   │
│ │ 1. Slow query detected (324 occurrences)                  │   │
│ │ 2. Memory usage high (198 occurrences)                    │   │
│ │ 3. Cache miss rate elevated (156 occurrences)             │   │
│ └────────────────────────────────────────────────────────────┘   │
│                                                                  │
│ Service Distribution:                                            │
│ web-api: 1,247 entries (44%)  database: 892 entries (31%)      │
│ auth: 654 entries (23%)       cache: 54 entries (2%)           │
│                                                                  │
│ Navigation: ↑/↓ Scroll, Mouse Wheel, ESC to close               │
└──────────────────────────────────────────────────────────────────┘

Time-Series Heatmap Analysis

Understanding the Heatmap

The time-series heatmap is one of Gonzo's most powerful visual analysis tools:

Time Axis (Horizontal):

Shows last 60 minutes in 1-minute buckets
Reading: 60 = 60 minutes ago, 0 = current minute
Updates in real-time as new logs arrive

Severity Axis (Vertical):

Each row represents a different log severity level
Separate tracking for ERROR, WARN, INFO, DEBUG, etc.
Independent scaling per severity level

Intensity Indicators:

░ = Low activity (1-25% of max for this severity)
▒ = Medium activity (25-50% of max)
▓ = High activity (50-75% of max)  
█ = Very high activity (75-100% of max)

Reading Heatmap Patterns

Identifying When Problems Started:

Time: 60  50  40  30  20  10  0
ERROR ░░░░░░░░████████████░░░░░░
WARN  ░░░░░░████████████████░░░░

Analysis:

Problem started around 40 minutes ago
Peak error activity 30-20 minutes ago
Warnings preceded errors (early warning signs)
System appears to be recovering now

Use Case: Incident timeline reconstruction

Daily Performance Cycles:

Time: 60  50  40  30  20  10  0
ERROR ░░░░░░░░░░░░░░░░░░░░░░░░░░
WARN  ░░░░████░░░░████░░░░████░░
INFO  ████░░░░████░░░░████░░░░░░

Analysis:

Regular 20-minute cycles in warnings
High info activity alternating with warnings
Suggests scheduled job or batch processing
No critical errors, normal operational pattern

Use Case: Capacity planning and optimization

System Failure Propagation:

Time: 60  50  40  30  20  10  0
ERROR ░░░░░░░░░░█░░██████████████
WARN  ░░░░░░░░█████████████████░
INFO  ░░░░░░████████████░░░░░░░░

Analysis:

Single error triggered cascade
Warnings spread quickly after initial error
Info logs dropped off (services became unresponsive)
Classic cascade failure pattern

Use Case: System resilience analysis

Heatmap Best Practices

🔍 Investigation Techniques:

Start wide, zoom in - Look for obvious patterns first
Compare severity levels - How do different levels correlate?
Identify inflection points - When did patterns change?
Look for cycles - Are there recurring patterns?

⚡ Quick Analysis:

# Quick heatmap analysis workflow:
1. Press Enter on Counts panel
2. Scan heatmap for obvious spikes or patterns
3. Note correlation between severity levels
4. Identify time ranges for deeper investigation
5. Use time information to filter main log view

Pattern Detection with Drain3

How Drain3 Works

Gonzo uses the Drain3 algorithm for automatic pattern detection:

What Drain3 Does:

Clusters similar log entries into pattern templates
Extracts variable parts (IDs, timestamps, values) from static text
Maintains pattern counts in real-time
Adapts to new patterns as they appear

Example Pattern Detection:

# Original log entries:
"User 12345 login failed"
"User 67890 login failed"  
"User 54321 login failed"

# Drain3 pattern:
"User <ID> login failed" (3 occurrences)

Pattern Analysis Features

Top Patterns by Severity:

In the analysis modal, patterns are grouped by severity level:

ERROR Patterns:
1. Database connection timeout (247 occurrences)
2. User authentication failed (156 occurrences)  
3. API rate limit exceeded (89 occurrences)

WARN Patterns:
1. Slow query detected (324 occurrences)
2. Memory usage high (198 occurrences)
3. Cache miss rate elevated (156 occurrences)

What This Tells You:

Focus on High-Count Patterns:

Database connection timeout (247) - Critical infrastructure issue
User authentication failed (156) - Security/user experience impact
Slow query detected (324) - Performance degradation

Analysis Priority:

Address database connectivity first (highest error count)
Investigate authentication system second
Optimize slow queries for long-term performance

Pattern Correlation:

High slow query warnings might be causing database timeouts
Memory usage warnings could be related to query performance
Cache misses might be increasing database load

Investigation Path:

Check if slow queries correlate with connection timeouts
Verify if memory pressure affects query performance
Analyze cache efficiency impact on database load

Pattern Evolution:

Are certain patterns increasing over time?
Do patterns appear in clusters or continuously?
Are new patterns emerging that weren't seen before?

Long-term Monitoring:

Track pattern counts over days/weeks
Identify patterns that are growing vs shrinking
Spot new issues before they become critical

Working with Pattern Data

Pattern-Based Filtering:

Once you identify interesting patterns, use them for focused analysis:

# From analysis modal, identify pattern:
"Database connection timeout"

# Create filter in main view:
/database.*connection.*timeout

# Or use structured filtering:
/error.*database.*timeout

Pattern Evolution Tracking:

# Compare patterns over time:
# 1. Note current top patterns
# 2. Wait 30-60 minutes  
# 3. Check analysis modal again
# 4. See which patterns increased/decreased
# 5. Identify trends and new issues

Service Distribution Analysis

Understanding Service Metrics

The service distribution section shows which services are generating logs:

Service Distribution:
web-api: 1,247 entries (44%)    database: 892 entries (31%)
auth: 654 entries (23%)         cache: 54 entries (2%)

What This Reveals:

Metric

Meaning

Investigation Questions

High Percentage

Service is very active

Is this normal? Performance issue?

Low Percentage

Service is quiet

Is it supposed to be active? Down?

Sudden Changes

Activity shift

What caused the change?

Missing Services

Service not logging

Is it running? Configuration issue?

Service-Based Analysis

Normal Load Patterns:

web-api: 45%    # Frontend traffic - high normal
database: 30%   # Backend queries - moderate normal  
auth: 20%       # Authentication - low-moderate normal
cache: 5%       # Cache operations - low normal

Red Flags:

Database suddenly becomes 60%+ (performance issue)
Auth drops to 0% (service down)
New unknown service appears with high percentage

Troublesome Patterns:

# Before incident:
web-api: 40%, database: 35%, auth: 20%, cache: 5%

# During incident:
database: 70%, web-api: 25%, auth: 3%, cache: 2%

Analysis:

Database percentage spike indicates it's struggling
Web-api percentage drop suggests it's being blocked
Auth/cache drops suggest they can't reach database
Classic database bottleneck pattern

Health Indicators:

Consistent percentages = Healthy steady state
Gradual changes = Normal load evolution
Sudden spikes = Performance issues or load shifts
Disappearing services = Potential outages

Monitoring Strategy:

Note baseline percentages for each service
Watch for deviations > 20% from baseline
Investigate services with sudden percentage changes
Correlate with error patterns from same services

Advanced Analysis Techniques

Correlation Analysis

Cross-Reference Multiple Data Points:

# Workflow for comprehensive analysis:
1. Heatmap: When did issues occur?
2. Patterns: What specific issues happened?
3. Services: Which services were affected?
4. Main logs: Examine specific instances

# Example correlation:
# Heatmap shows error spike at 30 min ago
# Patterns show "database timeout" as top error
# Services show database % increased during that time
# Conclusion: Database performance issue at 30 min ago

Trend Identification

Long-Term Pattern Analysis:

# Daily analysis routine:
1. Open analysis modal first thing
2. Note current top patterns and service distribution
3. Compare with yesterday's patterns (manual tracking)
4. Identify:
   - New patterns that appeared
   - Patterns that increased in frequency
   - Services with changed activity levels
5. Investigate significant changes

Performance Baseline Establishment

Creating Performance Baselines:

# Week 1: Establish baselines
# Track normal patterns:
# - Typical service distribution percentages
# - Common pattern frequencies  
# - Normal heatmap intensity levels
# - Typical entries/minute rates

# Week 2+: Compare against baselines
# Alert on:
# - Service distribution changes > 25%
# - New high-frequency error patterns
# - Unusual heatmap spike patterns
# - Entry rate changes > 50%

Analysis Workflows

Incident Investigation Workflow

# 1. Get timeline overview
# Press Enter on Counts panel
# Check heatmap for incident timing

# 2. Identify problem patterns  
# Look at top error patterns
# Note pattern frequencies

# 3. Determine affected services
# Check service distribution
# Identify services with unusual activity

# 4. Deep dive investigation
# Filter main log view by time: /2024-01-15.*14:[2-3][0-9]
# Filter by pattern: /database.*connection.*timeout
# Examine specific log entries

# 5. Root cause analysis
# Correlate timing + patterns + services
# Look for cascade failure indicators
# Check for external factors (deployments, load changes)

Performance Monitoring Workflow

# 1. Establish current state
# Review heatmap for activity patterns
# Note baseline service distributions
# Check current pattern frequencies

# 2. Set monitoring expectations
# Note normal intensity levels
# Track typical pattern counts
# Baseline entries/minute rate

# 3. Continuous monitoring
# Check analysis modal every 30-60 minutes
# Look for deviations from baseline
# Track trend directions (improving/degrading)

# 4. Proactive investigation
# Investigate patterns with increasing frequency
# Monitor services with changing activity levels
# Use early warning patterns (WARN level issues)

Capacity Planning Workflow

# 1. Long-term trend analysis
# Track entries/minute over weeks
# Monitor service distribution evolution
# Note peak activity periods

# 2. Pattern evolution tracking
# Which patterns are becoming more frequent?
# Are new error patterns emerging?
# How do patterns correlate with load?

# 3. Resource correlation
# Compare log activity with system metrics
# Identify log volume vs performance relationship
# Plan capacity based on log analysis trends

Troubleshooting Analysis Issues

Performance Issues

Analysis Modal Loading Slowly:

# Causes and solutions:
# - Large datasets: Reduce log buffer size
# - Complex patterns: Reset data periodically
# - High update frequency: Increase update interval

# Optimization:
gonzo -f logs.log --log-buffer=5000 --update-interval=5s

Pattern Detection Not Working:

# Common issues:
# - Logs too unstructured: Pattern detection works best with consistent formats
# - High variability: Some logs have too many unique elements
# - Insufficient data: Need meaningful sample size

# Solutions:
# - Use structured logs (JSON, logfmt) when possible
# - Filter out highly variable elements before analysis
# - Allow more time for pattern establishment

Interpretation Issues

Unclear Heatmap Patterns:

# If heatmap seems random:
# - Check if time range is appropriate
# - Verify log timestamps are accurate
# - Consider if activity is actually irregular

# Debugging steps:
# 1. Focus on single severity level
# 2. Use smaller time windows
# 3. Check raw log timestamps

Misleading Service Distribution:

# Common causes:
# - Inconsistent service name extraction
# - Missing service identifiers in logs
# - Mixed log formats affecting parsing

# Solutions:
# - Standardize service naming in logs
# - Use structured logging with consistent fields
# - Manually filter by service if needed

Best Practices

🎯 Effective Analysis Strategies

Start with overview, drill down - Use heatmap for big picture, patterns for specifics
Correlate multiple data sources - Combine timing, patterns, and services
Track trends over time - Compare current state with historical baselines
Focus on high-impact patterns - Prioritize by frequency and severity

📊 Data Quality Optimization

Use structured logging - JSON and logfmt provide better analysis
Consistent field naming - Helps with service distribution accuracy
Meaningful log levels - Proper ERROR/WARN/INFO usage improves analysis
Include context - Service names, trace IDs, and relevant metadata

⚡ Performance Optimization

Right-size analysis windows - Balance detail with performance
Reset data periodically - Prevent memory buildup in long sessions
Filter appropriately - Reduce dataset size for complex analysis
Monitor resource usage - Adjust settings based on system capacity

What's Next?

Now that you understand log analysis, explore these complementary features:

AI Integration - Combine algorithmic analysis with AI insights
Format Detection - Optimize data input for better analysis
Configuration - Tune analysis settings for your needs
Integration Examples - Apply analysis to real-world scenarios

You now have mastery over Gonzo's analytical capabilities! 🚀 The combination of time-series analysis, pattern detection, and service distribution gives you unprecedented insight into log behavior and system health.

Last updated 3 months ago

Analysis Overview

The Analysis Dashboard

Counts Panel Overview

Detailed Analysis Modal

Time-Series Heatmap Analysis

Understanding the Heatmap

Reading Heatmap Patterns

Heatmap Best Practices

Pattern Detection with Drain3

How Drain3 Works

Pattern Analysis Features

Working with Pattern Data

Service Distribution Analysis

Understanding Service Metrics

Service-Based Analysis

Advanced Analysis Techniques

Correlation Analysis

Trend Identification

Performance Baseline Establishment

Analysis Workflows

Incident Investigation Workflow

Performance Monitoring Workflow

Capacity Planning Workflow

Troubleshooting Analysis Issues

Performance Issues

Interpretation Issues

Best Practices

🎯 Effective Analysis Strategies

📊 Data Quality Optimization

⚡ Performance Optimization

What's Next?