Format Detection

Gonzo's intelligent format detection automatically recognizes and parses different log formats, extracting structured data for powerful analysis. From JSON to plain text, Gonzo adapts to your logs without configuration.

circle-info

Automatic Operation: Format detection works automatically - no configuration required. Gonzo analyzes each log entry and applies the appropriate parsing strategy in real-time.

Supported Log Formats

Gonzo intelligently detects and processes multiple log formats simultaneously:

Format
Detection Method
Structured Data
Best For

JSON

Valid JSON syntax

Complete object parsing

Modern applications, microservices

Logfmt

Key=value pairs

Field extraction

Go applications, Heroku-style logs

Plain Text

Pattern recognition

Smart field extraction

Traditional applications, system logs

Mixed Formats

Per-line analysis

Format-specific parsing

Legacy systems, log aggregation

JSON Log Format

Automatic JSON Detection

Gonzo automatically detects JSON logs and extracts all fields:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "error",
  "service": "payment-api",
  "message": "Database connection timeout",
  "user_id": "12345",
  "request_id": "req_abc123",
  "duration": 30000,
  "endpoint": "/api/payment/process",
  "error": {
    "type": "TimeoutError",
    "details": "Connection pool exhausted"
  },
  "metadata": {
    "trace_id": "trace_xyz789",
    "span_id": "span_456def"
  }
}

What Gonzo Extracts:

  • All top-level fields appear in Attributes panel

  • Nested objects are flattened (e.g., error.type, metadata.trace_id)

  • Arrays are handled intelligently

  • Data types are preserved (strings, numbers, booleans)

JSON Processing Features

Automatic Field Detection:

Search and Filter by Fields:

JSON Best Practices

Optimize Your JSON Logs:

Logfmt Format

Automatic Logfmt Detection

Gonzo detects logfmt (key=value) format commonly used by Go applications:

Parsing Features:

  • Key-value extraction - All key=value pairs become searchable fields

  • Quoted value support - Handles msg="complex message with spaces"

  • Type inference - Numbers and booleans detected automatically

  • Space handling - Robust parsing of various spacing patterns

Logfmt Processing Examples

Input Logfmt:

Extracted Attributes:

Filtering Examples:

Logfmt Optimization Tips

Structure Your Logfmt Logs:

Plain Text Format

Intelligent Plain Text Processing

Gonzo analyzes plain text logs and extracts structured information using pattern recognition:

Automatic Extraction:

  • Timestamp detection - Various timestamp formats

  • Log level identification - ERROR, WARN, INFO, DEBUG, etc.

  • Service name extraction - Service or component names

  • Contextual information - IDs, durations, endpoints extracted via patterns

Plain Text Pattern Recognition

Supported Timestamp Formats:

What Gonzo Extracts:

  • Normalizes all timestamps for time-series analysis

  • Maintains original format in log display

  • Enables time-based filtering and sorting

Enhanced Plain Text Processing

Contextual Information Extraction:

Mixed Format Handling

Multi-Format Log Sources

Gonzo handles mixed format sources intelligently:

Per-Line Format Detection:

  • Each log line analyzed independently

  • Appropriate parser applied automatically

  • Consistent field extraction across formats

  • Unified analysis in Attributes panel

Format Transition Handling

Real-World Scenario: Application log format changes over time

Gonzo Handles This Seamlessly:

  • Detects format change automatically

  • Continues processing without interruption

  • Maintains consistent analysis across formats

  • No configuration changes required

Format Detection Optimization

Performance Considerations

Optimizing Format Detection:

Memory Usage Optimization

Field Extraction Memory Management:

Format-Specific Features

JSON-Specific Enhancements

Advanced JSON Processing:

Logfmt-Specific Enhancements

Logfmt Processing Optimizations:

Plain Text Intelligence

Advanced Pattern Learning:

Troubleshooting Format Detection

Common Issues and Solutions

Field Not Extracted:

Performance Issues:

Mixed Format Confusion:

Best Practices

🎯 Optimize Log Format Design

  1. Choose appropriate formats - JSON for rich structure, logfmt for simplicity, plain text for human readability

  2. Maintain consistency - Use the same field names, timestamp formats, and log levels across your application

  3. Include context - Always include request IDs, user IDs, and correlation identifiers

  4. Structure your data - Group related fields in objects (JSON) or use consistent prefixes (logfmt)

📊 Maximize Analysis Value

  1. Use structured logging - JSON and logfmt provide much richer analysis than plain text

  2. Include performance metrics - Add duration, memory usage, and other quantitative data

  3. Add business context - Include user IDs, feature flags, and business-relevant information

  4. Maintain type consistency - Use numbers for numeric data, booleans for flags

Performance Optimization

  1. Validate JSON syntax - Invalid JSON significantly slows processing

  2. Use consistent patterns - Helps Gonzo optimize parsing strategies

  3. Avoid extreme nesting - Deep JSON objects can impact performance

  4. Monitor resource usage - Adjust buffer sizes based on log volume and system capacity

🔍 Debugging Format Issues

  1. Test with small samples - Verify format detection with simple examples

  2. Check encoding and line endings - Ensure logs are UTF-8 with Unix line endings

  3. Monitor field extraction - Verify expected fields appear in Attributes panel

  4. Use format-specific tools - Validate JSON with jq, test logfmt patterns manually

Real-World Format Examples

Learn how to create custom formats for popular tools:

What's Next?

Now that you understand format detection, explore how it enhances other Gonzo features:

  • Log Analysis - Structured data improves pattern detection and analytics

  • AI Integration - Better format detection provides richer context for AI analysis

  • Configuration - Tune format detection settings for your specific logs

  • Integration Examples - See format detection in real-world scenarios

Or start optimizing your log formats immediately:


You now understand how Gonzo intelligently processes any log format! 🎯 From automatic JSON parsing to intelligent plain text recognition, format detection ensures you get maximum analytical value from any log source without manual configuration.

Last updated