Input Methods Overflow

Gonzo offers multiple powerful ways to feed log data into its analysis engine. From simple file analysis to real-time OpenTelemetry streams, this guide covers all input methods and their optimal use cases.

Input Method Overview

Gonzo supports four primary input methods, each optimized for different scenarios:

Method	Best For	Command Example	Real-time?
📁 File Input	Static log analysis, debugging	gonzo -f app.log	No
📈 File Following	Real-time monitoring	gonzo -f app.log --follow	Yes
🔄 Stdin Piping	Integration with tools	kubectl logs -f pod | gonzo	Yes
🌐 OTLP Receiver	OpenTelemetry integration	gonzo --otlp-enabled	Yes

File Input Methods

Single File Analysis

Perfect for debugging specific issues or analyzing archived logs:

# Basic file analysis
gonzo -f application.log

# Absolute paths
gonzo -f /var/log/nginx/access.log

# Relative paths  
gonzo -f logs/app.log

# Compressed files (via piping)
zcat app.log.gz | gonzo
gunzip -c error.log.gz | gonzo

Best Practices:

• Start with single files to understand the data structure

• Use absolute paths to avoid "file not found" errors

• Check file permissions if Gonzo can't access the file

Multiple File Analysis

Combine logs from related services for comprehensive analysis:

# Multiple specific files
gonzo -f api.log -f database.log -f cache.log

# Mix different log types
gonzo -f application.log -f /var/log/syslog -f error.log

# Different directories
gonzo -f /var/log/app.log -f /home/user/debug.log

Related Services

Microservices Architecture:

# Analyze all service logs together
gonzo -f api-gateway.log -f user-service.log -f payment-service.log

# Include infrastructure logs
gonzo -f app.log -f nginx.log -f postgres.log

Benefits:

• Correlate events across services

• See timing relationships

• Identify cascade failures

Different Severities

Glob Pattern Input

Automatically include multiple files using pattern matching:

# All .log files in current directory
gonzo -f "*.log"

# All log files in a directory
gonzo -f "/var/log/*.log"

# Specific patterns
gonzo -f "/var/log/app-*.log"
gonzo -f "/logs/**/*.log"

# Multiple patterns
gonzo -f "/var/log/app*.log" -f "/var/log/nginx*.log"

Glob Pattern Examples:

Pattern	Matches	Use Case
"*.log"	All .log files in current dir	Development debugging
"/var/log/*.log"	All .log files in /var/log	System administration
"app-*.log"	app-2024.log, app-error.log	Application-specific logs
"/logs/**/*.log"	All .log files recursively	Complex log hierarchies
"*-{error,warn}.log"	Files ending in -error.log or -warn.log	Specific severity levels

Important: Always quote glob patterns to prevent shell expansion. Use "*.log" not *.log.

Advanced Glob Techniques:

# Rotated log files
gonzo -f "/var/log/app.log*"  # app.log, app.log.1, app.log.2, etc.

# Date-based logs
gonzo -f "/logs/app-2024-*.log"  # app-2024-01-15.log, etc.

# Multiple services with patterns
gonzo -f "/var/log/{nginx,apache}/*.log"

# Exclude certain files (use find + piping)
find /var/log -name "*.log" ! -name "*debug*" -exec gonzo -f {} +

Real-Time File Following

Monitor logs as they grow, similar to tail -f:

Basic Following

# Follow a single file
gonzo -f /var/log/app.log --follow

# Follow multiple files
gonzo -f app.log -f error.log --follow

# Follow with glob patterns
gonzo -f "/var/log/*.log" --follow

How Following Works:

• Gonzo monitors files for changes using inotify (Linux) or similar mechanisms

• New log entries appear automatically in the interface

• Handles log rotation gracefully

• Efficiently processes only new content

Advanced Following Scenarios

Application Monitoring

Development Environment:

# Follow application logs during development
gonzo -f logs/app.log --follow

# Include error logs
gonzo -f logs/app.log -f logs/error.log --follow

# With configuration
gonzo -f logs/*.log --follow --log-buffer=5000

Production Monitoring:

# Monitor production logs
gonzo -f /var/log/myapp/*.log --follow

# Include system logs
gonzo -f /var/log/myapp.log -f /var/log/syslog --follow

System Administration

Following Best Practices

Performance Optimization:

# Large files - adjust buffer size
gonzo -f huge.log --follow --log-buffer=10000

# High-frequency logs - slower updates
gonzo -f busy.log --follow --update-interval=5s

# Memory-constrained systems
gonzo -f app.log --follow --memory-size=5000

Monitoring Multiple Sources:

# Start with most important logs
gonzo -f app.log --follow

# Add more sources gradually
gonzo -f app.log -f error.log --follow

# Full monitoring setup
gonzo -f "/var/log/app/*.log" --follow --log-buffer=5000 --update-interval=2s

Stdin Processing (Piping)

Integrate Gonzo with existing tools and command pipelines:

Basic Piping

# Basic piping from other commands
cat application.log | gonzo

# Process compressed logs
zcat app.log.gz | gonzo
gunzip -c logs.gz | gonzo

# Stream from remote systems
ssh server "cat /var/log/app.log" | gonzo

Real-Time Streaming

Traditional Unix Tools:

# Stream with tail
tail -f /var/log/app.log | gonzo

# Multiple files with tail
tail -f /var/log/*.log | gonzo

# Advanced tailing with multitail
multitail -f /var/log/app.log /var/log/error.log | gonzo

System Monitoring:

# System logs
sudo tail -f /var/log/syslog | gonzo

# Journal logs (systemd)
journalctl -f | gonzo

# Specific service logs
journalctl -f -u nginx | gonzo

Container Integration

Docker

Single Container:

# Current container logs
docker logs my-container | gonzo

# Follow container logs
docker logs -f my-container 2>&1 | gonzo

# Multiple containers
docker logs -f container1 container2 2>&1 | gonzo

Docker Compose:

# All services
docker-compose logs -f | gonzo

# Specific services
docker-compose logs -f web api database | gonzo

# With timestamps
docker-compose logs -f -t | gonzo

Kubernetes

Advanced Piping Scenarios

Network Sources:

# Remote syslog
nc -l 514 | gonzo

# SSH tunneling
ssh -L 8514:logserver:514 server "nc localhost 8514" | gonzo

# HTTP log streaming
curl -s http://logserver/stream | gonzo

Complex Processing:

# Filter before Gonzo
tail -f /var/log/app.log | grep -v DEBUG | gonzo

# Transform data
tail -f /var/log/app.log | sed 's/old/new/g' | gonzo

# Multiple sources combined
( tail -f app.log & tail -f error.log ) | gonzo

Input Method Selection Guide

Choose the right input method based on your scenario:

Development & Debugging

# Local development - single application
gonzo -f logs/app.log --follow

# Debugging specific issues
gonzo -f error.log

# Multiple local services
gonzo -f "logs/*.log" --follow

Production Monitoring

# Container environments
kubectl logs -f deployment/app | gonzo

# Traditional servers
gonzo -f "/var/log/app/*.log" --follow

# OpenTelemetry-enabled applications
gonzo --otlp-enabled

System Administration

# System health monitoring
sudo gonzo -f "/var/log/{syslog,auth.log,kern.log}" --follow

# Security analysis
sudo tail -f /var/log/auth.log | gonzo

# Performance investigation
gonzo -f "/var/log/nginx/*.log"

Integration Scenarios

# CI/CD pipelines
build-logs | gonzo

# Monitoring alerts
alert-stream | gonzo

# Log aggregation
rsyslog-stream | gonzo

Performance Considerations

Buffer Sizing

Optimize for your log volume and available memory:

# High-volume logs
gonzo -f busy.log --log-buffer=10000 --memory-size=20000

# Memory-constrained systems
gonzo -f app.log --log-buffer=1000 --memory-size=5000

# Balanced configuration
gonzo -f app.log --log-buffer=2000 --memory-size=15000

Update Intervals

Balance responsiveness with system resources:

# Real-time responsiveness (default)
gonzo -f app.log --follow --update-interval=1s

# Balanced performance
gonzo -f app.log --follow --update-interval=2s

# Resource conservation
gonzo -f app.log --follow --update-interval=5s

Input-Specific Optimizations

Input Method	Optimization	Configuration
File Following	Use larger buffers	--log-buffer=5000
Stdin Piping	Ensure smooth pipeline	`command
OTLP	Tune batch sizes	OpenTelemetry collector config
Multiple Files	Balance file count vs performance	Combine related files

Troubleshooting Input Issues

File Access Problems

# Permission denied
sudo gonzo -f /var/log/secure.log

# File not found with globs
gonzo -f "/path/to/*.log"  # Use quotes!

# Check file permissions
ls -la logfile.log

Following Issues

# File rotation problems
# Gonzo handles this automatically, but check:
ls -la /var/log/app.log*

# High CPU usage
# Reduce update frequency:
gonzo -f app.log --follow --update-interval=5s

Piping Problems

# Broken pipe errors
# Ensure Gonzo can handle the data rate:
slow-producer | gonzo --log-buffer=10000

# Buffer full issues
# Increase buffer size or filter at source:
noisy-source | grep -v DEBUG | gonzo

Advanced Input Patterns

Hybrid Approaches

Combine multiple input methods for comprehensive monitoring:

# OTLP + file backup
gonzo --otlp-enabled -f backup.log --follow

# Real-time + historical analysis
# Terminal 1: gonzo -f app.log --follow
# Terminal 2: gonzo -f historical-logs.log

# Primary + secondary sources
gonzo -f primary.log --follow & kubectl logs -f secondary | gonzo

Custom Input Scripts

Create wrapper scripts for complex scenarios:

#!/bin/bash
# monitor-all.sh
exec gonzo \
  -f "/var/log/app/*.log" \
  -f "/var/log/nginx/*.log" \
  --follow \
  --log-buffer=5000 \
  --update-interval=2s