# Log Format Issues Troubleshooting guide for log parsing and format detection problems in Gonzo. ### Format Detection Issues #### Logs Not Being Parsed **Symptom**: Structured logs appear as plain text instead of being parsed. **Diagnosis**: Gonzo automatically detects formats per line based on these rules: * **JSON**: Lines starting with `{` * **Logfmt**: Lines containing `key=value` patterns * **Plain text**: Everything else **Solutions**: 1. **Verify JSON is valid** ```bash # Test each line individually head -1 logs.json | jq . # If error, JSON is malformed ``` 2. **Check line starts with `{`** ```bash # JSON must start with opening brace head logs.json # ✅ Good: {"level":"info"... # ❌ Bad: 2024-01-15 {"level":"info"... ``` 3. **Inspect logfmt format** ```bash # Should have key=value pairs head logs.txt # ✅ Good: level=info service=api msg="started" # ❌ Bad: [INFO] service api started ``` 4. **Use custom format for non-standard logs** ```bash gonzo --format=my-custom-format -f logs.txt ``` #### Mixed Format Logs **Symptom**: Some lines parse correctly, others don't. **Explanation**: Gonzo detects format per line, so mixed formats will have inconsistent parsing. **Example**: ``` {"level":"info","msg":"API started"} ← Parsed as JSON level=info msg="processing request" ← Parsed as logfmt [ERROR] Connection timeout ← Plain text ``` **Solutions**: 1. **Accept mixed display** * This is expected behavior * Each format renders appropriately 2. **Pre-filter to single format** ```bash # Extract only JSON lines grep '^{' mixed.log | gonzo # Or only logfmt grep '=' mixed.log | gonzo ``` 3. **Convert to uniform format** ```bash # Convert all to JSON with jq cat mixed.log | jq -R -c '. | fromjson? // {"message": .}' | gonzo ``` #### Attributes Not Extracted **Symptom**: Logs parse but attributes panel is empty. **Causes & Solutions**: 1. **Plain text logs have no structured attributes** * Plain text can't be parsed into fields * Only JSON and logfmt have attributes 2. **Check JSON structure** ```bash # Verify JSON has expected fields head -1 logs.json | jq . # Should show key-value pairs ``` 3. **Nested JSON** ```json { "log": { "level": "info", "service": "api" } } ``` * Gonzo extracts nested attributes * May need to check nested paths 4. **Missing common attribute names** * Gonzo looks for common fields: `level`, `service`, `host`, etc. * Custom fields may not be highlighted ### JSON Issues #### Malformed JSON **Symptom**: JSON logs show as plain text or cause errors. **Common JSON Issues**: 1. **Trailing commas** ```json {"level":"info","msg":"test",} ❌ Invalid {"level":"info","msg":"test"} ✅ Valid ``` 2. **Single quotes instead of double** ```json {'level':'info'} ❌ Invalid {"level":"info"} ✅ Valid ``` 3. **Unescaped quotes in strings** ```json {"msg":"He said "hi""} ❌ Invalid {"msg":"He said \"hi\""} ✅ Valid {"msg":"He said 'hi'"} ✅ Valid (single quotes in string) ``` 4. **Missing quotes on keys** ```json {level:"info"} ❌ Invalid {"level":"info"} ✅ Valid ``` **Validate & Fix**: ```bash # Validate JSON cat logs.json | jq . > /dev/null # If errors, shows line number # Pretty-print to find issues jq . logs.json # Fix and re-format jq -c . logs.json > fixed.json gonzo -f fixed.json ``` #### Multi-line JSON **Symptom**: JSON objects span multiple lines, not parsed correctly. **Example**: ```json { "level": "info", "message": "test" } ``` **Solution**: Gonzo expects one JSON object per line (JSONL/NDJSON format). ```bash # Compact multi-line JSON to single lines jq -c . pretty.json > compact.json gonzo -f compact.json # Or pipe directly jq -c . pretty.json | gonzo ``` #### JSON with Metadata Prefix **Symptom**: Lines have timestamp or metadata before JSON. **Example**: ``` 2024-01-15 10:30:05 {"level":"info","msg":"test"} ``` **Solution**: Remove prefix before piping to Gonzo: ```bash # Remove timestamp prefix sed 's/^[0-9-]* [0-9:]* //' logs.txt | gonzo # Or use awk awk '{$1=$2=""; print}' logs.txt | gonzo # Extract just JSON part grep -o '{.*}' logs.txt | gonzo ``` #### Escaped JSON in Strings **Symptom**: JSON contains escaped JSON strings. **Example**: ```json {"log":"{\"level\":\"info\",\"msg\":\"test\"}"} ``` **Solution**: ```bash # Unescape inner JSON jq -r '.log | fromjson' logs.json | gonzo # Or handle both levels jq -c '.log | fromjson? // .' logs.json | gonzo ``` ### Logfmt Issues #### Logfmt Not Detected **Symptom**: Key=value logs appear as plain text. **Requirements for logfmt detection**: * Must have `key=value` patterns * Multiple pairs per line * Values can be quoted: `key="value with spaces"` **Example**: ``` level=info service=api user=123 msg="request completed" ✅ Detected INFO service api user 123 request completed ❌ Not logfmt ``` **Solutions**: 1. **Verify format** ```bash # Check for key=value pattern grep -E '\w+=\w+' logs.txt | head ``` 2. **Add more key=value pairs** * Single pair may not trigger detection * Multiple pairs more reliably detected 3. **Use custom format** ```bash gonzo --format=my-logfmt -f logs.txt ``` #### Spaces in Logfmt Values **Symptom**: Values with spaces not parsed correctly. **Examples**: ``` msg=hello world ❌ Breaks: "world" seen as separate key msg="hello world" ✅ Correct: quotes preserve spaces msg=hello\ world ✅ Correct: escape preserves spaces ``` **Solution**: Ensure spaces in values are properly quoted or escaped: ```bash # Fix unquoted spaces (requires log generation fix) # Or accept partial parsing of problematic lines ``` #### Logfmt with Nested Structures **Symptom**: Nested objects in logfmt don't parse well. **Example**: ``` user.id=123 user.name=john ``` **Explanation**: Logfmt is flat by design. Nested structures need JSON. **Solution**: 1. **Accept flat representation** * Gonzo extracts `user.id` and `user.name` as separate attributes 2. **Convert to JSON if needed** ```bash # If you control log format, use JSON for nested data ``` ### Plain Text Issues #### No Structure Extracted from Text Logs **Symptom**: Plain text logs show no attributes. **Explanation**: Plain text logs can't be parsed into structured fields automatically. **Examples**: ``` [2024-01-15 10:30:05] ERROR: Connection failed INFO - api-service - User login successful ``` **Solutions**: 1. **Accept plain text display** * Logs still searchable and analyzable * Just no structured attributes 2. **Create custom format parser** ```bash # Define regex-based parser # See Custom Formats Guide gonzo --format=my-text-format -f logs.txt ``` 3. **Convert logs to structured format** * Modify application to output JSON/logfmt * Use log shipper to add structure (Fluent Bit, Logstash) #### Severity Not Detected in Text Logs **Symptom**: Plain text logs don't show color-coded severity. **Explanation**: Gonzo looks for common severity keywords in text logs: * ERROR, FATAL, CRITICAL → Red * WARN, WARNING → Yellow * INFO → Green * DEBUG → Blue * TRACE → White **Solutions**: 1. **Include severity keywords** ``` [ERROR] Connection failed ✅ Detected Connection failed ❌ Not detected WARN: Low disk space ✅ Detected Disk space is low ❌ Not detected ``` 2. **Use consistent format** * Put severity at start of line * Use standard keywords (ERROR, WARN, INFO, DEBUG) 3. **Create custom format** * Define severity extraction pattern * Map custom levels to standard severities ### OTLP Format Issues #### OTLP Logs Not Appearing **Symptom**: OTLP receiver running but no logs in Gonzo. **Diagnosis**: 1. **Verify receiver is enabled** ```bash # Check Gonzo started with OTLP gonzo --otlp-enabled # Should show listening on ports 4317 and 4318 ``` 2. **Check sender configuration** ```yaml # Verify endpoint in sender endpoint: localhost:4317 # gRPC endpoint: http://localhost:4318/v1/logs # HTTP ``` 3. **Test with curl (HTTP)** ```bash curl -X POST http://localhost:4318/v1/logs \ -H "Content-Type: application/json" \ -d '{"resourceLogs":[]}' # Should return 200 OK ``` 4. **Check for port conflicts** ```bash lsof -i :4317 lsof -i :4318 ``` **Solutions**: See Common Issues - OTLP Receiver for detailed fixes. #### OTLP Attributes Missing **Symptom**: OTLP logs appear but without expected attributes. **Causes**: 1. **Attributes in resource vs log record** * Resource attributes: service.name, host, etc. * Log record attributes: user\_id, request\_id, etc. * Both should be extracted 2. **Verify sender includes attributes** ```python # Ensure attributes are set logger_provider.add_log_record_processor(processor) # Check resource attributes and log attributes ``` 3. **Check attribute names** * Gonzo shows all attributes * May just be named differently than expected ### Custom Format Issues #### Custom Format Not Working **Symptom**: `--format=my-format` shows error or doesn't parse. **Diagnosis**: 1. **Verify format file exists** ```bash ls ~/.config/gonzo/formats/my-format.yaml ``` 2. **Check YAML syntax** ```bash cat ~/.config/gonzo/formats/my-format.yaml # Validate if you have yamllint yamllint ~/.config/gonzo/formats/my-format.yaml ``` 3. **Test with built-in format first** ```bash # Verify custom formats work at all gonzo --format=loki-stream -f test.json ``` **Solutions**: See Custom Formats Guide for: * Format file syntax * Regex patterns * Field mapping * Testing formats #### Regex Not Matching **Symptom**: Custom format regex doesn't extract fields. **Solutions**: 1. **Test regex separately** ```bash # Test your regex pattern echo "sample log line" | grep -E "your-regex-pattern" ``` 2. **Use online regex tester** * Test at regex101.com * Use example log lines * Verify capture groups 3. **Check for special characters** ```yaml # Escape special characters pattern: '\[(\d+)\]' # Brackets escaped ``` 4. **Start simple, iterate** ```yaml # Begin with basic pattern pattern: '(\w+)' # Then add complexity pattern: '(\w+)=(\w+)' # Finally, full pattern pattern: '(\w+)="([^"]*)"' ``` ### Encoding Issues #### Special Characters Garbled **Symptom**: Non-ASCII characters display incorrectly. **Solutions**: 1. **Ensure UTF-8 encoding** ```bash export LANG=en_US.UTF-8 export LC_ALL=en_US.UTF-8 ``` 2. **Check file encoding** ```bash file logs.txt # Should show: UTF-8 Unicode text # Convert if needed iconv -f ISO-8859-1 -t UTF-8 logs.txt > logs_utf8.txt ``` 3. **Terminal font support** * Use font with good Unicode support * JetBrains Mono, Fira Code, Cascadia Code #### Binary or Non-Text Data **Symptom**: Binary data causes display issues. **Solution**: ```bash # Filter out binary data strings logs.bin | gonzo # Or ensure only text logs are processed file logs.txt # Should be "text" not "data" ``` ### Performance with Complex Formats #### Slow Parsing with Complex Regex **Symptom**: Custom format with complex regex causes slowdowns. **Solutions**: 1. **Simplify regex patterns** ```yaml # Instead of: .*?(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z).* # Use: (\d{4}-\d{2}-\d{2}T[\d:.]+Z) ``` 2. **Reduce backtracking** * Avoid nested quantifiers: `(.*)*` * Use possessive quantifiers when possible * Anchor patterns: `^` and \`# Log Format Issues Troubleshooting guide for log parsing and format detection problems in Gonzo. ### Format Detection Issues #### Logs Not Being Parsed **Symptom**: Structured logs appear as plain text instead of being parsed. **Diagnosis**: Gonzo automatically detects formats per line based on these rules: * **JSON**: Lines starting with `{` * **Logfmt**: Lines containing `key=value` patterns * **Plain text**: Everything else **Solutions**: 1. **Verify JSON is valid** ```bash # Test each line individually head -1 logs.json | jq . # If error, JSON is malformed ``` 2. **Check line starts with `{`** ```bash # JSON must start with opening brace head logs.json # ✅ Good: {"level":"info"... # ❌ Bad: 2024-01-15 {"level":"info"... ``` 3. **Inspect logfmt format** ```bash # Should have key=value pairs head logs.txt # ✅ Good: level=info service=api msg="started" # ❌ Bad: [INFO] service api started ``` 4. **Use custom format for non-standard logs** ```bash gonzo --format=my-custom-format -f logs.txt ``` #### Mixed Format Logs **Symptom**: Some lines parse correctly, others don't. **Explanation**: Gonzo detects format per line, so mixed formats will have inconsistent parsing. **Example**: ``` {"level":"info","msg":"API started"} ← Parsed as JSON level=info msg="processing request" ← Parsed as logfmt [ERROR] Connection timeout ← Plain text ``` **Solutions**: 1. **Accept mixed display** * This is expected behavior * Each format renders appropriately 2. **Pre-filter to single format** ```bash # Extract only JSON lines grep '^{' mixed.log | gonzo # Or only logfmt grep '=' mixed.log | gonzo ``` 3. **Convert to uniform format** ```bash # Convert all to JSON with jq cat mixed.log | jq -R -c '. | fromjson? // {"message": .}' | gonzo ``` #### Attributes Not Extracted **Symptom**: Logs parse but attributes panel is empty. **Causes & Solutions**: 1. **Plain text logs have no structured attributes** * Plain text can't be parsed into fields * Only JSON and logfmt have attributes 2. **Check JSON structure** ```bash # Verify JSON has expected fields head -1 logs.json | jq . # Should show key-value pairs ``` 3. **Nested JSON** ```json { "log": { "level": "info", "service": "api" } } ``` * Gonzo extracts nested attributes * May need to check nested paths 4. **Missing common attribute names** * Gonzo looks for common fields: `level`, `service`, `host`, etc. * Custom fields may not be highlighted ### JSON Issues #### Malformed JSON **Symptom**: JSON logs show as plain text or cause errors. **Common JSON Issues**: 1. **Trailing commas** ```json {"level":"info","msg":"test",} ❌ Invalid {"level":"info","msg":"test"} ✅ Valid ``` 2. **Single quotes instead of double** ```json {'level':'info'} ❌ Invalid {"level":"info"} ✅ Valid ``` 3. **Unescaped quotes in strings** ```json {"msg":"He said "hi""} ❌ Invalid {"msg":"He said \"hi\""} ✅ Valid {"msg":"He said 'hi'"} ✅ Valid (single quotes in string) ``` 4. **Missing quotes on keys** ```json {level:"info"} ❌ Invalid {"level":"info"} ✅ Valid ``` **Validate & Fix**: ```bash # Validate JSON cat logs.json | jq . > /dev/null # If errors, shows line number # Pretty-print to find issues jq . logs.json # Fix and re-format jq -c . logs.json > fixed.json gonzo -f fixed.json ``` #### Multi-line JSON **Symptom**: JSON objects span multiple lines, not parsed correctly. **Example**: ```json { "level": "info", "message": "test" } ``` **Solution**: Gonzo expects one JSON object per line (JSONL/NDJSON format). ```bash # Compact multi-line JSON to single lines jq -c . pretty.json > compact.json gonzo -f compact.json # Or pipe directly jq -c . pretty.json | gonzo ``` #### JSON with Metadata Prefix **Symptom**: Lines have timestamp or metadata before JSON. **Example**: ``` 2024-01-15 10:30:05 {"level":"info","msg":"test"} ``` **Solution**: Remove prefix before piping to Gonzo: ```bash # Remove timestamp prefix sed 's/^[0-9-]* [0-9:]* //' logs.txt | gonzo # Or use awk awk '{$1=$2=""; print}' logs.txt | gonzo # Extract just JSON part grep -o '{.*}' logs.txt | gonzo ``` #### Escaped JSON in Strings **Symptom**: JSON contains escaped JSON strings. **Example**: ```json {"log":"{\"level\":\"info\",\"msg\":\"test\"}"} ``` **Solution**: ```bash # Unescape inner JSON jq -r '.log | fromjson' logs.json | gonzo # Or handle both levels jq -c '.log | fromjson? // .' logs.json | gonzo ``` ### Logfmt Issues #### Logfmt Not Detected **Symptom**: Key=value logs appear as plain text. **Requirements for logfmt detection**: * Must have `key=value` patterns * Multiple pairs per line * Values can be quoted: `key="value with spaces"` **Example**: ``` level=info service=api user=123 msg="request completed" ✅ Detected INFO service api user 123 request completed ❌ Not logfmt ``` **Solutions**: 1. **Verify format** ```bash # Check for key=value pattern grep -E '\w+=\w+' logs.txt | head ``` 2. **Add more key=value pairs** * Single pair may not trigger detection * Multiple pairs more reliably detected 3. **Use custom format** ```bash gonzo --format=my-logfmt -f logs.txt ``` #### Spaces in Logfmt Values **Symptom**: Values with spaces not parsed correctly. **Examples**: ``` msg=hello world ❌ Breaks: "world" seen as separate key msg="hello world" ✅ Correct: quotes preserve spaces msg=hello\ world ✅ Correct: escape preserves spaces ``` **Solution**: Ensure spaces in values are properly quoted or escaped: ```bash # Fix unquoted spaces (requires log generation fix) # Or accept partial parsing of problematic lines ``` #### Logfmt with Nested Structures **Symptom**: Nested objects in logfmt don't parse well. **Example**: ``` user.id=123 user.name=john ``` **Explanation**: Logfmt is flat by design. Nested structures need JSON. **Solution**: 1. **Accept flat representation** * Gonzo extracts `user.id` and `user.name` as separate attributes 2. **Convert to JSON if needed** ```bash # If you control log format, use JSON for nested data ``` ### Plain Text Issues #### No Structure Extracted from Text Logs **Symptom**: Plain text logs show no attributes. **Explanation**: Plain text logs can't be parsed into structured fields automatically. **Examples**: ``` [2024-01-15 10:30:05] ERROR: Connection failed INFO - api-service - User login successful ``` **Solutions**: 1. **Accept plain text display** * Logs still searchable and analyzable * Just no structured attributes 2. **Create custom format parser** ```bash # Define regex-based parser # See Custom Formats Guide gonzo --format=my-text-format -f logs.txt ``` 3. **Convert logs to structured format** * Modify application to output JSON/logfmt * Use log shipper to add structure (Fluent Bit, Logstash) #### Severity Not Detected in Text Logs **Symptom**: Plain text logs don't show color-coded severity. **Explanation**: Gonzo looks for common severity keywords in text logs: * ERROR, FATAL, CRITICAL → Red * WARN, WARNING → Yellow * INFO → Green * DEBUG → Blue * TRACE → White **Solutions**: 1. **Include severity keywords** ``` [ERROR] Connection failed ✅ Detected Connection failed ❌ Not detected WARN: Low disk space ✅ Detected Disk space is low ❌ Not detected ``` 2. **Use consistent format** * Put severity at start of line * Use standard keywords (ERROR, WARN, INFO, DEBUG) 3. **Create custom format** * Define severity extraction pattern * Map custom levels to standard severities ### OTLP Format Issues #### OTLP Logs Not Appearing **Symptom**: OTLP receiver running but no logs in Gonzo. **Diagnosis**: 1. **Verify receiver is enabled** ```bash # Check Gonzo started with OTLP gonzo --otlp-enabled # Should show listening on ports 4317 and 4318 ``` 2. **Check sender configuration** ```yaml # Verify endpoint in sender endpoint: localhost:4317 # gRPC endpoint: http://localhost:4318/v1/logs # HTTP ``` 3. **Test with curl (HTTP)** ```bash curl -X POST http://localhost:4318/v1/logs \ -H "Content-Type: application/json" \ -d '{"resourceLogs":[]}' # Should return 200 OK ``` 4. **Check for port conflicts** ```bash lsof -i :4317 lsof -i :4318 ``` **Solutions**: See Common Issues - OTLP Receiver for detailed fixes. #### OTLP Attributes Missing **Symptom**: OTLP logs appear but without expected attributes. **Causes**: 1. **Attributes in resource vs log record** * Resource attributes: service.name, host, etc. * Log record attributes: user\_id, request\_id, etc. * Both should be extracted 2. **Verify sender includes attributes** ```python # Ensure attributes are set logger_provider.add_log_record_processor(processor) # Check resource attributes and log attributes ``` 3. **Check attribute names** * Gonzo shows all attributes * May just be named differently than expected ### Custom Format Issues #### Custom Format Not Working **Symptom**: `--format=my-format` shows error or doesn't parse. **Diagnosis**: 1. **Verify format file exists** ```bash ls ~/.config/gonzo/formats/my-format.yaml ``` 2. **Check YAML syntax** ```bash cat ~/.config/gonzo/formats/my-format.yaml # Validate if you have yamllint yamllint ~/.config/gonzo/formats/my-format.yaml ``` 3. **Test with built-in format first** ```bash # Verify custom formats work at all gonzo --format=loki-stream -f test.json ``` **Solutions**: See Custom Formats Guide for: * Format file syntax * Regex patterns * Field mapping * Testing formats #### Regex Not Matching **Symptom**: Custom format regex doesn't extract fields. **Solutions**: 1. **Test regex separately** ```bash # Test your regex pattern echo "sample log line" | grep -E "your-regex-pattern" ``` 2. **Use online regex tester** * Test at regex101.com * Use example log lines * Verify capture groups 3. **Check for special characters** ```yaml # Escape special characters pattern: '\[(\d+)\]' # Brackets escaped ``` 4. **Start simple, iterate** ```yaml # Begin with basic pattern pattern: '(\w+)' # Then add complexity pattern: '(\w+)=(\w+)' # Finally, full pattern pattern: '(\w+)="([^"]*)"' ``` ### Encoding Issues #### Special Characters Garbled **Symptom**: Non-ASCII characters display incorrectly. **Solutions**: 1. **Ensure UTF-8 encoding** ```bash export LANG=en_US.UTF-8 export LC_ALL=en_US.UTF-8 ``` 2. **Check file encoding** ```bash file logs.txt # Should show: UTF-8 Unicode text # Convert if needed iconv -f ISO-8859-1 -t UTF-8 logs.txt > logs_utf8.txt ``` 3. **Pre-filter logs** ```bash # Filter before complex parsing grep "pattern" logs.txt | gonzo --format=complex ``` 4. **Use built-in formats when possible** * JSON and logfmt parsing is optimized * Custom regex is slower ### Timestamp Issues #### Timestamps Not Recognized **Symptom**: Logs appear in wrong order or timestamp not extracted. **Common timestamp formats Gonzo recognizes**: ``` 2024-01-15T10:30:05Z # ISO 8601 2024-01-15T10:30:05.123456Z # ISO 8601 with microseconds 2024-01-15 10:30:05 # Common format Jan 15 10:30:05 # Syslog format 1705315805 # Unix timestamp ``` **Solutions**: 1. **Use ISO 8601 format (recommended)** ```json {"timestamp":"2024-01-15T10:30:05Z","msg":"test"} ``` 2. **Ensure timestamp field name** * Common names: `timestamp`, `time`, `@timestamp`, `ts` * Gonzo checks these automatically 3. **Custom format for unusual timestamps** * Define timestamp extraction in format file * Specify timestamp format #### Timezone Issues **Symptom**: Timestamps appear in wrong timezone. **Solutions**: 1. **Use UTC in logs (recommended)** ```json {"timestamp":"2024-01-15T10:30:05Z"} # Z indicates UTC ``` 2. **Include timezone offset** ```json {"timestamp":"2024-01-15T10:30:05-05:00"} # EST ``` 3. **Gonzo displays timestamps as received** * No automatic conversion * Format logs consistently at source ### Large Log Line Issues #### Very Long Lines Truncated **Symptom**: Extremely long log lines appear cut off. **Solutions**: 1. **Use horizontal scrolling** ``` ← → or h l # Scroll horizontally ``` 2. **View in detail modal** ``` Enter # Opens full log entry ``` 3. **Split long lines at source** * Configure application to use reasonable line length * Use structured logging to avoid massive single-line logs #### Lines Exceed Buffer **Symptom**: Some log lines cause errors or don't appear. **Solution**: Gonzo handles lines up to typical buffer limits. For extremely large lines: ```bash # Pre-process to truncate lines cut -c 1-10000 massive.log | gonzo # Or filter out problematic lines awk 'length($0) < 10000' massive.log | gonzo ``` ### Debugging Format Issues #### Test Format Detection ```bash # Test with minimal sample echo '{"level":"info","msg":"test"}' | gonzo # Should parse as JSON echo 'level=info msg=test' | gonzo # Should parse as logfmt echo 'INFO test message' | gonzo # Should show as plain text ``` #### Examine Raw Logs ```bash # Check first few lines head -5 logs.txt # Check for hidden characters cat -A logs.txt | head # Validate JSON structure head -1 logs.json | jq . # Check line endings file logs.txt # Shows CRLF vs LF ``` #### Compare with Known-Good Format ```bash # Test with working format first echo '{"level":"info","message":"test"}' > good.json gonzo -f good.json # Then compare with problem logs diff <(head -1 good.json) <(head -1 problem.json) ``` ### Common Format Patterns #### Application Logs **Go/Logrus**: ```json {"level":"info","msg":"started","time":"2024-01-15T10:30:05Z"} ``` ✅ Parses as JSON automatically **Python/Logging**: ``` 2024-01-15 10:30:05,123 INFO module: message ``` ⚠️ Plain text - create custom format for structure **Node.js/Winston**: ```json {"level":"info","message":"started","timestamp":"2024-01-15T10:30:05Z"} ``` ✅ Parses as JSON automatically #### System Logs **Syslog**: ``` Jan 15 10:30:05 hostname service[123]: message ``` ⚠️ Plain text - consider custom format **Systemd Journal**: ``` MESSAGE=Test log PRIORITY=6 _HOSTNAME=server ``` ⚠️ Key=value but special format - needs custom parser #### Container Logs **Docker JSON**: ```json {"log":"application log message\n","stream":"stdout","time":"2024-01-15T10:30:05Z"} ``` ✅ Parses as JSON, extracts nested log **Kubernetes**: ```json {"level":"info","msg":"test","pod":"app-123"} ``` ✅ Parses as JSON with K8s attributes ### Format Best Practices #### When Choosing Log Format 1. **Prefer structured formats** * JSON or logfmt over plain text * Easier to parse and analyze * Better attribute extraction 2. **Use consistent format** * Same format across all services * Easier to aggregate and search 3. **Include standard fields** * `level` or `severity`: ERROR, WARN, INFO, DEBUG * `timestamp`: ISO 8601 format * `message` or `msg`: Human-readable message * `service` or `service.name`: Service identifier 4. **Example good JSON log**: ```json { "timestamp": "2024-01-15T10:30:05Z", "level": "error", "service": "api", "message": "Database connection failed", "error": "connection timeout", "host": "prod-server-01" } ``` #### When You Can't Change Format 1. **Use custom format definition** * Create regex-based parser * Map fields to standard attributes 2. **Pre-process logs** * Use awk/sed to restructure * Convert to JSON/logfmt before Gonzo 3. **Use log shipping layer** * Fluent Bit, Logstash, Vector * Transform logs to standard format ### Getting Help #### Provide This Info for Format Issues ```bash # Sample log lines (3-5 lines) head -5 logs.txt # File encoding file logs.txt # Attempted command echo "gonzo -f logs.txt [--format=...]" # Expected vs actual behavior echo "Expected: Parse as JSON" echo "Actual: Shows as plain text" # Gonzo version gonzo --version ``` #### Resources * **Common Issues** - General troubleshooting * **Custom Formats Guide** - Creating format parsers * [**GitHub Issues**](https://github.com/control-theory/gonzo/issues) - Report format bugs * [**Examples Directory**](https://github.com/control-theory/gonzo/tree/main/examples) - Sample format files {% hint style="info" %} **Testing formats?** Start with small sample files before processing large logs. This makes debugging much faster. {% endhint %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.controltheory.com/controltheory-documentation/gonzo-docs/troubleshooting/log-format-issues.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.