# AWS CloudWatch Logs

## AWS CloudWatch Integration

Analyze AWS CloudWatch Logs with Gonzo for powerful, interactive log analysis right from your terminal. Stream logs in real-time or analyze historical data with AI-powered insights.

{% hint style="info" %}
**Prerequisites:** AWS CLI installed and configured with appropriate credentials. Use `aws configure` to set up your AWS access.
{% endhint %}

### Why Use Gonzo with CloudWatch?

AWS CloudWatch Logs is a powerful log aggregation service, but analyzing logs in the console can be limiting. Gonzo brings:

* **🎨 Visual Analysis** - Beautiful terminal interface vs web console
* **🤖 AI Insights** - Intelligent pattern recognition and root cause analysis
* **⚡ Real-Time Streaming** - Live log analysis as events occur
* **🔍 Advanced Filtering** - Powerful regex and search capabilities
* **📊 Pattern Detection** - Automatic clustering of similar log entries
* **💰 Cost Efficiency** - Analyze logs locally without CloudWatch Insights costs

### Quick Start

#### Getting Your Log Groups

First, discover what log groups are available in your AWS region:

```bash
# List all log groups with names and ARNs
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,ARN:arn}' \
  --output json

# List log groups in table format
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,ARN:arn}' \
  --output table

# Filter by prefix
aws logs describe-log-groups \
  --log-group-name-prefix /aws/lambda/ \
  --query 'logGroups[*].logGroupName' \
  --output text
```

#### Basic Log Tailing with Gonzo

The simplest way to use Gonzo with CloudWatch Logs is to pipe `aws logs tail` output:

```bash
# Basic tail (default: last 10 minutes)
aws logs tail "/aws/eks/stage/cluster" --follow | gonzo

# Tail with JSON output for better attribute extraction
aws logs tail "/aws/eks/stage/cluster" --follow --format json | gonzo

# Tail with AI analysis
aws logs tail "/aws/eks/stage/cluster" --follow --format json | \
gonzo --ai-model="gpt-4"
```

{% hint style="info" %}
**Output Format:** The default "detailed" format outputs one log per line (non-JSON). Use `--format json` to gain attribute visibility in Gonzo, though each JSON log message spans two lines.
{% endhint %}

#### Multiple Log Groups

Monitor multiple log groups simultaneously:

```bash
# Method 1: Using process substitution
gonzo - < <(
  aws logs tail "/aws/eks/stage/cluster" --follow --format json &
  aws logs tail "RDS" --follow --format json &
  wait
)

# Method 2: Named pipes (alternative)
mkfifo log_stream
(aws logs tail "/aws/eks/stage/cluster" --follow --format json &
 aws logs tail "RDS" --follow --format json) > log_stream &
gonzo - < log_stream

# Method 3: File-based aggregation
aws logs tail "/aws/eks/stage/cluster" --follow --format json > logs1.txt &
aws logs tail "RDS" --follow --format json > logs2.txt &
gonzo -f logs1.txt -f logs2.txt --follow
```

**How This Works:**

* Starts multiple `aws logs tail` processes in the background (`&`)
* Combines their output streams
* Pipes the combined output to Gonzo
* Uses `wait` to ensure all background processes complete properly

### AWS CLI Live Tail Integration

Live Tail provides near real-time streaming of log events as they are ingested into CloudWatch.

#### Basic Live Tail Usage

```bash
# Single log group with ARN
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" | gonzo

# Multiple log groups
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" \
  "arn:aws:logs:us-east-1:767397775588:log-group:RDS" | gonzo

# With AI analysis
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" | \
gonzo --ai-model="gpt-4"
```

**Live Tail Benefits:**

* ✅ Near real-time streaming (lower latency than `tail`)
* ✅ Native JSON output (better attribute extraction)
* ✅ Supports multiple log groups natively
* ✅ Automatic handling of high-throughput log groups

#### Live Tail vs Regular Tail

| Feature                 | `aws logs tail`            | `aws logs start-live-tail` |
| ----------------------- | -------------------------- | -------------------------- |
| **Latency**             | \~1-2 seconds              | Near real-time (<1 second) |
| **Multiple Log Groups** | Requires workarounds       | Native support             |
| **Output Format**       | Detailed (default) or JSON | JSON (default)             |
| **Historical Logs**     | Yes (--since flag)         | No (only new logs)         |
| **Best For**            | Historical analysis        | Real-time monitoring       |

**Choose Live Tail When:**

* Monitoring production incidents in real-time
* Need lowest possible latency
* Monitoring multiple log groups simultaneously
* JSON output is preferred

**Choose Regular Tail When:**

* Need historical log analysis
* Want to start from specific time (--since)
* Need simple single log group monitoring

### AWS CLI Integration

#### Installing and Configuring AWS CLI

```bash
# Install AWS CLI (if not already installed)
# macOS
brew install awscli

# Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Windows
# Download from: https://awscli.amazonaws.com/AWSCLIV2.msi

# Configure AWS CLI
aws configure
# Enter:
#   AWS Access Key ID
#   AWS Secret Access Key
#   Default region (e.g., us-east-1)
#   Default output format (json recommended)

# Verify configuration
aws sts get-caller-identity
```

#### Essential CloudWatch Logs Commands

{% tabs %}
{% tab title="Listing Log Groups" %}

```bash
# List all log groups
aws logs describe-log-groups

# List with specific prefix
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/

# Get specific log group details
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/my-function

# Count log groups
aws logs describe-log-groups --query 'length(logGroups)'
```

{% endtab %}

{% tab title="Tailing Logs" %}

```bash
# Basic tail (last 10 minutes)
aws logs tail /aws/lambda/my-function

# Follow in real-time
aws logs tail /aws/lambda/my-function --follow

# Since specific time
aws logs tail /aws/lambda/my-function --since 1h
aws logs tail /aws/lambda/my-function --since 30m
aws logs tail /aws/lambda/my-function --since 2024-01-15T10:00:00

# Filter pattern
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR"

# Format options
aws logs tail /aws/lambda/my-function --format short
aws logs tail /aws/lambda/my-function --format detailed
```

{% endtab %}

{% tab title="Filtering Logs" %}

```bash
# Filter by pattern
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR"

# Multiple terms (AND logic)
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR timeout"

# JSON field filtering
aws logs tail /aws/lambda/my-function --filter-pattern '{ $.level = "error" }'

# Numeric filtering
aws logs tail /aws/lambda/my-function --filter-pattern '{ $.duration > 1000 }'

# Combined with Gonzo
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR" --follow | gonzo
```

{% endtab %}
{% endtabs %}

### Real-World Integration Patterns

#### Lambda Function Monitoring

```bash
# Monitor specific Lambda function
aws logs tail /aws/lambda/my-function --follow | gonzo

# Monitor with AI for error analysis
aws logs tail /aws/lambda/my-function --follow | \
gonzo --ai-model="gpt-4"

# Monitor multiple Lambda functions
(aws logs tail /aws/lambda/function1 --follow & \
 aws logs tail /aws/lambda/function2 --follow) | gonzo

# Focus on errors only
aws logs tail /aws/lambda/my-function --follow | \
grep -i error | \
gonzo --log-buffer=5000
```

#### ECS/Fargate Container Logs

```bash
# ECS task logs
aws logs tail /ecs/my-service --follow | gonzo

# Filter by container
aws logs tail /ecs/my-service --follow | \
grep "container-name" | \
gonzo

# Multiple ECS services
(aws logs tail /ecs/api --follow & \
 aws logs tail /ecs/worker --follow) | \
gonzo --log-buffer=10000
```

#### API Gateway Logs

```bash
# API Gateway execution logs
aws logs tail /aws/apigateway/my-api --follow | gonzo

# Filter by HTTP status
aws logs tail /aws/apigateway/my-api --follow | \
grep -E "(4[0-9]{2}|5[0-9]{2})" | \
gonzo --ai-model="gpt-4"

# Access logs analysis
aws logs tail /aws/apigateway/my-api/access --since 1h | \
gonzo --log-buffer=20000
```

#### RDS Database Logs

```bash
# RDS error logs
aws logs tail /aws/rds/instance/my-db/error --follow | gonzo

# Slow query logs
aws logs tail /aws/rds/instance/my-db/slowquery --follow | \
gonzo --ai-model="gpt-4"

# General logs
aws logs tail /aws/rds/instance/my-db/general --since 1h | gonzo
```

#### CloudTrail Logs

```bash
# Monitor CloudTrail events
aws logs tail aws-cloudtrail-logs --follow | gonzo

# Security-focused monitoring
aws logs tail aws-cloudtrail-logs --follow | \
grep -E "(unauthorized|denied|failed)" | \
gonzo --ai-model="gpt-4"

# Specific service events
aws logs tail aws-cloudtrail-logs --follow | \
grep "ec2.amazonaws.com" | \
gonzo
```

### Advanced Usage Patterns

#### Multi-Account Monitoring

```bash
# Switch AWS profiles
export AWS_PROFILE=production
aws logs tail /aws/lambda/function --follow | gonzo

# Monitor multiple accounts simultaneously
AWS_PROFILE=prod aws logs tail /aws/lambda/api --follow > prod-logs.txt &
AWS_PROFILE=staging aws logs tail /aws/lambda/api --follow > staging-logs.txt &
gonzo -f prod-logs.txt -f staging-logs.txt --follow

# Account-specific aliases
alias gonzo-prod='AWS_PROFILE=production aws logs tail'
alias gonzo-staging='AWS_PROFILE=staging aws logs tail'

# Usage
gonzo-prod /aws/lambda/function --follow | gonzo
```

#### Cost-Optimized Analysis

```bash
# Export logs once, analyze multiple times
aws logs tail /aws/lambda/function --since 24h > logs-$(date +%Y%m%d).txt

# Analyze without streaming (no ongoing CloudWatch costs)
gonzo -f logs-20240115.txt --ai-model="gpt-4"

# Batch processing for cost savings
for log_group in $(aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output text); do
    echo "Analyzing $log_group..."
    aws logs tail "$log_group" --since 1h > "logs/${log_group//\//_}.txt"
done

# Analyze all exported logs
gonzo -f logs/*.txt
```

#### Automated Monitoring Scripts

```bash
#!/bin/bash
# monitor-cloudwatch.sh
# Continuous monitoring with alerting

LOG_GROUP="$1"
ALERT_PATTERN="${2:-ERROR}"
ALERT_WEBHOOK="${3}"

echo "Monitoring $LOG_GROUP for pattern: $ALERT_PATTERN"

aws logs tail "$LOG_GROUP" --follow | \
while IFS= read -r line; do
    echo "$line"
    
    # Check for alert pattern
    if echo "$line" | grep -qi "$ALERT_PATTERN"; then
        # Send alert
        if [ -n "$ALERT_WEBHOOK" ]; then
            curl -X POST "$ALERT_WEBHOOK" \
                -H "Content-Type: application/json" \
                -d "{\"text\": \"Alert: $line\"}"
        fi
        
        # Analyze with Gonzo + AI
        echo "$line" | gonzo --ai-model="gpt-4"
    fi
done
```

#### Performance Analysis

```bash
# Analyze Lambda cold starts
aws logs tail /aws/lambda/function --since 1h | \
grep "Init Duration" | \
gonzo --ai-model="gpt-4"

# Memory usage analysis
aws logs tail /aws/lambda/function --since 1h | \
grep "Max Memory Used" | \
gonzo

# Duration analysis
aws logs tail /aws/lambda/function --since 1h | \
grep "Duration" | \
gonzo --log-buffer=20000
```

### CloudWatch Logs Insights Integration

#### Export Insights Query Results

```bash
# Run CloudWatch Logs Insights query
query_id=$(aws logs start-query \
    --log-group-name /aws/lambda/my-function \
    --start-time $(date -u -d '1 hour ago' +%s) \
    --end-time $(date -u +%s) \
    --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 1000' \
    --query 'queryId' \
    --output text)

# Wait for query to complete
aws logs wait query-complete --query-id "$query_id"

# Get results and analyze with Gonzo
aws logs get-query-results --query-id "$query_id" | \
jq -r '.results[] | .[1].value' | \
gonzo --ai-model="gpt-4"
```

### Configuration for CloudWatch

#### Optimized Configuration File

```yaml
# ~/.config/gonzo/cloudwatch.yml
# Optimized for CloudWatch Logs

# CloudWatch logs are streamed via stdin
# No file configuration needed

# Performance for CloudWatch streaming
update-interval: 2s              # Responsive updates
log-buffer: 10000               # Large buffer for CloudWatch volume
memory-size: 50000              # Extensive pattern tracking

# AI configuration for AWS analysis
ai-model: "gpt-4"               # Best for complex AWS issues
ai-context-size: 8000           # Large context for AWS logs

# Display
show-service: true              # Show AWS service context
```

**Usage:**

```bash
aws logs tail /aws/lambda/function --follow | \
gonzo --config ~/.config/gonzo/cloudwatch.yml
```

#### Environment-Specific Aliases

```bash
# Add to ~/.bashrc or ~/.zshrc

# Production monitoring
alias gonzo-prod='AWS_PROFILE=production aws logs tail --follow'

# Development monitoring
alias gonzo-dev='AWS_PROFILE=development aws logs tail --follow'

# Lambda-specific
alias gonzo-lambda='aws logs tail --follow'

# With AI analysis
alias gonzo-lambda-ai='aws logs tail --follow | gonzo --ai-model="gpt-4"'

# Usage
gonzo-lambda /aws/lambda/my-function
gonzo-lambda-ai /aws/lambda/critical-function
```

### Complete Guide

For step-by-step CloudWatch integration: 📖 [**Gonzo: A Better Way to Tail AWS CloudWatch Logs**](https://www.controltheory.com/blog/gonzo-a-better-way-to-tail-aws-cloudwatch-logs/)

### Best Practices

#### 🎯 **Efficient Log Retrieval**

```bash
# Use time ranges to limit data
aws logs tail /aws/lambda/function --since 1h    # Last hour only

# Use filter patterns for pre-filtering
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow

# Export for repeated analysis
aws logs tail /aws/lambda/function --since 24h > daily-export.txt
gonzo -f daily-export.txt  # No repeated CloudWatch API calls
```

### Cost Considerations

{% hint style="warning" %}
**Important:** AWS costs may be incurred for using the AWS CLI `tail` or `start-live-tail` commands. Please consult the [AWS CloudWatch Logs pricing documentation](https://aws.amazon.com/cloudwatch/pricing/) for details on potential charges related to log data retrieval.
{% endhint %}

#### Understanding CloudWatch Logs Pricing

**Data Retrieval Costs:**

* **GetLogEvents API**: Used by `aws logs tail` - included with storage
* **StartLiveTail API**: Used by `aws logs start-live-tail` - additional charges may apply
* **Data Transfer**: Egress charges may apply for large volumes

#### Cost Optimization Strategies

```bash
# 1. Use time-based filtering to reduce data volume
aws logs tail /aws/lambda/function --since 1h --follow | gonzo  # Last hour only

# 2. Use filter patterns at the source
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow | gonzo

# 3. Export once, analyze multiple times (no repeated API calls)
aws logs tail /aws/lambda/function --since 24h > daily-export.txt
gonzo -f daily-export.txt  # Analyze locally
gonzo -f daily-export.txt --ai-model="gpt-4"  # Re-analyze with AI

# 4. Use regular tail instead of live tail when real-time isn't critical
aws logs tail /aws/lambda/function --follow | gonzo  # Lower cost

# 5. Monitor specific log groups instead of all logs
aws logs tail "/aws/lambda/critical-function" --follow | gonzo  # Focused monitoring
```

**Cost Comparison Example:**

```bash
# Higher cost: Live tail multiple groups for 24 hours
aws logs start-live-tail --log-group-identifiers arn1 arn2 arn3 | gonzo

# Lower cost: Regular tail with filtering
aws logs tail "/aws/lambda/function" --filter-pattern "ERROR" --follow | gonzo

# Lowest cost: Export and analyze locally
aws logs tail "/aws/lambda/function" --since 1h > export.txt
gonzo -f export.txt  # No ongoing API costs
```

#### 🔒 **Security Best Practices**

```bash
# Use IAM roles instead of access keys
aws configure set role_arn arn:aws:iam::123456789012:role/LogAnalyst
aws configure set source_profile default

# Least privilege IAM policy for Gonzo usage:
# {
#   "Version": "2012-10-17",
#   "Statement": [{
#     "Effect": "Allow",
#     "Action": [
#       "logs:DescribeLogGroups",
#       "logs:DescribeLogStreams",
#       "logs:GetLogEvents",
#       "logs:FilterLogEvents"
#     ],
#     "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/*"
#   }]
# }

# Use MFA for production access
aws configure set mfa_serial arn:aws:iam::123456789012:mfa/user

# Audit log access
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=/aws/lambda/function
```

#### ⚡ **Performance Optimization**

```bash
# Pre-filter for performance
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow | gonzo

# Adjust Gonzo buffers for volume
aws logs tail /aws/lambda/high-traffic --follow | \
gonzo --log-buffer=20000 --update-interval=5s

# Use local AI for sensitive logs
export OPENAI_API_BASE="http://localhost:11434"
aws logs tail /aws/lambda/sensitive --follow | \
gonzo --ai-model="llama3"
```

### Troubleshooting

#### Common Issues

**Authentication Errors:**

```bash
# Verify AWS credentials
aws sts get-caller-identity

# Check IAM permissions
aws iam get-user

# Use specific profile
AWS_PROFILE=your-profile aws logs tail /aws/lambda/function
```

**Log Group Not Found:**

```bash
# List available log groups
aws logs describe-log-groups | grep logGroupName

# Search for specific pattern
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/

# Verify log group name (case-sensitive)
aws logs describe-log-groups --log-group-name /aws/lambda/my-function
```

**No Logs Appearing:**

```bash
# Check if logs exist
aws logs tail /aws/lambda/function --since 24h

# Verify time range
aws logs tail /aws/lambda/function --since 5m  # Last 5 minutes

# Check filter pattern
aws logs tail /aws/lambda/function --filter-pattern ""  # All logs
```

**Performance Issues:**

```bash
# Reduce log volume
aws logs tail /aws/lambda/function --filter-pattern "ERROR"

# Increase Gonzo buffers
aws logs tail /aws/lambda/function --follow | \
gonzo --log-buffer=20000 --memory-size=50000

# Use longer update intervals
aws logs tail /aws/lambda/function --follow | \
gonzo --update-interval=10s
```

### What's Next?

Now that you've mastered CloudWatch integration, explore related topics:

* **Kubernetes Integration** - Analyze EKS cluster logs
* **Configuration** - Optimize for CloudWatch workflows
* **AI Integration** - Enhanced AWS log analysis
* **Advanced Features** - Pattern detection in AWS logs

Or try these advanced CloudWatch patterns:

```bash
# Multi-region monitoring
for region in us-east-1 us-west-2 eu-west-1; do
    AWS_REGION=$region aws logs tail /aws/lambda/function --follow > logs-${region}.txt &
done
gonzo -f logs-*.txt --follow

# Cross-account analysis
AWS_PROFILE=account1 aws logs tail /aws/lambda/function --since 1h > account1.txt
AWS_PROFILE=account2 aws logs tail /aws/lambda/function --since 1h > account2.txt
gonzo -f account1.txt -f account2.txt --ai-model="gpt-4"
```

***

**Unlock the full potential of AWS CloudWatch Logs with Gonzo!** ☁️ From real-time Lambda monitoring to comprehensive AWS infrastructure analysis, Gonzo brings powerful log analysis to your CloudWatch workflows.