# AWS CloudWatch Logs

## AWS CloudWatch Integration

Analyze AWS CloudWatch Logs with Gonzo for powerful, interactive log analysis right from your terminal. Stream logs in real-time or analyze historical data with AI-powered insights.

{% hint style="info" %}
**Prerequisites:** AWS CLI installed and configured with appropriate credentials. Use `aws configure` to set up your AWS access.
{% endhint %}

### Why Use Gonzo with CloudWatch?

AWS CloudWatch Logs is a powerful log aggregation service, but analyzing logs in the console can be limiting. Gonzo brings:

* **🎨 Visual Analysis** - Beautiful terminal interface vs web console
* **🤖 AI Insights** - Intelligent pattern recognition and root cause analysis
* **⚡ Real-Time Streaming** - Live log analysis as events occur
* **🔍 Advanced Filtering** - Powerful regex and search capabilities
* **📊 Pattern Detection** - Automatic clustering of similar log entries
* **💰 Cost Efficiency** - Analyze logs locally without CloudWatch Insights costs

### Quick Start

#### Getting Your Log Groups

First, discover what log groups are available in your AWS region:

```bash
# List all log groups with names and ARNs
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,ARN:arn}' \
  --output json

# List log groups in table format
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,ARN:arn}' \
  --output table

# Filter by prefix
aws logs describe-log-groups \
  --log-group-name-prefix /aws/lambda/ \
  --query 'logGroups[*].logGroupName' \
  --output text
```

#### Basic Log Tailing with Gonzo

The simplest way to use Gonzo with CloudWatch Logs is to pipe `aws logs tail` output:

```bash
# Basic tail (default: last 10 minutes)
aws logs tail "/aws/eks/stage/cluster" --follow | gonzo

# Tail with JSON output for better attribute extraction
aws logs tail "/aws/eks/stage/cluster" --follow --format json | gonzo

# Tail with AI analysis
aws logs tail "/aws/eks/stage/cluster" --follow --format json | \
gonzo --ai-model="gpt-4"
```

{% hint style="info" %}
**Output Format:** The default "detailed" format outputs one log per line (non-JSON). Use `--format json` to gain attribute visibility in Gonzo, though each JSON log message spans two lines.
{% endhint %}

#### Multiple Log Groups

Monitor multiple log groups simultaneously:

```bash
# Method 1: Using process substitution
gonzo - < <(
  aws logs tail "/aws/eks/stage/cluster" --follow --format json &
  aws logs tail "RDS" --follow --format json &
  wait
)

# Method 2: Named pipes (alternative)
mkfifo log_stream
(aws logs tail "/aws/eks/stage/cluster" --follow --format json &
 aws logs tail "RDS" --follow --format json) > log_stream &
gonzo - < log_stream

# Method 3: File-based aggregation
aws logs tail "/aws/eks/stage/cluster" --follow --format json > logs1.txt &
aws logs tail "RDS" --follow --format json > logs2.txt &
gonzo -f logs1.txt -f logs2.txt --follow
```

**How This Works:**

* Starts multiple `aws logs tail` processes in the background (`&`)
* Combines their output streams
* Pipes the combined output to Gonzo
* Uses `wait` to ensure all background processes complete properly

### AWS CLI Live Tail Integration

Live Tail provides near real-time streaming of log events as they are ingested into CloudWatch.

#### Basic Live Tail Usage

```bash
# Single log group with ARN
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" | gonzo

# Multiple log groups
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" \
  "arn:aws:logs:us-east-1:767397775588:log-group:RDS" | gonzo

# With AI analysis
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" | \
gonzo --ai-model="gpt-4"
```

**Live Tail Benefits:**

* ✅ Near real-time streaming (lower latency than `tail`)
* ✅ Native JSON output (better attribute extraction)
* ✅ Supports multiple log groups natively
* ✅ Automatic handling of high-throughput log groups

#### Live Tail vs Regular Tail

| Feature                 | `aws logs tail`            | `aws logs start-live-tail` |
| ----------------------- | -------------------------- | -------------------------- |
| **Latency**             | \~1-2 seconds              | Near real-time (<1 second) |
| **Multiple Log Groups** | Requires workarounds       | Native support             |
| **Output Format**       | Detailed (default) or JSON | JSON (default)             |
| **Historical Logs**     | Yes (--since flag)         | No (only new logs)         |
| **Best For**            | Historical analysis        | Real-time monitoring       |

**Choose Live Tail When:**

* Monitoring production incidents in real-time
* Need lowest possible latency
* Monitoring multiple log groups simultaneously
* JSON output is preferred

**Choose Regular Tail When:**

* Need historical log analysis
* Want to start from specific time (--since)
* Need simple single log group monitoring

### AWS CLI Integration

#### Installing and Configuring AWS CLI

```bash
# Install AWS CLI (if not already installed)
# macOS
brew install awscli

# Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Windows
# Download from: https://awscli.amazonaws.com/AWSCLIV2.msi

# Configure AWS CLI
aws configure
# Enter:
#   AWS Access Key ID
#   AWS Secret Access Key
#   Default region (e.g., us-east-1)
#   Default output format (json recommended)

# Verify configuration
aws sts get-caller-identity
```

#### Essential CloudWatch Logs Commands

{% tabs %}
{% tab title="Listing Log Groups" %}

```bash
# List all log groups
aws logs describe-log-groups

# List with specific prefix
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/

# Get specific log group details
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/my-function

# Count log groups
aws logs describe-log-groups --query 'length(logGroups)'
```

{% endtab %}

{% tab title="Tailing Logs" %}

```bash
# Basic tail (last 10 minutes)
aws logs tail /aws/lambda/my-function

# Follow in real-time
aws logs tail /aws/lambda/my-function --follow

# Since specific time
aws logs tail /aws/lambda/my-function --since 1h
aws logs tail /aws/lambda/my-function --since 30m
aws logs tail /aws/lambda/my-function --since 2024-01-15T10:00:00

# Filter pattern
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR"

# Format options
aws logs tail /aws/lambda/my-function --format short
aws logs tail /aws/lambda/my-function --format detailed
```

{% endtab %}

{% tab title="Filtering Logs" %}

```bash
# Filter by pattern
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR"

# Multiple terms (AND logic)
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR timeout"

# JSON field filtering
aws logs tail /aws/lambda/my-function --filter-pattern '{ $.level = "error" }'

# Numeric filtering
aws logs tail /aws/lambda/my-function --filter-pattern '{ $.duration > 1000 }'

# Combined with Gonzo
aws logs tail /aws/lambda/my-function --filter-pattern "ERROR" --follow | gonzo
```

{% endtab %}
{% endtabs %}

### Real-World Integration Patterns

#### Lambda Function Monitoring

```bash
# Monitor specific Lambda function
aws logs tail /aws/lambda/my-function --follow | gonzo

# Monitor with AI for error analysis
aws logs tail /aws/lambda/my-function --follow | \
gonzo --ai-model="gpt-4"

# Monitor multiple Lambda functions
(aws logs tail /aws/lambda/function1 --follow & \
 aws logs tail /aws/lambda/function2 --follow) | gonzo

# Focus on errors only
aws logs tail /aws/lambda/my-function --follow | \
grep -i error | \
gonzo --log-buffer=5000
```

#### ECS/Fargate Container Logs

```bash
# ECS task logs
aws logs tail /ecs/my-service --follow | gonzo

# Filter by container
aws logs tail /ecs/my-service --follow | \
grep "container-name" | \
gonzo

# Multiple ECS services
(aws logs tail /ecs/api --follow & \
 aws logs tail /ecs/worker --follow) | \
gonzo --log-buffer=10000
```

#### API Gateway Logs

```bash
# API Gateway execution logs
aws logs tail /aws/apigateway/my-api --follow | gonzo

# Filter by HTTP status
aws logs tail /aws/apigateway/my-api --follow | \
grep -E "(4[0-9]{2}|5[0-9]{2})" | \
gonzo --ai-model="gpt-4"

# Access logs analysis
aws logs tail /aws/apigateway/my-api/access --since 1h | \
gonzo --log-buffer=20000
```

#### RDS Database Logs

```bash
# RDS error logs
aws logs tail /aws/rds/instance/my-db/error --follow | gonzo

# Slow query logs
aws logs tail /aws/rds/instance/my-db/slowquery --follow | \
gonzo --ai-model="gpt-4"

# General logs
aws logs tail /aws/rds/instance/my-db/general --since 1h | gonzo
```

#### CloudTrail Logs

```bash
# Monitor CloudTrail events
aws logs tail aws-cloudtrail-logs --follow | gonzo

# Security-focused monitoring
aws logs tail aws-cloudtrail-logs --follow | \
grep -E "(unauthorized|denied|failed)" | \
gonzo --ai-model="gpt-4"

# Specific service events
aws logs tail aws-cloudtrail-logs --follow | \
grep "ec2.amazonaws.com" | \
gonzo
```

### Advanced Usage Patterns

#### Multi-Account Monitoring

```bash
# Switch AWS profiles
export AWS_PROFILE=production
aws logs tail /aws/lambda/function --follow | gonzo

# Monitor multiple accounts simultaneously
AWS_PROFILE=prod aws logs tail /aws/lambda/api --follow > prod-logs.txt &
AWS_PROFILE=staging aws logs tail /aws/lambda/api --follow > staging-logs.txt &
gonzo -f prod-logs.txt -f staging-logs.txt --follow

# Account-specific aliases
alias gonzo-prod='AWS_PROFILE=production aws logs tail'
alias gonzo-staging='AWS_PROFILE=staging aws logs tail'

# Usage
gonzo-prod /aws/lambda/function --follow | gonzo
```

#### Cost-Optimized Analysis

```bash
# Export logs once, analyze multiple times
aws logs tail /aws/lambda/function --since 24h > logs-$(date +%Y%m%d).txt

# Analyze without streaming (no ongoing CloudWatch costs)
gonzo -f logs-20240115.txt --ai-model="gpt-4"

# Batch processing for cost savings
for log_group in $(aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output text); do
    echo "Analyzing $log_group..."
    aws logs tail "$log_group" --since 1h > "logs/${log_group//\//_}.txt"
done

# Analyze all exported logs
gonzo -f logs/*.txt
```

#### Automated Monitoring Scripts

```bash
#!/bin/bash
# monitor-cloudwatch.sh
# Continuous monitoring with alerting

LOG_GROUP="$1"
ALERT_PATTERN="${2:-ERROR}"
ALERT_WEBHOOK="${3}"

echo "Monitoring $LOG_GROUP for pattern: $ALERT_PATTERN"

aws logs tail "$LOG_GROUP" --follow | \
while IFS= read -r line; do
    echo "$line"
    
    # Check for alert pattern
    if echo "$line" | grep -qi "$ALERT_PATTERN"; then
        # Send alert
        if [ -n "$ALERT_WEBHOOK" ]; then
            curl -X POST "$ALERT_WEBHOOK" \
                -H "Content-Type: application/json" \
                -d "{\"text\": \"Alert: $line\"}"
        fi
        
        # Analyze with Gonzo + AI
        echo "$line" | gonzo --ai-model="gpt-4"
    fi
done
```

#### Performance Analysis

```bash
# Analyze Lambda cold starts
aws logs tail /aws/lambda/function --since 1h | \
grep "Init Duration" | \
gonzo --ai-model="gpt-4"

# Memory usage analysis
aws logs tail /aws/lambda/function --since 1h | \
grep "Max Memory Used" | \
gonzo

# Duration analysis
aws logs tail /aws/lambda/function --since 1h | \
grep "Duration" | \
gonzo --log-buffer=20000
```

### CloudWatch Logs Insights Integration

#### Export Insights Query Results

```bash
# Run CloudWatch Logs Insights query
query_id=$(aws logs start-query \
    --log-group-name /aws/lambda/my-function \
    --start-time $(date -u -d '1 hour ago' +%s) \
    --end-time $(date -u +%s) \
    --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 1000' \
    --query 'queryId' \
    --output text)

# Wait for query to complete
aws logs wait query-complete --query-id "$query_id"

# Get results and analyze with Gonzo
aws logs get-query-results --query-id "$query_id" | \
jq -r '.results[] | .[1].value' | \
gonzo --ai-model="gpt-4"
```

### Configuration for CloudWatch

#### Optimized Configuration File

```yaml
# ~/.config/gonzo/cloudwatch.yml
# Optimized for CloudWatch Logs

# CloudWatch logs are streamed via stdin
# No file configuration needed

# Performance for CloudWatch streaming
update-interval: 2s              # Responsive updates
log-buffer: 10000               # Large buffer for CloudWatch volume
memory-size: 50000              # Extensive pattern tracking

# AI configuration for AWS analysis
ai-model: "gpt-4"               # Best for complex AWS issues
ai-context-size: 8000           # Large context for AWS logs

# Display
show-service: true              # Show AWS service context
```

**Usage:**

```bash
aws logs tail /aws/lambda/function --follow | \
gonzo --config ~/.config/gonzo/cloudwatch.yml
```

#### Environment-Specific Aliases

```bash
# Add to ~/.bashrc or ~/.zshrc

# Production monitoring
alias gonzo-prod='AWS_PROFILE=production aws logs tail --follow'

# Development monitoring
alias gonzo-dev='AWS_PROFILE=development aws logs tail --follow'

# Lambda-specific
alias gonzo-lambda='aws logs tail --follow'

# With AI analysis
alias gonzo-lambda-ai='aws logs tail --follow | gonzo --ai-model="gpt-4"'

# Usage
gonzo-lambda /aws/lambda/my-function
gonzo-lambda-ai /aws/lambda/critical-function
```

### Complete Guide

For step-by-step CloudWatch integration: 📖 [**Gonzo: A Better Way to Tail AWS CloudWatch Logs**](https://www.controltheory.com/blog/gonzo-a-better-way-to-tail-aws-cloudwatch-logs/)

### Best Practices

#### 🎯 **Efficient Log Retrieval**

```bash
# Use time ranges to limit data
aws logs tail /aws/lambda/function --since 1h    # Last hour only

# Use filter patterns for pre-filtering
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow

# Export for repeated analysis
aws logs tail /aws/lambda/function --since 24h > daily-export.txt
gonzo -f daily-export.txt  # No repeated CloudWatch API calls
```

### Cost Considerations

{% hint style="warning" %}
**Important:** AWS costs may be incurred for using the AWS CLI `tail` or `start-live-tail` commands. Please consult the [AWS CloudWatch Logs pricing documentation](https://aws.amazon.com/cloudwatch/pricing/) for details on potential charges related to log data retrieval.
{% endhint %}

#### Understanding CloudWatch Logs Pricing

**Data Retrieval Costs:**

* **GetLogEvents API**: Used by `aws logs tail` - included with storage
* **StartLiveTail API**: Used by `aws logs start-live-tail` - additional charges may apply
* **Data Transfer**: Egress charges may apply for large volumes

#### Cost Optimization Strategies

```bash
# 1. Use time-based filtering to reduce data volume
aws logs tail /aws/lambda/function --since 1h --follow | gonzo  # Last hour only

# 2. Use filter patterns at the source
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow | gonzo

# 3. Export once, analyze multiple times (no repeated API calls)
aws logs tail /aws/lambda/function --since 24h > daily-export.txt
gonzo -f daily-export.txt  # Analyze locally
gonzo -f daily-export.txt --ai-model="gpt-4"  # Re-analyze with AI

# 4. Use regular tail instead of live tail when real-time isn't critical
aws logs tail /aws/lambda/function --follow | gonzo  # Lower cost

# 5. Monitor specific log groups instead of all logs
aws logs tail "/aws/lambda/critical-function" --follow | gonzo  # Focused monitoring
```

**Cost Comparison Example:**

```bash
# Higher cost: Live tail multiple groups for 24 hours
aws logs start-live-tail --log-group-identifiers arn1 arn2 arn3 | gonzo

# Lower cost: Regular tail with filtering
aws logs tail "/aws/lambda/function" --filter-pattern "ERROR" --follow | gonzo

# Lowest cost: Export and analyze locally
aws logs tail "/aws/lambda/function" --since 1h > export.txt
gonzo -f export.txt  # No ongoing API costs
```

#### 🔒 **Security Best Practices**

```bash
# Use IAM roles instead of access keys
aws configure set role_arn arn:aws:iam::123456789012:role/LogAnalyst
aws configure set source_profile default

# Least privilege IAM policy for Gonzo usage:
# {
#   "Version": "2012-10-17",
#   "Statement": [{
#     "Effect": "Allow",
#     "Action": [
#       "logs:DescribeLogGroups",
#       "logs:DescribeLogStreams",
#       "logs:GetLogEvents",
#       "logs:FilterLogEvents"
#     ],
#     "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/*"
#   }]
# }

# Use MFA for production access
aws configure set mfa_serial arn:aws:iam::123456789012:mfa/user

# Audit log access
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=/aws/lambda/function
```

#### ⚡ **Performance Optimization**

```bash
# Pre-filter for performance
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow | gonzo

# Adjust Gonzo buffers for volume
aws logs tail /aws/lambda/high-traffic --follow | \
gonzo --log-buffer=20000 --update-interval=5s

# Use local AI for sensitive logs
export OPENAI_API_BASE="http://localhost:11434"
aws logs tail /aws/lambda/sensitive --follow | \
gonzo --ai-model="llama3"
```

### Troubleshooting

#### Common Issues

**Authentication Errors:**

```bash
# Verify AWS credentials
aws sts get-caller-identity

# Check IAM permissions
aws iam get-user

# Use specific profile
AWS_PROFILE=your-profile aws logs tail /aws/lambda/function
```

**Log Group Not Found:**

```bash
# List available log groups
aws logs describe-log-groups | grep logGroupName

# Search for specific pattern
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/

# Verify log group name (case-sensitive)
aws logs describe-log-groups --log-group-name /aws/lambda/my-function
```

**No Logs Appearing:**

```bash
# Check if logs exist
aws logs tail /aws/lambda/function --since 24h

# Verify time range
aws logs tail /aws/lambda/function --since 5m  # Last 5 minutes

# Check filter pattern
aws logs tail /aws/lambda/function --filter-pattern ""  # All logs
```

**Performance Issues:**

```bash
# Reduce log volume
aws logs tail /aws/lambda/function --filter-pattern "ERROR"

# Increase Gonzo buffers
aws logs tail /aws/lambda/function --follow | \
gonzo --log-buffer=20000 --memory-size=50000

# Use longer update intervals
aws logs tail /aws/lambda/function --follow | \
gonzo --update-interval=10s
```

### What's Next?

Now that you've mastered CloudWatch integration, explore related topics:

* **Kubernetes Integration** - Analyze EKS cluster logs
* **Configuration** - Optimize for CloudWatch workflows
* **AI Integration** - Enhanced AWS log analysis
* **Advanced Features** - Pattern detection in AWS logs

Or try these advanced CloudWatch patterns:

```bash
# Multi-region monitoring
for region in us-east-1 us-west-2 eu-west-1; do
    AWS_REGION=$region aws logs tail /aws/lambda/function --follow > logs-${region}.txt &
done
gonzo -f logs-*.txt --follow

# Cross-account analysis
AWS_PROFILE=account1 aws logs tail /aws/lambda/function --since 1h > account1.txt
AWS_PROFILE=account2 aws logs tail /aws/lambda/function --since 1h > account2.txt
gonzo -f account1.txt -f account2.txt --ai-model="gpt-4"
```

***

**Unlock the full potential of AWS CloudWatch Logs with Gonzo!** ☁️ From real-time Lambda monitoring to comprehensive AWS infrastructure analysis, Gonzo brings powerful log analysis to your CloudWatch workflows.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.controltheory.com/controltheory-documentation/gonzo-docs/integration-examples/aws-cloudwatch-logs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.