ControlTheory Website

AWS CloudWatch ☁️

AWS CloudWatch Integration

Analyze AWS CloudWatch Logs with Gonzo for powerful, interactive log analysis right from your terminal. Stream logs in real-time or analyze historical data with AI-powered insights.

Prerequisites: AWS CLI installed and configured with appropriate credentials. Use aws configure to set up your AWS access.

Why Use Gonzo with CloudWatch?

AWS CloudWatch Logs is a powerful log aggregation service, but analyzing logs in the console can be limiting. Gonzo brings:

  • 🎨 Visual Analysis - Beautiful terminal interface vs web console

  • 🤖 AI Insights - Intelligent pattern recognition and root cause analysis

  • ⚡ Real-Time Streaming - Live log analysis as events occur

  • 🔍 Advanced Filtering - Powerful regex and search capabilities

  • 📊 Pattern Detection - Automatic clustering of similar log entries

  • 💰 Cost Efficiency - Analyze logs locally without CloudWatch Insights costs

Quick Start

Getting Your Log Groups

First, discover what log groups are available in your AWS region:

# List all log groups with names and ARNs
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,ARN:arn}' \
  --output json

# List log groups in table format
aws logs describe-log-groups \
  --query 'logGroups[*].{Name:logGroupName,ARN:arn}' \
  --output table

# Filter by prefix
aws logs describe-log-groups \
  --log-group-name-prefix /aws/lambda/ \
  --query 'logGroups[*].logGroupName' \
  --output text

Basic Log Tailing with Gonzo

The simplest way to use Gonzo with CloudWatch Logs is to pipe aws logs tail output:

# Basic tail (default: last 10 minutes)
aws logs tail "/aws/eks/stage/cluster" --follow | gonzo

# Tail with JSON output for better attribute extraction
aws logs tail "/aws/eks/stage/cluster" --follow --format json | gonzo

# Tail with AI analysis
aws logs tail "/aws/eks/stage/cluster" --follow --format json | \
gonzo --ai-model="gpt-4"

Output Format: The default "detailed" format outputs one log per line (non-JSON). Use --format json to gain attribute visibility in Gonzo, though each JSON log message spans two lines.

Multiple Log Groups

Monitor multiple log groups simultaneously:

# Method 1: Using process substitution
gonzo - < <(
  aws logs tail "/aws/eks/stage/cluster" --follow --format json &
  aws logs tail "RDS" --follow --format json &
  wait
)

# Method 2: Named pipes (alternative)
mkfifo log_stream
(aws logs tail "/aws/eks/stage/cluster" --follow --format json &
 aws logs tail "RDS" --follow --format json) > log_stream &
gonzo - < log_stream

# Method 3: File-based aggregation
aws logs tail "/aws/eks/stage/cluster" --follow --format json > logs1.txt &
aws logs tail "RDS" --follow --format json > logs2.txt &
gonzo -f logs1.txt -f logs2.txt --follow

How This Works:

  • Starts multiple aws logs tail processes in the background (&)

  • Combines their output streams

  • Pipes the combined output to Gonzo

  • Uses wait to ensure all background processes complete properly

AWS CLI Live Tail Integration

Live Tail provides near real-time streaming of log events as they are ingested into CloudWatch.

Basic Live Tail Usage

# Single log group with ARN
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" | gonzo

# Multiple log groups
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" \
  "arn:aws:logs:us-east-1:767397775588:log-group:RDS" | gonzo

# With AI analysis
aws logs start-live-tail --log-group-identifiers \
  "arn:aws:logs:us-east-1:767397775588:log-group:/aws/eks/stage/cluster" | \
gonzo --ai-model="gpt-4"

Live Tail Benefits:

  • ✅ Near real-time streaming (lower latency than tail)

  • ✅ Native JSON output (better attribute extraction)

  • ✅ Supports multiple log groups natively

  • ✅ Automatic handling of high-throughput log groups

Live Tail vs Regular Tail

Feature

aws logs tail

aws logs start-live-tail

Latency

~1-2 seconds

Near real-time (<1 second)

Multiple Log Groups

Requires workarounds

Native support

Output Format

Detailed (default) or JSON

JSON (default)

Historical Logs

Yes (--since flag)

No (only new logs)

Best For

Historical analysis

Real-time monitoring

Choose Live Tail When:

  • Monitoring production incidents in real-time

  • Need lowest possible latency

  • Monitoring multiple log groups simultaneously

  • JSON output is preferred

Choose Regular Tail When:

  • Need historical log analysis

  • Want to start from specific time (--since)

  • Need simple single log group monitoring

AWS CLI Integration

Installing and Configuring AWS CLI

# Install AWS CLI (if not already installed)
# macOS
brew install awscli

# Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Windows
# Download from: https://awscli.amazonaws.com/AWSCLIV2.msi

# Configure AWS CLI
aws configure
# Enter:
#   AWS Access Key ID
#   AWS Secret Access Key
#   Default region (e.g., us-east-1)
#   Default output format (json recommended)

# Verify configuration
aws sts get-caller-identity

Essential CloudWatch Logs Commands

# List all log groups
aws logs describe-log-groups

# List with specific prefix
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/

# Get specific log group details
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/my-function

# Count log groups
aws logs describe-log-groups --query 'length(logGroups)'

Real-World Integration Patterns

Lambda Function Monitoring

# Monitor specific Lambda function
aws logs tail /aws/lambda/my-function --follow | gonzo

# Monitor with AI for error analysis
aws logs tail /aws/lambda/my-function --follow | \
gonzo --ai-model="gpt-4"

# Monitor multiple Lambda functions
(aws logs tail /aws/lambda/function1 --follow & \
 aws logs tail /aws/lambda/function2 --follow) | gonzo

# Focus on errors only
aws logs tail /aws/lambda/my-function --follow | \
grep -i error | \
gonzo --log-buffer=5000

ECS/Fargate Container Logs

# ECS task logs
aws logs tail /ecs/my-service --follow | gonzo

# Filter by container
aws logs tail /ecs/my-service --follow | \
grep "container-name" | \
gonzo

# Multiple ECS services
(aws logs tail /ecs/api --follow & \
 aws logs tail /ecs/worker --follow) | \
gonzo --log-buffer=10000

API Gateway Logs

# API Gateway execution logs
aws logs tail /aws/apigateway/my-api --follow | gonzo

# Filter by HTTP status
aws logs tail /aws/apigateway/my-api --follow | \
grep -E "(4[0-9]{2}|5[0-9]{2})" | \
gonzo --ai-model="gpt-4"

# Access logs analysis
aws logs tail /aws/apigateway/my-api/access --since 1h | \
gonzo --log-buffer=20000

RDS Database Logs

# RDS error logs
aws logs tail /aws/rds/instance/my-db/error --follow | gonzo

# Slow query logs
aws logs tail /aws/rds/instance/my-db/slowquery --follow | \
gonzo --ai-model="gpt-4"

# General logs
aws logs tail /aws/rds/instance/my-db/general --since 1h | gonzo

CloudTrail Logs

# Monitor CloudTrail events
aws logs tail aws-cloudtrail-logs --follow | gonzo

# Security-focused monitoring
aws logs tail aws-cloudtrail-logs --follow | \
grep -E "(unauthorized|denied|failed)" | \
gonzo --ai-model="gpt-4"

# Specific service events
aws logs tail aws-cloudtrail-logs --follow | \
grep "ec2.amazonaws.com" | \
gonzo

Advanced Usage Patterns

Multi-Account Monitoring

# Switch AWS profiles
export AWS_PROFILE=production
aws logs tail /aws/lambda/function --follow | gonzo

# Monitor multiple accounts simultaneously
AWS_PROFILE=prod aws logs tail /aws/lambda/api --follow > prod-logs.txt &
AWS_PROFILE=staging aws logs tail /aws/lambda/api --follow > staging-logs.txt &
gonzo -f prod-logs.txt -f staging-logs.txt --follow

# Account-specific aliases
alias gonzo-prod='AWS_PROFILE=production aws logs tail'
alias gonzo-staging='AWS_PROFILE=staging aws logs tail'

# Usage
gonzo-prod /aws/lambda/function --follow | gonzo

Cost-Optimized Analysis

# Export logs once, analyze multiple times
aws logs tail /aws/lambda/function --since 24h > logs-$(date +%Y%m%d).txt

# Analyze without streaming (no ongoing CloudWatch costs)
gonzo -f logs-20240115.txt --ai-model="gpt-4"

# Batch processing for cost savings
for log_group in $(aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output text); do
    echo "Analyzing $log_group..."
    aws logs tail "$log_group" --since 1h > "logs/${log_group//\//_}.txt"
done

# Analyze all exported logs
gonzo -f logs/*.txt

Automated Monitoring Scripts

#!/bin/bash
# monitor-cloudwatch.sh
# Continuous monitoring with alerting

LOG_GROUP="$1"
ALERT_PATTERN="${2:-ERROR}"
ALERT_WEBHOOK="${3}"

echo "Monitoring $LOG_GROUP for pattern: $ALERT_PATTERN"

aws logs tail "$LOG_GROUP" --follow | \
while IFS= read -r line; do
    echo "$line"
    
    # Check for alert pattern
    if echo "$line" | grep -qi "$ALERT_PATTERN"; then
        # Send alert
        if [ -n "$ALERT_WEBHOOK" ]; then
            curl -X POST "$ALERT_WEBHOOK" \
                -H "Content-Type: application/json" \
                -d "{\"text\": \"Alert: $line\"}"
        fi
        
        # Analyze with Gonzo + AI
        echo "$line" | gonzo --ai-model="gpt-4"
    fi
done

Performance Analysis

# Analyze Lambda cold starts
aws logs tail /aws/lambda/function --since 1h | \
grep "Init Duration" | \
gonzo --ai-model="gpt-4"

# Memory usage analysis
aws logs tail /aws/lambda/function --since 1h | \
grep "Max Memory Used" | \
gonzo

# Duration analysis
aws logs tail /aws/lambda/function --since 1h | \
grep "Duration" | \
gonzo --log-buffer=20000

CloudWatch Logs Insights Integration

Export Insights Query Results

# Run CloudWatch Logs Insights query
query_id=$(aws logs start-query \
    --log-group-name /aws/lambda/my-function \
    --start-time $(date -u -d '1 hour ago' +%s) \
    --end-time $(date -u +%s) \
    --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 1000' \
    --query 'queryId' \
    --output text)

# Wait for query to complete
aws logs wait query-complete --query-id "$query_id"

# Get results and analyze with Gonzo
aws logs get-query-results --query-id "$query_id" | \
jq -r '.results[] | .[1].value' | \
gonzo --ai-model="gpt-4"

Configuration for CloudWatch

Optimized Configuration File

# ~/.config/gonzo/cloudwatch.yml
# Optimized for CloudWatch Logs

# CloudWatch logs are streamed via stdin
# No file configuration needed

# Performance for CloudWatch streaming
update-interval: 2s              # Responsive updates
log-buffer: 10000               # Large buffer for CloudWatch volume
memory-size: 50000              # Extensive pattern tracking

# AI configuration for AWS analysis
ai-model: "gpt-4"               # Best for complex AWS issues
ai-context-size: 8000           # Large context for AWS logs

# Display
show-service: true              # Show AWS service context

Usage:

aws logs tail /aws/lambda/function --follow | \
gonzo --config ~/.config/gonzo/cloudwatch.yml

Environment-Specific Aliases

# Add to ~/.bashrc or ~/.zshrc

# Production monitoring
alias gonzo-prod='AWS_PROFILE=production aws logs tail --follow'

# Development monitoring
alias gonzo-dev='AWS_PROFILE=development aws logs tail --follow'

# Lambda-specific
alias gonzo-lambda='aws logs tail --follow'

# With AI analysis
alias gonzo-lambda-ai='aws logs tail --follow | gonzo --ai-model="gpt-4"'

# Usage
gonzo-lambda /aws/lambda/my-function
gonzo-lambda-ai /aws/lambda/critical-function

Complete Guide

For step-by-step CloudWatch integration: 📖 Gonzo: A Better Way to Tail AWS CloudWatch Logs

Best Practices

🎯 Efficient Log Retrieval

# Use time ranges to limit data
aws logs tail /aws/lambda/function --since 1h    # Last hour only

# Use filter patterns for pre-filtering
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow

# Export for repeated analysis
aws logs tail /aws/lambda/function --since 24h > daily-export.txt
gonzo -f daily-export.txt  # No repeated CloudWatch API calls

Cost Considerations

Important: AWS costs may be incurred for using the AWS CLI tail or start-live-tail commands. Please consult the AWS CloudWatch Logs pricing documentation for details on potential charges related to log data retrieval.

Understanding CloudWatch Logs Pricing

Data Retrieval Costs:

  • GetLogEvents API: Used by aws logs tail - included with storage

  • StartLiveTail API: Used by aws logs start-live-tail - additional charges may apply

  • Data Transfer: Egress charges may apply for large volumes

Cost Optimization Strategies

# 1. Use time-based filtering to reduce data volume
aws logs tail /aws/lambda/function --since 1h --follow | gonzo  # Last hour only

# 2. Use filter patterns at the source
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow | gonzo

# 3. Export once, analyze multiple times (no repeated API calls)
aws logs tail /aws/lambda/function --since 24h > daily-export.txt
gonzo -f daily-export.txt  # Analyze locally
gonzo -f daily-export.txt --ai-model="gpt-4"  # Re-analyze with AI

# 4. Use regular tail instead of live tail when real-time isn't critical
aws logs tail /aws/lambda/function --follow | gonzo  # Lower cost

# 5. Monitor specific log groups instead of all logs
aws logs tail "/aws/lambda/critical-function" --follow | gonzo  # Focused monitoring

Cost Comparison Example:

# Higher cost: Live tail multiple groups for 24 hours
aws logs start-live-tail --log-group-identifiers arn1 arn2 arn3 | gonzo

# Lower cost: Regular tail with filtering
aws logs tail "/aws/lambda/function" --filter-pattern "ERROR" --follow | gonzo

# Lowest cost: Export and analyze locally
aws logs tail "/aws/lambda/function" --since 1h > export.txt
gonzo -f export.txt  # No ongoing API costs

🔒 Security Best Practices

# Use IAM roles instead of access keys
aws configure set role_arn arn:aws:iam::123456789012:role/LogAnalyst
aws configure set source_profile default

# Least privilege IAM policy for Gonzo usage:
# {
#   "Version": "2012-10-17",
#   "Statement": [{
#     "Effect": "Allow",
#     "Action": [
#       "logs:DescribeLogGroups",
#       "logs:DescribeLogStreams",
#       "logs:GetLogEvents",
#       "logs:FilterLogEvents"
#     ],
#     "Resource": "arn:aws:logs:*:*:log-group:/aws/lambda/*"
#   }]
# }

# Use MFA for production access
aws configure set mfa_serial arn:aws:iam::123456789012:mfa/user

# Audit log access
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=/aws/lambda/function

⚡ Performance Optimization

# Pre-filter for performance
aws logs tail /aws/lambda/function --filter-pattern "ERROR" --follow | gonzo

# Adjust Gonzo buffers for volume
aws logs tail /aws/lambda/high-traffic --follow | \
gonzo --log-buffer=20000 --update-interval=5s

# Use local AI for sensitive logs
export OPENAI_API_BASE="http://localhost:11434"
aws logs tail /aws/lambda/sensitive --follow | \
gonzo --ai-model="llama3"

Troubleshooting

Common Issues

Authentication Errors:

# Verify AWS credentials
aws sts get-caller-identity

# Check IAM permissions
aws iam get-user

# Use specific profile
AWS_PROFILE=your-profile aws logs tail /aws/lambda/function

Log Group Not Found:

# List available log groups
aws logs describe-log-groups | grep logGroupName

# Search for specific pattern
aws logs describe-log-groups --log-group-name-prefix /aws/lambda/

# Verify log group name (case-sensitive)
aws logs describe-log-groups --log-group-name /aws/lambda/my-function

No Logs Appearing:

# Check if logs exist
aws logs tail /aws/lambda/function --since 24h

# Verify time range
aws logs tail /aws/lambda/function --since 5m  # Last 5 minutes

# Check filter pattern
aws logs tail /aws/lambda/function --filter-pattern ""  # All logs

Performance Issues:

# Reduce log volume
aws logs tail /aws/lambda/function --filter-pattern "ERROR"

# Increase Gonzo buffers
aws logs tail /aws/lambda/function --follow | \
gonzo --log-buffer=20000 --memory-size=50000

# Use longer update intervals
aws logs tail /aws/lambda/function --follow | \
gonzo --update-interval=10s

What's Next?

Now that you've mastered CloudWatch integration, explore related topics:

  • Kubernetes Integration - Analyze EKS cluster logs

  • Configuration - Optimize for CloudWatch workflows

  • AI Integration - Enhanced AWS log analysis

  • Advanced Features - Pattern detection in AWS logs

Or try these advanced CloudWatch patterns:

# Multi-region monitoring
for region in us-east-1 us-west-2 eu-west-1; do
    AWS_REGION=$region aws logs tail /aws/lambda/function --follow > logs-${region}.txt &
done
gonzo -f logs-*.txt --follow

# Cross-account analysis
AWS_PROFILE=account1 aws logs tail /aws/lambda/function --since 1h > account1.txt
AWS_PROFILE=account2 aws logs tail /aws/lambda/function --since 1h > account2.txt
gonzo -f account1.txt -f account2.txt --ai-model="gpt-4"

Unlock the full potential of AWS CloudWatch Logs with Gonzo! ☁️ From real-time Lambda monitoring to comprehensive AWS infrastructure analysis, Gonzo brings powerful log analysis to your CloudWatch workflows.

PreviousDocker 🐳NextGrafana Loki Integration 🔥

Last updated