Stream CloudWatch Logs

This guide explains how to collect logs from AWS CloudWatch and forward them to your Dstl8 Docker agent using the OpenTelemetry Collector.

Overview

The setup uses the OpenTelemetry Collector with the AWS CloudWatch receiver to:

Poll CloudWatch log groups at regular intervals
Transform and enrich log data with AWS-specific attributes
Forward logs to the Dstl8 Docker agent via OTLP

Architecture:

AWS CloudWatch → OTel Collector (CloudWatch Receiver) → Dstl 8 Docker Agent (OTLP Receiver)

Prerequisites

Dstl8 Docker agent installed and running (see Docker Installation Guide)
Docker installed on the same host as the Dstl8 agent
AWS credentials with CloudWatch read access
Access to the AWS region containing your log groups

AWS IAM Permissions

Your AWS credentials need read-only access to CloudWatch Logs. AWS provides a managed policy for this:

Managed Policy: CloudWatchReadOnlyAccess Policy ARN: arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess

View policy details →

Alternatively, you can create a custom IAM policy with minimum required permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents",
        "logs:FilterLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Configuration

Step 1: Create the Collector Configuration File

Create a file named config.yaml with the following content. This configuration includes:

AWS CloudWatch receiver to poll log groups
Transform processor to enrich logs with AWS service metadata
OTLP exporter to forward logs to the Dstl8 agent

receivers:
  awscloudwatch:
    region: us-east-1  # Change to your AWS region
    logs:
      poll_interval: 1m
      groups:
        named:
          # Add your CloudWatch log groups here
          /aws/lambda/my-function:
          /aws/eks/production/my-cluster:
          /aws/ecs/my-service:

processors:
  batch:
    send_batch_size: 256
    send_batch_max_size: 512
    timeout: 1s
    
  transform:
    log_statements:
      - context: resource
        statements:
          # Store original attributes for reference
          - set(attributes["aws.log_group.original"], attributes["cloudwatch.log.group.name"]) where attributes["cloudwatch.log.group.name"] != nil
          - set(attributes["aws.log_stream.original"], attributes["cloudwatch.log.stream"]) where attributes["cloudwatch.log.stream"] != nil
          
          # Extract cloud provider attributes
          - set(attributes["cloud.provider"], "aws")
          - set(attributes["cloud.region"], attributes["aws.region"]) where attributes["aws.region"] != nil
          
          # === Lambda Detection: /aws/lambda/<function-name> ===
          - set(attributes["service.name"], Split(attributes["cloudwatch.log.group.name"], "/")[3]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/lambda/.*")
          - set(attributes["faas.name"], Split(attributes["cloudwatch.log.group.name"], "/")[3]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/lambda/.*")
          - set(attributes["cloud.platform"], "aws_lambda") where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/lambda/.*")
          
          # === EKS Detection: /aws/eks/<cluster-name>/cluster ===
          - set(attributes["k8s.cluster.name"], Split(attributes["cloudwatch.log.group.name"], "/")[3]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/eks/[^/]+/cluster$")
          - set(attributes["service.name"], Concat(["eks-", Split(attributes["cloudwatch.log.group.name"], "/")[3]], "")) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/eks/[^/]+/cluster$")
          - set(attributes["cloud.platform"], "aws_eks") where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/eks/.*")
          
          # Handle custom EKS patterns like /aws/eks/<env>/<cluster>
          - set(attributes["deployment.environment"], Split(attributes["cloudwatch.log.group.name"], "/")[3]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/eks/[^/]+/[^/]+$") and Len(Split(attributes["cloudwatch.log.group.name"], "/")) == 5
          - set(attributes["k8s.cluster.name"], Split(attributes["cloudwatch.log.group.name"], "/")[4]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/eks/[^/]+/[^/]+$") and Len(Split(attributes["cloudwatch.log.group.name"], "/")) == 5
          - set(attributes["k8s.pod.name"], attributes["cloudwatch.log.stream"]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/eks/.*")
          
          # === ECS/Fargate Detection: /ecs/<task-definition> ===
          - set(attributes["service.name"], Split(attributes["cloudwatch.log.group.name"], "/")[2]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/ecs/.*")
          - set(attributes["aws.ecs.task.family"], Split(attributes["cloudwatch.log.group.name"], "/")[2]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/ecs/.*")
          - set(attributes["cloud.platform"], "aws_ecs") where IsMatch(attributes["cloudwatch.log.group.name"], "^/ecs/.*")
          - set(attributes["container.name"], Split(attributes["cloudwatch.log.stream"], "/")[1]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/ecs/.*") and Len(Split(attributes["cloudwatch.log.stream"], "/")) >= 2
          - set(attributes["aws.ecs.task.id"], Split(attributes["cloudwatch.log.stream"], "/")[2]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/ecs/.*") and Len(Split(attributes["cloudwatch.log.stream"], "/")) >= 3
          
          # === RDS Detection: /aws/rds/instance/<instance>/<log-type> ===
          - set(attributes["db.system"], "rds") where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/rds/.*")
          - set(attributes["db.instance.id"], Split(attributes["cloudwatch.log.group.name"], "/")[4]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/rds/instance/.*")
          - set(attributes["service.name"], Concat(["rds-instance-", Split(attributes["cloudwatch.log.group.name"], "/")[4]], "")) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/rds/instance/.*")
          - set(attributes["db.cluster.id"], Split(attributes["cloudwatch.log.group.name"], "/")[4]) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/rds/cluster/.*")
          - set(attributes["service.name"], Concat(["rds-cluster-", Split(attributes["cloudwatch.log.group.name"], "/")[4]], "")) where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/rds/cluster/.*")
          - set(attributes["cloud.platform"], "aws_rds") where IsMatch(attributes["cloudwatch.log.group.name"], "^/aws/rds/.*")
          
          # === API Gateway Detection: API-Gateway-Execution-Logs_<api-id>/<stage> ===
          - set(attributes["service.name"], Split(attributes["cloudwatch.log.group.name"], "_")[1]) where IsMatch(attributes["cloudwatch.log.group.name"], "^API-Gateway-Execution-Logs_.*")
          - set(attributes["aws.api_gateway.api_id"], Split(Split(attributes["cloudwatch.log.group.name"], "_")[1], "/")[0]) where IsMatch(attributes["cloudwatch.log.group.name"], "^API-Gateway-Execution-Logs_.*")
          - set(attributes["cloud.platform"], "aws_api_gateway") where IsMatch(attributes["cloudwatch.log.group.name"], "^API-Gateway-Execution-Logs_.*")
          
          # === Fallback: Use log group name as service name if no pattern matched ===
          - set(attributes["service.name"], attributes["cloudwatch.log.group.name"]) where attributes["service.name"] == nil and attributes["cloudwatch.log.group.name"] != nil

exporters:
  otlp:
    endpoint: localhost:4317  # Dstl 8 Docker agent OTLP endpoint
    compression: none
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers:
        - awscloudwatch
      processors:
        - batch
        - transform
      exporters:
        - otlp

Step 2: Customize the Configuration

Update the following sections in your config.yaml:

1. AWS Region:

receivers:
  awscloudwatch:
    region: us-east-1  # Change to your AWS region (e.g., us-west-2, eu-west-1)

2. CloudWatch Log Groups:

Add the log groups you want to monitor under groups.named:

receivers:
  awscloudwatch:
    region: us-east-1
    logs:
      poll_interval: 1m
      groups:
        named:
          /aws/lambda/my-function:
          /aws/lambda/another-function:
          /aws/eks/production/my-cluster:
          /aws/ecs/my-api-service:
          /var/log/application.log:

3. OTLP Exporter Endpoint:

The endpoint depends on your operating system and network configuration:

Linux (Docker default bridge network):

exporters:
  otlp:
    endpoint: 172.17.0.1:4317  # Docker host gateway

macOS:

exporters:
  otlp:
    endpoint: host.docker.internal:4317  # Docker Desktop special DNS name

Windows:

exporters:
  otlp:
    endpoint: host.docker.internal:4317  # Docker Desktop special DNS name

Same Docker Network (if both containers are on the same custom network):

exporters:
  otlp:
    endpoint: dstl8-agent:4317  # Use container name

Running the OpenTelemetry Collector

Using AWS Access Keys

Once your config.yaml is configured, start the OpenTelemetry Collector:

Linux/macOS:

docker run -d \
  --name otel-cloudwatch-collector \
  -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
  -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY \
  -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY \
  otel/opentelemetry-collector-contrib:latest \
  --config /etc/otelcol-contrib/config.yaml

Windows (PowerShell):

docker run -d `
  --name otel-cloudwatch-collector `
  -v ${PWD}/config.yaml:/etc/otelcol-contrib/config.yaml `
  -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY `
  -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY `
  otel/opentelemetry-collector-contrib:latest `
  --config /etc/otelcol-contrib/config.yaml

Using AWS IAM Role (EC2 or ECS)

If running on an EC2 instance or ECS task with an IAM role attached, you don't need to provide credentials:

docker run -d \
  --name otel-cloudwatch-collector \
  -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
  otel/opentelemetry-collector-contrib:latest \
  --config /etc/otelcol-contrib/config.yaml

The collector will automatically use the instance's IAM role credentials.

Using AWS Profile

If you have AWS credentials configured in ~/.aws/credentials:

docker run -d \
  --name otel-cloudwatch-collector \
  -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
  -v ~/.aws:/root/.aws:ro \
  -e AWS_PROFILE=default \
  otel/opentelemetry-collector-contrib:latest \
  --config /etc/otelcol-contrib/config.yaml

Verification

Check Collector Status

View collector logs:

docker logs otel-cloudwatch-collector

Follow logs in real-time:

docker logs -f otel-cloudwatch-collector

Successful startup should show:

2024-03-15T10:30:00.000Z info service/service.go:143 Everything is ready. Begin running and processing data.

Check Dstl8 Agent

Verify the Dstl8 agent is receiving logs:

# Check Dstl 8 agent logs
docker logs <dstl8-agent-container-id>

# Should show incoming OTLP connections

Verify in Dstl 8 UI

Log into your Dstl 8 dashboard and verify:

CloudWatch log groups appear as data sources
Logs are being ingested with proper AWS attributes
Service names are correctly extracted from log group names

Troubleshooting

Collector Can't Connect to Dstl 8 Agent

Symptom: Logs show connection refused or timeout errors

Solutions:

Verify Dstl 8 agent is running:
```
docker ps | grep controltheory
```
Check OTLP endpoint in config.yaml:
- Linux: Try 172.17.0.1:4317 or localhost:4317
- macOS/Windows: Use host.docker.internal:4317

Test connectivity from collector container:

docker exec otel-cloudwatch-collector ping 172.17.0.1

Verify port 4317 is exposed on Dstl 8 agent:
```
docker port <dstl8-agent-container-id>
```

AWS Authentication Errors

Symptom: Logs show "AccessDenied" or authentication errors

Solutions:

Verify AWS credentials are correct:

docker exec otel-cloudwatch-collector env | grep AWS

Check IAM permissions: Ensure the IAM user/role has CloudWatchReadOnlyAccess or equivalent permissions
Verify region is correct: Check that the region in config.yaml matches where your log groups exist

No Logs Being Collected

Symptom: Collector runs but no logs appear in Dstl 8

Solutions:

Verify log groups exist:

aws logs describe-log-groups --region us-east-1

Check log group names in config.yaml: Ensure they exactly match (case-sensitive)
Verify poll interval: Default is 1 minute, wait at least that long
Check for recent log entries: The collector only fetches recent logs, not historical data by default

High Memory or CPU Usage

Symptom: Collector consumes excessive resources

Solutions:

Reduce batch size in config.yaml:

processors:
  batch:
    send_batch_size: 128  # Reduced from 256
    send_batch_max_size: 256  # Reduced from 512

Increase poll interval:

receivers:
  awscloudwatch:
    logs:
      poll_interval: 5m  # Increased from 1m

Limit the number of log groups being monitored

Advanced Configuration

Filtering Log Groups by Prefix

Instead of listing individual log groups, you can use prefixes:

receivers:
  awscloudwatch:
    region: us-east-1
    logs:
      poll_interval: 1m
      groups:
        prefixes:
          - /aws/lambda/
          - /aws/eks/

This will collect logs from all log groups matching the prefixes.

Custom Poll Intervals

Adjust polling frequency based on your needs:

receivers:
  awscloudwatch:
    region: us-east-1
    logs:
      poll_interval: 30s  # Poll every 30 seconds (more frequent)
      # OR
      poll_interval: 5m   # Poll every 5 minutes (less frequent)

Note: More frequent polling increases AWS API calls and costs.

Adding Debug Exporter

For troubleshooting, add a debug exporter to see logs in collector output:

exporters:
  otlp:
    endpoint: localhost:4317
    compression: none
    tls:
      insecure: true
  
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers:
        - awscloudwatch
      processors:
        - batch
        - transform
      exporters:
        - otlp
        - debug  # Add debug exporter

Then view detailed logs:

docker logs -f otel-cloudwatch-collector

Running on a Custom Docker Network

To improve networking between containers, create a custom network:

# Create network
docker network create dstl8-network

# Connect Dstl 8 agent to network
docker network connect dstl8-network <dstl8-agent-container-id>

# Run collector on same network
docker run -d \
  --name otel-cloudwatch-collector \
  --network dstl8-network \
  -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml \
  -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY \
  -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY \
  otel/opentelemetry-collector-contrib:latest \
  --config /etc/otelcol-contrib/config.yaml

Update the endpoint in config.yaml:

exporters:
  otlp:
    endpoint: <dstl8-agent-container-name>:4317

Managing the Collector

Stop the Collector

docker stop otel-cloudwatch-collector

Start the Collector

docker start otel-cloudwatch-collector

Restart the Collector

docker restart otel-cloudwatch-collector

Update Configuration

After modifying config.yaml, restart the collector:

docker restart otel-cloudwatch-collector

Remove the Collector

docker stop otel-cloudwatch-collector
docker rm otel-cloudwatch-collector

Additional Resources

PreviousDocker NextAgent Health Monitoring

Last updated 1 month ago

hashtagOverview

hashtagPrerequisites

hashtagAWS IAM Permissions

hashtagConfiguration

hashtagStep 1: Create the Collector Configuration File

hashtagStep 2: Customize the Configuration

hashtagRunning the OpenTelemetry Collector

hashtagUsing AWS Access Keys

hashtagUsing AWS IAM Role (EC2 or ECS)

hashtagUsing AWS Profile

hashtagVerification

hashtagCheck Collector Status

hashtagCheck Dstl8 Agent

hashtagVerify in Dstl 8 UI

hashtagTroubleshooting

hashtagCollector Can't Connect to Dstl 8 Agent

hashtagAWS Authentication Errors

hashtagNo Logs Being Collected

hashtagHigh Memory or CPU Usage

hashtagAdvanced Configuration

hashtagFiltering Log Groups by Prefix

hashtagCustom Poll Intervals

hashtagAdding Debug Exporter

hashtagRunning on a Custom Docker Network

hashtagManaging the Collector

hashtagStop the Collector

hashtagStart the Collector

hashtagRestart the Collector

hashtagUpdate Configuration

hashtagRemove the Collector

hashtagAdditional Resources

Overview

Prerequisites

AWS IAM Permissions

Configuration

Step 1: Create the Collector Configuration File

Step 2: Customize the Configuration

Running the OpenTelemetry Collector

Using AWS Access Keys

Using AWS IAM Role (EC2 or ECS)

Using AWS Profile

Verification

Check Collector Status

Check Dstl8 Agent

Verify in Dstl 8 UI

Troubleshooting

Collector Can't Connect to Dstl 8 Agent

AWS Authentication Errors

No Logs Being Collected

High Memory or CPU Usage

Advanced Configuration

Filtering Log Groups by Prefix

Custom Poll Intervals

Adding Debug Exporter

Running on a Custom Docker Network

Managing the Collector

Stop the Collector

Start the Collector

Restart the Collector

Update Configuration

Remove the Collector

Additional Resources