> For the complete documentation index, see [llms.txt](https://docs.controltheory.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/aws-cloudwatch.md). # AWS CloudWatch Stream logs from AWS CloudWatch Log Groups into Dstl8. Dstl8 receives CloudWatch Logs through an Amazon Data Firehose stream in your own AWS account. A CloudFormation stack that you launch from Dstl8 sets everything up: * a **Firehose stream** that delivers to your Dstl8 source's endpoint, authenticated with the source's access key; * either an **account-level subscription filter policy** that sends every log group in the region (Standard log class) to that stream, including log groups created later, except the ones you leave out; or, if you'd rather pick, a **subscription filter on each log group you choose**; * an **S3 bucket** that keeps any batch Dstl8 doesn't accept within the retry window, so nothing is lost if delivery is interrupted; * the two **IAM roles** Firehose and CloudWatch Logs need to do this. Dstl8 gets no access to your AWS account, and you don't create any IAM users or AWS access keys for it; the only credential involved is your Dstl8 source's own access key, which Firehose sends with each delivery. Logs usually arrive within a minute or two of being written, and busy log groups aren't limited by CloudWatch's read quotas the way polling was. ### Before you start You need permission in the AWS account to deploy the stack: create CloudFormation stacks, Firehose streams, S3 buckets, IAM roles (including `iam:PassRole`), CloudWatch Logs log groups, and CloudWatch Logs subscriptions (`logs:PutSubscriptionFilter`, plus `logs:PutAccountPolicy` for the all-log-groups mode), and to delete each of these, so a failed create can roll back cleanly. An account administrator can launch it for you from the setup link. The dstl8 CLI's `--deploy` and log group picking also read from the account: `logs:DescribeLogGroups` to list log groups, `logs:DescribeAccountPolicies` to check for an existing account-level policy, and `logs:DescribeSubscriptionFilters` to spot log groups that can't take another subscription. Launching from the console needs none of these. {% hint style="info" %} An AWS account can have only **one** subscription filter account policy per region. If yours already has one (for example for central logging or another vendor), the all-log-groups mode fails at its **SubscriptionPolicy** step with `Resource limit exceeded` and rolls back without leaving anything behind. Choose **Only the log groups I choose** instead (see [Send only chosen log groups](#send-only-chosen-log-groups)); it doesn't need an account policy. {% endhint %} ### Add a CloudWatch source #### 1. Create the source From **Sources**, click **Add Source**, select **AWS CloudWatch**, give the source a name and click **Create**. Dstl8 creates the source's endpoint and access key. One source can receive every region of an AWS account. Add a separate source per AWS account. #### 2. Launch the stack On the **Connect your AWS account** step: 1. Pick the **Region** whose log groups you want to send. 2. Choose which log groups to send: **All log groups** (the default; new log groups are included automatically) or **Only the log groups I choose**. See [Choose what's sent](#choose-whats-sent). 3. With all log groups, optionally list **Log groups to leave out**: exact names, one per line or comma separated. With chosen log groups, tick the ones to send (see [Send only chosen log groups](#send-only-chosen-log-groups)). 4. Click **Launch stack in AWS**. The AWS console opens the stack with the endpoint, access key, source ID and your log group choice already filled in. 5. In the console, check the boxes acknowledging that the stack creates IAM roles and uses CloudFormation transforms, and click **Create stack**. The stack takes two to three minutes to create. Repeat for each region you want to collect from; every stack sends to the same source. #### 3. Wait for the first logs Keep the Dstl8 dialog open. It shows **Waiting for first data** until the first batch arrives, usually within a minute or two of the stack finishing, and then **Receiving logs**. You can also close it and check later: the source page shows the same status, and **Connection Details** reopens the setup. ### Streams Each log group becomes a stream, named the way you'd look for it. Three kinds of log group are split by their log streams instead: `RDSOSMetrics`, where AWS writes every DB instance's Enhanced Monitoring data to one log group, becomes one stream per instance; `/aws-glue/crawlers` becomes one stream per crawler; and `/ec2/...` log groups become one stream per log stream (usually one per instance). | Log group | Stream type | Stream name | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/aws/lambda/` | `lambda` | the function name | | `/ecs/` | `ecs` | the service name | | `/aws/eks//...` | `eks` | the cluster name | | `/aws/rds/instance//` | `rds` | the instance name | | `RDSOSMetrics` | `rds` | the instance's resource ID (`db-…`) | | `API-Gateway-Execution-Logs_/` | `api-gateway` | the API ID | | `/aws/states/` | `stepfunctions` | the state machine | | `/aws/codebuild/` | `codebuild` | the project | | `/aws/elasticbeanstalk//...` | `elasticbeanstalk` | the environment | | `/aws/vpc/flowlogs/...` | `vpc-flowlogs` | the flow log | | `/aws-glue/jobs/...`, `/aws-glue/python-jobs/...`, `/aws-glue/sessions/...`, `/aws-glue/ray/...` | `glue` | `glue-jobs`, `glue-python-jobs`, `glue-sessions` or `glue-ray-jobs`. Glue writes each run to its own log stream named by the run ID, so runs of the same kind share one stream | | `/aws-glue/crawlers` | `glue` | the crawler name | | `/ec2/` | `ec2/` (the log group without its leading `/`) | the log stream name | | any other log group with CloudTrail in its name | `cloudtrail` | the log group name without its leading `/` | | any other log group | `cloudwatch` | the log group name without its leading `/` | For example, `/aws/cloudtrail/main` becomes the stream `aws/cloudtrail/main`; use that exact name with `dstl8 sources streams assign` and in filters. Streams from different regions stay apart: a stream's cluster is its AWS region, except EKS streams, whose cluster is the EKS cluster name. Streams need to be assigned to a workspace to be accessible: open the source's actions menu and select **Assign Streams**. ### Choose what's sent The stack works in one of two modes, set when you launch it (its **Mode** parameter): * **All log groups** (default): **every log group in the region, including ones created later, except the ones you list.** One account-level subscription policy does this, so it needs the region's one policy slot. The stack's own Firehose log group (`/aws/kinesisfirehose/`) is always left out, so delivery errors never loop back into Dstl8. * **Only the log groups I choose:** a subscription filter on each log group you list, and nothing else. It works alongside an existing account-level policy. Log groups created later are **not** added until you add them to the list. See [Send only chosen log groups](#send-only-chosen-log-groups). In both modes only log groups in the Standard log class are sent: AWS doesn't allow subscriptions on Infrequent Access log groups. Names are exact in both lists; there are no wildcards or prefixes in the stack itself (the dstl8 CLI can expand prefixes for you when it creates the stack). #### What to leave out Leave out what you don't want analyzed, or what's large and adds little: | Log group | Why people leave it out | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `/aws/vpc/flowlogs/` | VPC flow logs: very high volume, network metadata rather than application logs | | `/aws/eks//cluster` | EKS control-plane logs (audit, API server): high volume; keep them if you investigate cluster problems | | `aws-cloudtrail-logs--`, `/aws/cloudtrail/...` | CloudTrail copies: audit events you may already review elsewhere | | `/aws/lambda/` for dev or test functions | Noise from environments you don't need incidents for | | Anything you already ship to Dstl8 another way | Avoids counting the same logs twice | To see which groups are largest, sort **CloudWatch → Log groups** by **Stored bytes**, or run: ```bash aws logs describe-log-groups --region us-east-1 \ --query 'reverse(sort_by(logGroups,&storedBytes))[:20].[logGroupName,storedBytes]' --output table ``` Stored bytes reflects history. For what a group writes now, look at its **IncomingBytes** metric in CloudWatch. #### Where to set exclusions * **In Dstl8, when you launch the stack:** one name per line, or comma separated, in **Log groups to leave out**. They're passed to the stack as its **ExcludeLogGroups** parameter. * **In the AWS console, later:** open the stack, choose **Update** → **Replace existing template** → **Amazon S3 URL**, paste the template URL below, and on the parameters page edit **ExcludeLogGroups**. It's a single comma-separated value, for example: ``` /aws/vpc/flowlogs/main,/aws/eks/prod/cluster ``` * **With the dstl8 CLI, when you create the source:** ```bash dstl8 sources add cloudwatch --regions us-east-1 \ --exclude /aws/vpc/flowlogs/main --exclude /aws/eks/prod/cluster ``` * **With the AWS CLI, on an existing stack:** pass the template URL again and keep every other parameter as it is. The stack name is the one from the setup link: `dstl8-` plus your source name, lowercased, with anything other than letters, digits and hyphens turned into `-`, up to 64 characters (for example, "Prod CW (us-east-1)" becomes `dstl8-prod-cw-us-east-1`). The stack list in the CloudFormation console shows it too. In the AWS CLI, commas inside a value are escaped with `\`. ```bash aws cloudformation update-stack --region us-east-1 --stack-name \ --template-url https://dstl8-public-templates.s3.amazonaws.com/cloudwatch-push/v2/dstl8-cloudwatch-push.yaml \ --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \ --parameters 'ParameterKey=ExcludeLogGroups,ParameterValue=/aws/vpc/flowlogs/main\,/aws/eks/prod/cluster' \ ParameterKey=Mode,UsePreviousValue=true ParameterKey=LogGroups,UsePreviousValue=true \ ParameterKey=EndpointUrl,UsePreviousValue=true ParameterKey=AccessKey,UsePreviousValue=true \ ParameterKey=SourceId,UsePreviousValue=true ParameterKey=BufferSizeMiB,UsePreviousValue=true \ ParameterKey=BufferIntervalSec,UsePreviousValue=true ParameterKey=RetrySeconds,UsePreviousValue=true \ ParameterKey=BackupRetentionDays,UsePreviousValue=true ``` Always update with the template URL rather than the console's **Use existing template** or the CLI's `--use-previous-template`: those reuse the already-processed template, and a change to the log group lists is then reported as "No updates are to be performed" and ignored. A change takes effect within a couple of minutes of the stack update finishing. Logs already sent stay in Dstl8. To include a group again, remove it from the list. The whole list can be up to 4,096 characters, CloudFormation's limit for a stack parameter: roughly 100 log groups with typical names. #### Send only chosen log groups Choose this when the account already has a subscription filter account policy, or when you only want some log groups. Each chosen log group gets a subscription filter named after the stack. **Which log groups there are.** Dstl8 has no access to your account, so it can't list them for you. In the launch step, copy the command shown and run it in your terminal or in AWS CloudShell, where the AWS CLI is already installed: ```bash aws logs describe-log-groups --region us-east-1 --output json ``` Paste the output into the launch step. Dstl8 turns it into a checklist with each group's stored size, with **Select all** and a search box. You can also type names directly, one per line, or use **Add custom** for a log group that isn't in the list yet. Infrequent Access log groups are shown but can't be selected. **Limits.** The list is one stack parameter, so it can be up to 4,096 characters (roughly 100 log groups). CloudWatch allows five subscription filters per log group; a group that already has five makes the stack roll back, so remove it from the list. Two names that differ only in punctuation (for example `/aws/lambda/foo-bar` and `/aws/lambda/foobar`) can't be in the same stack; put one of them in a second source. **Changing the list later.** Update the stack with the template URL (**Update** → **Replace existing template** → **Amazon S3 URL**; see [Where to set exclusions](#where-to-set-exclusions) for why not **Use existing template**) and set **LogGroups**, one comma-separated value. With the AWS CLI, set the new list and keep every other parameter (commas inside the value are escaped with `\`): ```bash aws cloudformation update-stack --region us-east-1 --stack-name \ --template-url https://dstl8-public-templates.s3.amazonaws.com/cloudwatch-push/v2/dstl8-cloudwatch-push.yaml \ --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \ --parameters 'ParameterKey=LogGroups,ParameterValue=/aws/lambda/checkout\,/ecs/orders' \ ParameterKey=Mode,UsePreviousValue=true ParameterKey=ExcludeLogGroups,UsePreviousValue=true \ ParameterKey=EndpointUrl,UsePreviousValue=true ParameterKey=AccessKey,UsePreviousValue=true \ ParameterKey=SourceId,UsePreviousValue=true ParameterKey=BufferSizeMiB,UsePreviousValue=true \ ParameterKey=BufferIntervalSec,UsePreviousValue=true ParameterKey=RetrySeconds,UsePreviousValue=true \ ParameterKey=BackupRetentionDays,UsePreviousValue=true ``` Switching an existing stack between the two modes is the same kind of update: set **Mode** to the other value and fill in the list that mode uses, and the stack swaps the account policy for per-group filters, or back. **With the dstl8 CLI**, pass exact names or prefixes: ```bash dstl8 sources add cloudwatch --regions us-east-1 --log-groups '/aws/lambda/*,/ecs/*,/custom/billing-worker' --deploy ``` A name ending in `*` is a prefix; the CLI expands it against the region's log groups when it creates the stack, so prefixes need `--deploy`. In the wizard, or with `--select-log-groups`, the CLI lists the region's log groups and lets you pick, if the AWS CLI is available. Log groups that can't be subscribed (not Standard class, or already at five subscription filters) are left out with a note. #### Stack settings | Parameter | Default | Allowed | What it does | | -------------------------------------------- | ------------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | **EndpointUrl**, **AccessKey**, **SourceId** | filled in by Dstl8 | | Where and as which source Firehose delivers. Don't change these, except to fix a wrong key (see [If logs stop arriving](#if-logs-stop-arriving)). | | **Mode** | AllLogGroups | AllLogGroups, SelectedLogGroups | Every log group except the excluded ones, or only the listed ones (above). | | **ExcludeLogGroups** | none | exact names, comma separated | Log groups to leave out (all log groups mode). | | **LogGroups** | none | exact names, comma separated | Log groups to send (chosen log groups mode). | | **BufferSizeMiB** | 4 | 1–8 | Firehose sends a batch when it reaches this size… Raising it makes larger requests, which Dstl8 can refuse as too large; keep the default unless asked. | | **BufferIntervalSec** | 30 | 0–60 | …or after this many seconds, whichever comes first. Lower means fresher logs and more requests. | | **RetrySeconds** | 180 | 0–180 | How long Firehose retries a batch Dstl8 doesn't accept before writing it to the backup bucket. | | **BackupRetentionDays** | 14 | 1 or more | How long undelivered batches stay in the backup bucket. | The buffer and retry limits keep Firehose's own buffering and retries well inside the 5-minute window Dstl8 counts late logs in; delays inside CloudWatch Logs, before Firehose, can still occasionally push a batch past it (see [Late and undelivered batches](#late-and-undelivered-batches)). #### More than one region or account * **Regions:** launch the stack once per region, all against the same source (pick each region in the launch step, or pass several to `--regions`). Each region's streams carry that region. * **Accounts:** add one Dstl8 source per AWS account. With AWS Organizations you can deploy the same template to many accounts with CloudFormation StackSets, using each account's own source endpoint, key and ID. * A region can have only one of these stacks in all-log-groups mode (one subscription policy per account and region). Stacks in chosen-log-groups mode can coexist with it and with each other. ### Change or remove the setup To change exclusions, the chosen log groups or settings, update the stack as above (with the template URL). To stop sending a region, delete that region's stack. **If you delete the source in Dstl8, also delete its stack in every region you set up**: Dstl8 can't remove it, and a stack left running keeps sending logs, which Firehose bills for, that are no longer received. Deleting a stack removes everything it created except the backup bucket, which is kept in case it holds undelivered batches; delete it yourself once you no longer need them. ### If logs stop arriving The source page shows **No data for 15 minutes** when a source that was receiving logs goes quiet. Check that: * the stack still exists in each region and its status is `CREATE_COMPLETE` or `UPDATE_COMPLETE`; * the Firehose stream is **Active**, and its **Destination error logs** (in the stack's `/aws/kinesisfirehose/` log group) don't show delivery errors; * the account is actually logging. A quiet account is not an error. If the source shows deliveries **rejected for a wrong or missing access key**, the stack is using a different source's key. Update the stack (**Update** → **Replace existing template** with the template URL) and set **AccessKey** to the key shown in this source's **Connection Details** → **Set up the stream yourself**. If deliveries are **rejected as too large**, lower the stack's **BufferSizeMiB** (Update → Replace existing template with the template URL); the default of 4 is safe. If deliveries are **rejected as unreadable**, the Firehose stream is changing the records before sending them. Turn off any record transformation or decompression on the stream; the stack doesn't enable either. ### Late and undelivered batches Every log keeps the time it was written, not the time it reached Dstl8, and Dstl8 counts batches that arrive within 5 minutes of being written. Normally logs arrive well inside that: the stack has Firehose send at least every 30 seconds and retry a batch Dstl8 can't accept for up to 3 minutes (**BufferIntervalSec** and **RetrySeconds**). Delays before Firehose, inside CloudWatch Logs, aren't under the stack's control, so a batch can occasionally arrive later than that and not be counted. A batch that still can't be delivered after the retry window is written to the backup bucket shown in the stack's outputs, where it's kept for **BackupRetentionDays** (14 days by default). To replay those batches, contact us with the bucket name and time range. ### FAQ **What does AWS charge for?** The stack uses Amazon Data Firehose, which AWS bills per GB of data ingested, and S3 storage for batches that couldn't be delivered (normally none). Leaving noisy log groups out reduces both. See [Amazon Data Firehose pricing](https://aws.amazon.com/firehose/pricing/). **Does Dstl8 read anything from my account?** No. The stack only pushes log events out to your source's endpoint. Dstl8 holds no AWS credentials for your account and there is no role Dstl8 can assume. The one secret in your account is your source's access key, which the Firehose stream sends to Dstl8 so the endpoint accepts its deliveries; it grants nothing in AWS. **Can I set it up without CloudFormation, for example with Terraform?** Yes: create an Amazon Data Firehose stream with an HTTP endpoint destination yourself with the endpoint URL and access key from **Connection Details** → **Set up the stream yourself**, and subscribe your log groups to it. ### Converting from a polling source CloudWatch sources created before streaming polled CloudWatch with an IAM access key. Polling is limited by CloudWatch's read quota and can miss logs from busy log groups. To switch: 1. Add a new AWS CloudWatch source and launch its stack, as above. 2. Once the new source shows **Receiving logs**, assign its streams to the same workspaces as the old source. 3. Disable the old source, then delete it. Running both sends the same logs twice. 4. If the old source used an IAM user or access key created only for Dstl8, delete it. Check that nothing else uses it first. History from the old source stays searchable until it ages out. Lambda, ECS and the other recognized log groups get the same stream names as before, so dashboards and incidents read the same; log groups the old source grouped as `unknown` now get their own streams.
Legacy polling sources: IAM permissions Polling sources created before streaming read your logs with two read-only CloudWatch Logs actions, using the access key they were set up with: | Action | Why the source needs it | | ------------------------ | ------------------------------------------------------------------------------------------ | | `logs:DescribeLogGroups` | Finds log groups when **Auto Discover** is on, and verifies access on **Test Connection**. | | `logs:FilterLogEvents` | Reads log events from each log group. Without it, no logs are collected. | The AWS-managed `CloudWatchLogsReadOnlyAccess` policy includes both. A minimal custom policy allows `logs:DescribeLogGroups` on `"Resource": "*"` (it can't be limited to log groups) and `logs:FilterLogEvents` on `arn:aws:logs:::log-group:*` or on specific log groups. {% hint style="warning" %} **Test Connection** only checks that the source can list log groups. A policy without `logs:FilterLogEvents` passes the test, but the source never receives logs and AWS returns `AccessDeniedException` for every read. If a polling source stays **Pending**, check that this action is allowed, or convert it to streaming as above. {% endhint %}

--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.controltheory.com/controltheory-documentation/dstl8-docs/sources/aws-cloudwatch.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.