Home › AWS › AWS data services › CloudWatch
AWS data services · Observability, audit and cost · about 10 min read
Amazon CloudWatch (metrics, logs, alarms) and AWS X-Ray
AWS's built-in metrics, logs and alarms for every service, plus X-Ray tracing; here it watches the AWS movers.
In plain words
What it is
Every AWS service reports numbers about itself every minute or so: how many records came in, how many failed, how far behind a reader is. Those numbers are metrics, and CloudWatch is where AWS stores them. CloudWatch also stores logs, the lines of text that programs write, and runs alarms, which watch a metric and act when it crosses a line for long enough. You do not install anything to get the standard metrics. They appear as soon as you use the service.
AWS X-Ray adds tracing. A trace follows one request as it passes through several services, such as an API call that goes through API Gateway, a Lambda function and DynamoDB, and shows where the time went. X-Ray is now fed through OpenTelemetry, an open standard for metrics, logs and traces. The older X-Ray SDKs and daemon are in maintenance and reach end of support in February 2027.
A concrete example: a retail company copies its order database to S3 with AWS DMS, pulls SAP sales orders with AppFlow, and streams store-sensor readings through Kinesis. At 02:10 the DMS task's CDCLatencySource metric climbs past 15 minutes because the source database is busy with a month-end job. A CloudWatch alarm has watched that metric for three periods of five minutes. It changes to ALARM, sends a message to an SNS topic, and the on-call engineer's phone rings. The same state change also lands in S3, so the morning operations dashboard shows the incident next to the Databricks job that waited for the data.
CloudWatch has grown many parts over the years: dashboards, Logs Insights for querying logs, anomaly detection, synthetic canaries, Application Signals and, since September 2026, CloudWatch Omni. A data platform needs only a few of them. This page focuses on those few.
The instrument panel and warning lights of every AWS machine you rent, wired to a phone that rings when a light stays red.
Big picture
Where it fits
The lit lane is where CloudWatch works. Boxes on the left feed it; boxes on the right use what it produces.
CloudWatch sits in the band across the bottom of the picture, with audit and cost. On the reference platform, Databricks watches what happens inside Databricks through system tables, job notifications and SQL alerts. CloudWatch watches what happens before data reaches Databricks: the AWS movers. That means DMS replication lag and errors, AppFlow flow failures, Transfer Family uploads, IoT SiteWise ingestion, Kinesis reader lag and write throttling, Firehose delivery freshness, and in the fast lane Managed Flink checkpoints and DynamoDB throttles.
Two paths leave CloudWatch. The first is the alarm path. Alarms on mover metrics publish to Amazon SNS, which pages on-call and posts to the same Teams or Slack channel as Databricks alerts. The second is the evidence path. A CloudWatch metric stream sends the mover metrics through Data Firehose to S3 about every five minutes, and Auto Loader reads them into the prod_ops schema. Alarm state changes go through EventBridge to the same S3 path. The table ops.fct_mover_health holds one row per mover per five minutes.
This split matters. Engineers get paged by CloudWatch in seconds, but the morning question, 'why is the sales dashboard late?', is answered in one place. The run log carries the dataset id, the config maps the dataset to its mover id, and the mover id joins the CloudWatch metrics. One query shows that the gold job waited because DMS was 40 minutes behind.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Metric | A named number over time, such as CDCLatencySource for one DMS task, identified by a namespace and dimensions. |
| Dimension | A name and value that picks out one thing a metric is about, such as the task id or stream name. |
| Alarm | A rule that watches one metric or a metric expression and changes state to ALARM when it breaches a threshold for a set number of periods. |
| Log group | A container for log lines from one source, with its own retention period and optional KMS key. |
| Logs Insights | A query language for searching and summarising log groups, billed by the amount of log data scanned. |
| Metric stream | A continuous export of chosen metrics to Data Firehose, which delivers them to S3 or a partner tool within minutes. |
| Cross-account observability | A setup that lets one monitoring account see metrics, logs and traces from many source accounts. |
| Trace | The path of one request through several services, recorded by X-Ray from OpenTelemetry instrumentation. |
| Anomaly detection | A model trained on a metric's history that draws an expected band, so alarms fire on unusual values rather than a fixed line. |
Choosing
When to use it, and when not
Use it when
- You run AWS services such as DMS, AppFlow, Kinesis, Firehose, Flink or DynamoDB and need to know within minutes when they fall behind or fail.
- You need to page someone when an AWS-side mover breaks, through SNS to a pager, chat channel or ticket.
- You want AWS mover health in the same tables as Databricks job runs, quality results and cost, through a metric stream to S3.
- You run Lambda functions, containers or APIs whose logs need a home with retention and search.
- You need traces across API Gateway, Lambda and DynamoDB in the fast lane, using OpenTelemetry.
Avoid it when
- The signal is inside Databricks, such as job runs, pipeline events or query history; system tables already hold it.
- You want to keep Databricks cluster logs and Spark logs long term in CloudWatch; deliver them to S3 and read them as tables instead.
- You want business metrics, such as orders per hour, for business users; build those in gold and show them in BI.
- You would ship every debug log line from every service; ingestion cost grows faster than the value.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Missing data hides an outage | A mover stops sending metrics, the alarm sees no data, treats it as fine and stays green. | Set TreatMissingData to breaching for heartbeat-style alarms, and add a 'silent stream' check per plant that alarms when no records arrive. |
| Alarm noise | Hundreds of alarms with tight thresholds fire during every month-end load, and on-call starts ignoring them. | Alarm on symptoms that matter (lag past the freshness promise, failures), use several evaluation periods, and use composite alarms to page once per incident. |
| Log cost runaway | A chatty Lambda writes debug lines at high volume, and log ingestion becomes the biggest line on the observability bill. | Set log levels per environment, set a retention period on every log group, and use the Infrequent Access log class for logs you rarely query. |
| Logs kept forever | Log groups default to never expire, so old logs with personal data stay for years. | Set retention on every log group with infrastructure code, and alarm on any log group without retention. |
| Metrics spread over many accounts | Plant, data and shared-services accounts each have their own metrics, and no one sees the whole picture. | Use cross-account observability into one monitoring account, and run the metric stream from there. |
| X-Ray SDK end of support | Services instrumented with the X-Ray SDK will get only security fixes and reach end of support in February 2027. | Move instrumentation to OpenTelemetry through the AWS Distro for OpenTelemetry, and test sampling rules after the move. |
| Metric stream gaps | The Firehose delivery behind the metric stream fails silently, and ops.fct_mover_health stops updating. | Alarm on Firehose DeliveryToS3.DataFreshness, and add a freshness check on the prod_ops table itself. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick CloudWatch when |
|---|---|---|
| Databricks system tables and SQL alerts | The signal is inside Databricks: jobs, pipelines, queries, billing and audit. | The signal comes from an AWS service such as DMS or Kinesis. |
| Amazon Managed Grafana | Engineers want live multi-source boards over CloudWatch and Prometheus. | You need the alarm and the stored metrics; Grafana only reads them. |
| Azure Monitor | You are on Azure, with Event Hubs and Data Factory as movers. | You are on AWS. |
| Google Cloud Monitoring and Logging | You are on Google Cloud. | You are on AWS. |
| Prometheus, Grafana and Loki, or a vendor such as Datadog | You run Kubernetes-heavy workloads or want one tool across clouds. | You need native AWS service metrics without another agent or contract. |
Cost
How you pay
Basic metrics from AWS services are free. You pay for custom metrics per metric per month, for API calls such as GetMetricData, for alarms per alarm metric per month, for dashboards beyond a free number, for log ingestion and storage per GB, for Logs Insights per GB scanned, and for metric streams per thousand metric updates. X-Ray and Omni have their own charges per trace or per data volume. Check the CloudWatch pricing page for your region; prices differ by log class and tier.
What drives the bill
- Log ingestion volume, usually the largest line.
- Log storage when retention is long or never set.
- Number of custom metrics, which multiplies with every dimension value.
- Metric stream updates, which grow with the number of metrics streamed.
- Logs Insights and GetMetricData queries from dashboards and tools.
Cost traps
- A custom metric with a high-cardinality dimension, such as order id or sensor id, creates thousands of billed metrics.
- Streaming every namespace instead of the few mover metrics you join in prod_ops.
- Log groups with no retention keep growing every month.
- Grafana or other tools refreshing CloudWatch panels every few seconds drive API cost.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Give each team read access to the metrics and logs of its own accounts, and keep the central monitoring account read-only for most people.
- Encrypt sensitive log groups with a customer managed KMS key, and set retention on every log group.
- Use a VPC interface endpoint for CloudWatch and CloudWatch Logs so private subnets send telemetry without a public route.
- Never log personal data or secrets; use CloudWatch Logs data protection policies to mask patterns that slip through, and review the findings.
- CloudTrail records changes to alarms, log groups and metric streams; alarm on deletion of critical alarms.
- SNS topics used for paging should have access policies that allow only CloudWatch and EventBridge to publish.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| CDCLatencySource and CDCLatencyTarget (DMS) | How far DMS is behind the source and the target; compare with each dataset's freshness promise. |
| FlowExecutionsFailed (AppFlow) | Failed AppFlow runs, for example an SAP OData pull that timed out. |
| GetRecords.IteratorAgeMilliseconds (Kinesis) | How far behind readers of the stream are. |
| WriteProvisionedThroughputExceeded (Kinesis) | Writes throttled by a hot or undersized stream. |
| DeliveryToS3.DataFreshness (Firehose) | Age of the oldest record not yet in S3, including the metric stream itself. |
| lastCheckpointDuration (Managed Flink) | How long fast-lane checkpoints take; growing values signal back-pressure. |
| Alarm state changes | Every change to ALARM or OK, landed in S3 through EventBridge for the incident list. |
| Limit | Value and notes |
|---|---|
| Metric retention | 1-minute data kept 15 days, 5-minute data 63 days, 1-hour data 455 days |
| Log event size | 1,024 KB per log event; 1 MB per PutLogEvents batch |
| Subscription filters | 5 per log group |
| Metric filters | 100 per log group |
| PutMetricData rate | 500 requests per second per Region by default (adjustable) |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| DMS lag past the 15-minute freshness promise for orders | CloudWatch alarm on CDCLatencySource over three periods, SNS to on-call, state change to S3 | Pursue: the mover's own metric is the earliest and most reliable signal. |
| One morning view of what failed and why | Metric stream through Firehose to S3, Auto Loader into prod_ops, joined with job runs in Databricks AI/BI | Pursue: one place to answer, with evidence from both sides. |
| Spark driver logs from Databricks clusters | Cluster log delivery to S3, read as tables; not CloudWatch | Drop CloudWatch here: cheaper and closer to the data. |
| Lambda and API Gateway in the fast lane with X-Ray SDK | Move instrumentation to OpenTelemetry (ADOT) before February 2027 | Pursue with care: the SDK is in maintenance; plan the move now. |
FAQ
Questions people ask
Is AWS X-Ray being retired?
No. The X-Ray service continues. The X-Ray SDKs and daemon entered maintenance on 25 February 2026 and reach end of support on 25 February 2027, and AWS recommends instrumenting with OpenTelemetry through the AWS Distro for OpenTelemetry.
How do I get CloudWatch metrics into Databricks?
Create a CloudWatch metric stream for the namespaces you need, deliver it through Data Firehose to S3, and read the files with Auto Loader into a table. Latency is a few minutes, which suits operations views and joins.
Should Databricks logs go to CloudWatch?
Usually not on this platform. Databricks system tables hold job, pipeline, query and audit data, and cluster logs can be delivered to S3. CloudWatch is kept for AWS services that Databricks cannot see.
What is CloudWatch Omni?
An AI-first observability experience launched in September 2026 for applications and AI agents, built on open standards such as OpenTelemetry. It is useful for agent workloads; the core metrics, logs and alarms on this page work without it.
Why did my alarm not fire when the stream stopped?
By default an alarm with no data points may treat missing data as not breaching. For heartbeat signals set TreatMissingData to breaching, or alarm on a metric that counts records and expects more than zero.
Sources
Checked against
- AWS docs: What is Amazon CloudWatch
- AWS docs: CloudWatch service quotas
- AWS docs: CloudWatch Logs quotas
- AWS docs: X-Ray SDK and daemon end of support timeline
- AWS blog: Introducing Amazon CloudWatch Omni
Checked 2026-11-11.
Related
Where it connects
AWS and Databricks reference platform · Streaming on AWS and Databricks · AWS CloudTrail · Amazon SNS · Amazon Managed Grafana · All AWS data services