Domain 3 of 4 · Chapter 3 of 4

Pipeline Monitoring and Maintenance

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AWS Certified Data Engineer - Associate premium course — no separate purchase.

Included in this chapter:

  • The observe, alert, investigate loop
  • Metrics per service: what to watch
  • Alarms and SNS: turning a metric into a page
  • Reading logs: CloudWatch Logs Insights, Athena, and OpenSearch
  • Troubleshooting Glue and EMR performance
  • Exam-pattern recognition for monitoring questions

Choosing a log-analysis service for pipeline logs

Decision factorCloudWatch Logs InsightsAthena (logs in S3)OpenSearch Service
Where the logs already liveAmazon CloudWatch LogsAmazon S3 (delivered logs)Indexed into an OpenSearch domain/collection
Query interfacePurpose-built Logs Insights query languageStandard SQL (Trino/Presto engine)Full-text search + OpenSearch Dashboards
Best forAd-hoc troubleshooting of recent logsCheap SQL over large historical archivesSearch and dashboards at high volume
Setup costNone: query in placeDefine table/partitions over S3Stand up and size a domain, ingest logs
Pricing basisPer GB of log data scanned$5 per TB scanned (partition to scan less)Provisioned instance/storage (or serverless OCUs)

Decision tree

Logs already in CloudWatch?and an ad-hoc query is enoughYesCloudWatch Logs Insightsquery in place, no setupNoLogs delivered to S3?SQL over a large archiveYesAthena (SQL over S3)cheap; partition to scan lessNoHigh-volume + search?full-text + dashboardsYesOpenSearch Serviceindex logs, search, dashboardsAlways: alarm on the metric first, then pivot to logs.
Cheat sheet

Cheat sheet

  • Watch a pipeline through observe, alert, investigate
  • Alarm on a metric, then pivot to logs for the cause
  • Glue reports job metrics to CloudWatch every 30 seconds
  • EMR pushes cluster metrics to CloudWatch every 5 minutes
  • Alarm on numFailedTasks to catch Glue job failures
  • Rising Glue JVM heap usage means memory pressure
  • maxNeededExecutors above allExecutors means under-provisioning
  • Use the Spark UI to find Glue stragglers and skew
  • EMR IsIdle flags a cluster that is alive but doing nothing
  • ContainerPending with low YARN memory means a starved cluster
  • A rising Kinesis iterator age means consumers are falling behind
  • A CloudWatch alarm has three states: OK, ALARM, INSUFFICIENT_DATA
  • An alarm acts only when it changes state
  • Set missing data to notBreaching for a job idle between runs
  • Route a CloudWatch alarm to an SNS topic to notify the team
  • Use a composite alarm to cut noise; metric math to alarm on a rate
  • Logs live in groups and streams within CloudWatch Logs
  • CloudWatch Logs Insights queries CloudWatch logs in place
  • Logs Insights: 100 concurrent queries, 60-minute timeout, 7-day results
  • Choose the log-analysis tool by where logs live and how you query
  • Athena bills $5 per TB scanned, so partition the log table
  • Diagnose a slow distributed job before adding workers
  • Send EMR step and container logs to S3 to debug after termination
  • CloudTrail is a monitoring signal here, audit recording is separate
  • CloudWatch anomaly detection learns a metric's seasonal pattern so you skip static thresholds
  • Enable X-Ray on a state machine to see per-service latency in the service map

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Using Amazon CloudWatch metrics
  2. Monitoring AWS Glue using Amazon CloudWatch metrics
  3. Monitor Amazon EMR metrics with CloudWatch
  4. What is Amazon CloudWatch Logs?
  5. Using Amazon CloudWatch alarms
  6. What is Amazon SNS?
  7. Analyzing log data with CloudWatch Logs Insights
  8. What is Amazon Athena?
  9. What is Amazon OpenSearch Service?
  10. Enabling the Apache Spark web UI for AWS Glue jobs