Data Operations and Support

A pipeline that loads beautifully in a demo is worth nothing if it needs someone awake at 3 a.m. to start it, notice it broke, or catch that it loaded garbage. This domain (22% of the exam) is about the day-after: making the work start itself, surfacing trouble before a stakeholder does, and proving the data is fit to use. The mental model that ties every question together is a four-stage lifecycle, automate -> analyze -> monitor -> assure quality, and most exam scenarios are really asking which stage you are in. The classic trap is reaching for a heavyweight service when a lighter one owns the stage: standing up Amazon EMR to schedule a tiny job that EventBridge Scheduler should trigger, or building a custom dashboard when an Amazon CloudWatch alarm already answers "is it healthy?". Name the stage first and the right service usually follows.

Walk the four subtopics in the order the data flows. Data Processing Automation replaces "a person runs the job" with a trigger: an event such as an object landing in Amazon S3, or a clock such as an EventBridge Scheduler cron, fires AWS Lambda, an AWS Glue job, or an AWS Step Functions execution, and the AWS SDK (boto3 in Python) drives any of it from code. Data Analysis is how you query the result: Amazon Athena for serverless SQL straight over S3, Amazon Redshift when heavy joins repeat, and Amazon QuickSight (now offered as Amazon Quick Sight) for the dashboards on top. Pipeline Monitoring and Maintenance is the watch loop, observe with CloudWatch metrics, alert with CloudWatch alarms into an Amazon SNS topic, and investigate in CloudWatch Logs and each service's own diagnostics. Data Quality closes the loop by proving the data is correct, with AWS Glue Data Quality rules written in DQDL (Data Quality Definition Language) and AWS Glue DataBrew profiling for the no-code path.

Across all four subtopics the guiding instinct is the same: prefer the service that removes operational toil and let the telemetry, not a guess, name the fix. Diagnose a slow Glue job from its job metrics and the Apache Spark UI before adding workers; alarm on a pre-aggregated metric and only then open the logs, because metrics are cheap and log scans are billed per gigabyte; clean data with no-code DataBrew or plain SQL before writing custom code. The same instinct picks the trigger and the engine: EventBridge Scheduler over a self-managed cron host, Lambda for the short glue step but a Glue or EMR job the moment work would exceed Lambda's 15-minute and 10 GB limits.