Data Ingestion and Transformation

Picture a single record's journey: it lands from some source, gets reshaped into the format and layout downstream queries want, and every step of that journey is started, retried, and watched by something that is not itself touching the data. Those four jobs are exactly the four subtopics here, and they are the largest slice of the exam, at 34% the heaviest of the four domains. The mental model that makes most questions easy is to name the job before naming the service: is the scenario about getting data in (ingestion), changing its shape (transformation), deciding what runs when (orchestration), or the code and limits that carry all three (programming concepts)? The classic trap hands you a service that can technically do two jobs (Amazon Data Firehose can land and lightly convert; AWS Lambda can ingest, transform, or glue steps together) and rewards you for picking the service whose primary job matches the one the scenario is really asking about.

Read this page as a map, then follow the four subtopics in order. Data Ingestion is the front door: it sorts the streaming-versus-batch decision and matches each source to the right connector, from Amazon Kinesis Data Streams and Amazon MSK for low-latency streams to AWS Database Migration Service, AWS DataSync, and AWS Transfer Family for batch and file sources. Data Transformation is where the record changes shape, choosing among AWS Glue, Amazon EMR, AWS Lambda, and in-warehouse Amazon Redshift, and baking in the columnar, partitioned, compressed output that cuts every downstream cost. Pipeline Orchestration is the conductor that sequences those steps, handles failure, and alerts on-call, led by AWS Step Functions with Amazon MWAA (Managed Workflows for Apache Airflow), Glue workflows, and Amazon EventBridge as alternatives. Programming Concepts is the cross-cutting layer underneath all three: Lambda concurrency and limits, infrastructure as code, CI/CD, and the distributed-computing ideas that make a Spark job scale.

Across all four steps AWS rewards the same instinct: pick the option that meets the requirement with the least operational weight. That usually means a serverless or fully managed service over a cluster you size yourself, an event or schedule trigger over a polling loop, and a built-in retry or catch over a hand-written retry loop. You rarely need to memorise an exception. A managed delivery pipe beats a self-run consumer when both would land the data, Glue's serverless Spark beats a long-lived cluster for short bursty jobs, and Step Functions' built-in error handling beats orchestration code you maintain. Each subtopic shows where the genuine exceptions live, such as Amazon EMR when you need full framework control or deep Spot savings.