Data Transformation
Four engines that reshape data, and how to pick one
A pipeline reads raw data, reshapes it (joins, aggregations, cleansing, format changes), and writes the result. On AWS, four services do that reshaping, and a DEA-C01 question almost always hands you a workload and asks which one to reach for. The fastest way to decide is to ask two things in order: does the job need Apache Spark (a distributed engine for large, parallel transforms), and how much of the cluster do you want to operate.
The four engines
AWS Glue[1] runs Apache Spark as a serverless ETL (extract, transform, load) service: you write a Spark script (or build it visually) and Glue provisions, runs, and tears down the workers for you, billing per second of compute. It integrates tightly with the Glue Data Catalog, so a job can read a cataloged table and write a new one without you wiring up the schema by hand. Glue is the default answer for scheduled Spark ETL where you do not want to run a cluster.
Amazon EMR[2] runs the full open-source big-data stack, Apache Spark, Hive, Presto/Trino, and more, on clusters. You can run EMR on EC2 (you size the cluster and control bootstrap actions, custom libraries, and instance types) or on EMR Serverless[3], which auto-sizes Spark or Hive workers and releases them when the job finishes. Reach for EMR when you need a specific framework or version, heavy customization, or the deepest cost cut from EC2 Spot capacity on long-running jobs.
AWS Lambda[4] is the lightweight option: a function triggered by an event (an S3 object created, a stream record) that reshapes a small payload with no Spark cluster at all. It is bounded by a 15-minute maximum runtime and 10 GB of memory[5], so it suits per-file or per-record transforms, not multi-gigabyte joins.
Amazon Redshift transforms data that already lives in the warehouse using SQL, the ELT (extract, load, transform) pattern. Stored procedures and CREATE TABLE AS SELECT reshape data in place, which avoids pulling it back out to an external engine. If the data is not yet in Redshift, this is not your transform engine.
Reading the decision
The diagram below walks the same logic in order: no Spark and a tiny payload points to Lambda; data already in the warehouse points to Redshift ELT; otherwise the choice between Glue and EMR is whether you want a serverless Spark job or a cluster you control.
AWS Glue mechanics: DPUs, workers, bookmarks, Studio
Glue's pricing and scaling both revolve around one unit, so internalize it first. A Data Processing Unit (DPU) is a fixed bundle of 4 vCPUs and 16 GB of memory, and a Glue Spark job's cost is just DPUs multiplied by run time. You do not pick DPUs directly; you pick a worker type and a worker count, and each worker maps to a DPU multiple.
Worker types
| Worker type | DPU per worker | Recommended for |
|---|---|---|
| G.1X | 1 DPU (4 vCPU, 16 GB) | Most transforms, joins, queries (the default choice) |
| G.2X | 2 DPU (8 vCPU, 32 GB) | Same workloads, larger per-worker memory |
| G.4X | 4 DPU (16 vCPU, 64 GB) | The most demanding transforms and aggregations |
| G.8X | 8 DPU (32 vCPU, 128 GB) | The heaviest jobs needing maximum per-worker compute |
| G.025X | 0.25 DPU (2 vCPU, 4 GB) | Low-volume streaming jobs only |
Glue bills per second with a 1-minute minimum[6], requires a minimum of 2 DPUs, and allocates 10 DPUs to an ETL job by default. That per-second model, with nothing to pay between runs, is exactly why Glue beats a standing EMR cluster for short, scheduled jobs: an idle EMR cluster bills around the clock, whereas a Glue job bills only while it runs. The trade-off is Spark startup latency on each Glue run, so a stream of tiny jobs may waste more on cold starts than the compute itself.
Job bookmarks for incremental loads
The feature DEA-C01 tests most on Glue is job bookmarks[7]. A bookmark is persisted state that records what a job has already processed, so a scheduled rerun handles only new data instead of reprocessing the whole dataset. The bookmark has three settings: Enable (track state and process only new data each run), Disable (the default, always reprocess everything), and Pause (process new data without advancing the bookmark). For Amazon S3 sources, Glue decides what is new by the object's last-modified timestamp; for JDBC sources, it uses a bookmark key column, by default the primary key when it increases or decreases sequentially with no gaps. To reprocess everything deliberately (a backfill), you reset the bookmark with the ResetJobBookmark API, the console, or aws glue reset-job-bookmark.
Authoring: Glue Studio
You do not have to hand-write Spark. AWS Glue Studio[8] is a visual editor: you lay out source, transform, and target nodes on a canvas and Studio generates the Apache Spark code for you, which lowers the bar for building ETL while still producing a real Spark job underneath.
File-format optimization: CSV to Parquet, partition, compress
The cheapest query is the one that reads the fewest bytes, and the transform job is where you decide how many bytes every downstream query will read. The rule: write columnar, partitioned, compressed data. Each of the three works on a different axis, and they stack.
Columnar format (the biggest lever)
CSV and JSON store data row by row, so an engine that needs three columns out of fifty still reads all fifty. Apache Parquet and ORC store data column by column[9], which unlocks two savings at once: column pruning (read only the columns a query references) and predicate pushdown (each column chunk carries min/max statistics, so the engine skips entire chunks or files whose range cannot match the filter). Converting CSV to Parquet is the canonical DEA-C01 transform task, and the right tool is a Spark job (Glue or EMR), Athena CREATE TABLE AS SELECT, or Amazon Data Firehose record-format conversion for streams.
Partitioning (prune whole prefixes)
Partitioning writes data into S3 prefixes keyed by a column, for example s3://bucket/sales/year=2026/month=06/. A query filtered to June 2026 then lists only that prefix and never touches other months, which is partition pruning. Choose partition keys that match how queries filter (commonly date), and avoid over-partitioning into millions of tiny partitions, which trades scan cost for crippling metadata and listing overhead.
Compression and file sizing
Compress the columnar output with a splittable codec such as Snappy (fast, the common default), GZIP (smaller, slower), or ZSTD. Compression cuts both storage and bytes scanned. The companion habit is avoiding the small-files problem: thousands of tiny objects force an engine to open thousands of files, so jobs should coalesce output into reasonably sized files (a few hundred MB is a common target) rather than emitting one file per record.
Why it pays off
These choices matter most for scan-priced engines: Amazon Athena charges by the amount of data scanned, so columnar plus partitioning plus compression can cut a query's bill by an order of magnitude versus raw CSV. The transform job is the one place to bake all three in, because rewriting the lake later is far more expensive than writing it right the first time.
Connectivity, LLM enrichment, and cost knobs
Two blueprint items round out the transform step: pulling from many sources and, increasingly, calling a model mid-transform.
JDBC/ODBC and integrating multiple sources
Real pipelines combine data from several systems. Glue and EMR Spark read relational sources, Amazon RDS, Aurora, Amazon Redshift, and external on-prem databases, over JDBC connections[10]; BI and client tools more often connect over ODBC. A Glue connection stores the JDBC URL plus credentials (kept in AWS Secrets Manager) and the VPC networking a job needs to reach a private database, so one job can join a JDBC table against S3 data and write a single integrated dataset. The exam frames this as defining the volume, velocity, and variety of data: high volume and velocity push you toward Spark engines (Glue/EMR), while variety, mixing structured tables with semi-structured JSON or unstructured text, is what makes a flexible transform engine necessary rather than plain SQL.
LLM integration via Amazon Bedrock
A newer pattern is calling a large language model from inside a transform job to enrich records: classify support tickets, extract fields from free text, or summarize documents. You do this by invoking Amazon Bedrock[11] from your Glue or EMR Spark code (Bedrock is serverless and exposes a single inference API across foundation models), processing the text column and writing the model output as a new column. The transform engine still does the heavy data movement; Bedrock supplies the per-record intelligence.
Cost optimization while processing
The cost rules are consistent across engines. For long EMR jobs, run task nodes on EC2 Spot Instances[12] for up to ~90% off On-Demand, while keeping core and master nodes On-Demand so a reclaimed Spot node loses only recomputable work, never the cluster. For non-urgent Glue jobs, the Flex execution class[13] runs on spare capacity at lower cost when slower start times are acceptable. And the cheapest optimization is doing less work: the columnar/partition/compress output from the previous section means every downstream job and query scans fewer bytes.
Exam-pattern recognition
When a stem stresses "no servers to manage" plus "Apache Spark" plus "scheduled," the answer is Glue. "Spark/Hive/Presto," "custom configuration," or "lowest cost on a long batch with Spot" points to EMR. "Reduce Athena cost" or "convert CSV" points to writing columnar Parquet with partitioning. "Process only new files each night" is Glue job bookmarks. A sub-15-minute, event-driven, single-file transform is Lambda. And data "already in Redshift" that needs reshaping is SQL ELT, not an external engine.
Choosing a transform engine on AWS
| Decision axis | AWS Glue | Amazon EMR | AWS Lambda | Amazon Redshift |
|---|---|---|---|---|
| Engine / model | Serverless Apache Spark ETL | Spark, Hive, Presto on managed clusters or EMR Serverless | Function runtime, no Spark | SQL in-warehouse (ELT) |
| Cluster to manage | None (serverless) | You size EC2 clusters, or use EMR Serverless | None | Provisioned or Serverless warehouse |
| Billing model | Per-second DPU, 1-min minimum, 10 DPU default | Per-second EC2 + EMR fee; Spot up to ~90% off | Per-request + GB-second | Per cluster-hour or RPU-second |
| Best fit | Scheduled serverless Spark ETL | Framework control, custom libs, deep Spot savings | Small event-driven transforms <15 min | Reshape data already in Redshift |
| Hard limit to watch | Min 2 DPUs; Spark startup latency | Cluster provisioning/idle cost | 15-min runtime, 10 GB memory | Data must already be loaded |
Decision tree
Sharp facts the exam loves — give these one last read before exam day.
Cheat sheet
Sharp facts the exam loves — scan these before test day.
- Reach for AWS Glue when you want serverless Spark ETL on a schedule
AWS Glue runs Apache Spark with no cluster to manage and bills per second, so it fits scheduled extract-transform-load jobs that run and then stop. It integrates with the Glue Data Catalog and offers job bookmarks for incremental loads, which is why it beats a standing EMR cluster for short, bursty jobs that would otherwise leave a cluster idle and billing. Choose EMR instead when you need a specific framework version, custom cluster configuration, or the deepest Spot discount on long batches.
Trap Standing up a long-lived EMR cluster for short nightly jobs; the cluster bills around the clock while idle, whereas Glue bills only while a job runs.
- A Glue DPU is 4 vCPUs and 16 GB, and you pay per second with a 1-minute minimum
Glue measures capacity in Data Processing Units (DPUs), where one DPU is 4 vCPUs and 16 GB of memory, and a job's cost is DPUs times run time. Glue bills per second with a 1-minute minimum, needs at least 2 DPUs, and allocates 10 DPUs to an ETL job by default. The per-second-with-no-idle-cost model is the core reason Glue is cheaper than an always-on cluster for intermittent work.
- Use G.1X as the default Glue worker, scaling up only for the heaviest jobs
Glue worker types map to DPU multiples: G.1X is 1 DPU (4 vCPU, 16 GB) and is AWS's recommended default for most transforms, joins, and queries; G.2X is 2 DPU; and G.4X (4 DPU) and G.8X (8 DPU) are for the most demanding aggregations. The separate G.025X worker is 0.25 DPU and is meant only for low-volume streaming jobs, not batch ETL. Pick a bigger worker for memory-heavy joins, more workers for parallelism.
Trap Using G.025X for a batch ETL job to save money; it is a 0.25-DPU streaming worker and starves a large batch transform of memory.
- Enable Glue job bookmarks to process only new data on each rerun
A Glue job bookmark is persisted state recording what a job already processed, so a scheduled rerun handles only new data instead of reprocessing the whole dataset. For S3 sources Glue decides what is new by the object's last-modified timestamp; for JDBC sources it uses a bookmark key column, by default the primary key when it increases or decreases sequentially with no gaps. The setting has three modes: Enable (track and process new data), Disable (the default, reprocess everything), and Pause (process new data without advancing the bookmark).
Trap Leaving bookmarks at the default Disable and expecting incremental behavior; Disable reprocesses the entire dataset every run, so duplicates land downstream.
- Reset a Glue job bookmark to force a full backfill
When you intend to reprocess all source data with the same job, reset the bookmark using the ResetJobBookmark API, the console, or aws glue reset-job-bookmark. Resetting clears the processed-state so the next run reads everything again, but Glue does not clean target files, since only source files are tracked, so point the rerun at a fresh output location to avoid duplicate data.
- Build Glue jobs visually with Glue Studio when you don't want to hand-write Spark
AWS Glue Studio is a visual editor where you lay out source, transform, and target nodes on a canvas and Studio generates the underlying Apache Spark code. It lowers the bar for authoring ETL while still producing a real Spark job, so the choice between Studio and a hand-written script is about authoring convenience, not a different runtime.
- Pick Amazon EMR when you need framework control or deep Spot savings
Amazon EMR runs the full open-source stack, Apache Spark, Hive, and Presto/Trino, on clusters you size (EMR on EC2) or on EMR Serverless. It wins over Glue when you need a specific framework version, bootstrap actions, custom libraries, or the lowest cost on long batches via EC2 Spot. EMR Serverless removes cluster sizing by auto-scaling Spark or Hive workers and releasing them when the job finishes.
- Run EMR task nodes on Spot for up to ~90% off, keeping core and master On-Demand
EC2 Spot Instances sell spare capacity for up to ~90% off On-Demand but can be reclaimed on short notice, so on EMR you put interruption-tolerant task nodes on Spot while keeping core nodes (which hold HDFS data) and the master node On-Demand. That way a reclaimed Spot node loses only recomputable work, never cluster state or stored data. This is the canonical EMR cost-optimization answer for long, fault-tolerant batch jobs.
Trap Putting EMR core or master nodes on Spot to cut cost; reclaiming a core node can lose HDFS data and reclaiming the master kills the whole cluster.
- Use EMR Serverless to run Spark or Hive without sizing a cluster
EMR Serverless is a deployment option that auto-determines the resources a Spark or Hive job needs, schedules workers, and releases them when the job finishes, so you avoid over- or under-provisioning. For jobs that must start in seconds, pre-initialized capacity keeps a warm pool of workers ready. It is the EMR answer when you want managed open-source frameworks but no cluster operations.
- Use Lambda for small, event-driven transforms under 15 minutes
AWS Lambda reshapes a small payload in response to an event, such as an S3 object created or a stream record, with no Spark cluster. It is bounded by a 15-minute maximum runtime and 10 GB of memory, so it fits per-file or per-record transforms but not multi-gigabyte joins or aggregations, which belong on Glue or EMR.
Trap Choosing Lambda for a large batch join; the 15-minute runtime and 10 GB memory ceilings make it the wrong engine for heavy Spark-scale work.
- Transform data already in Redshift with SQL (ELT), not an external engine
When data already lives in Amazon Redshift, reshape it in place with SQL using stored procedures and CREATE TABLE AS SELECT, the extract-load-transform (ELT) pattern. This avoids pulling data out to Glue or EMR and back. If the data is not yet loaded into Redshift, this is not the transform engine to pick.
Trap Pulling data out of Redshift into a Glue Spark job to reshape it; in-warehouse SQL ELT avoids the round trip when the data is already loaded.
- Convert CSV/JSON to columnar Parquet or ORC to enable column pruning and pushdown
Row-based CSV and JSON force an engine to read every column even when a query needs a few, while columnar Parquet and ORC let it read only referenced columns (column pruning) and skip whole chunks using min/max statistics (predicate pushdown). Converting CSV to Parquet is the canonical DEA-C01 transform task, done with a Glue or EMR Spark job, Athena CREATE TABLE AS SELECT, or Amazon Data Firehose record-format conversion for streams.
Trap Leaving curated data as CSV or JSON to query later; row formats read every column on every scan, so Athena costs and query times stay high.
- Partition output by query-filter columns to prune whole S3 prefixes
Partitioning writes data into S3 prefixes keyed by a column, for example year=2026/month=06/, so a query filtered to that range lists only the matching prefix and skips the rest (partition pruning). Choose partition keys that match how queries actually filter, commonly date, and avoid over-partitioning into millions of tiny partitions, which trades scan savings for crippling metadata and listing overhead.
Trap Partitioning on a high-cardinality column like user ID; it creates millions of tiny partitions whose listing and metadata cost outweighs the pruning benefit.
- Compress columnar output and avoid the small-files problem
Compress Parquet/ORC output with a splittable codec, Snappy (fast, common default), GZIP (smaller, slower), or ZSTD, to cut both storage and bytes scanned. Pair this with coalescing output into reasonably sized files (a few hundred MB is a common target) rather than emitting one file per record, because thousands of tiny objects force an engine to open thousands of files and inflate task overhead.
Trap Writing one output file per input record; the resulting small-files explosion makes every downstream scan open thousands of objects and slows jobs more than the data volume warrants.
- Connect to relational and on-prem sources over JDBC, client tools over ODBC
Glue and EMR Spark read relational sources, RDS, Aurora, Amazon Redshift, and external on-prem databases, over JDBC connections, while BI and client tools more often connect over ODBC. A Glue connection stores the JDBC URL, credentials (kept in AWS Secrets Manager), and the VPC networking a job needs to reach a private database, so one job can join a JDBC table against S3 data into a single integrated dataset.
- Let volume, velocity, and variety drive the engine choice
Defining a workload by volume (how much), velocity (how fast it arrives), and variety (structured, semi-structured, unstructured) points to the right engine. High volume and velocity push toward distributed Spark engines (Glue or EMR), while high variety, mixing tables with JSON or free text, is what makes a flexible transform engine necessary rather than plain SQL over uniform rows.
- Call Amazon Bedrock from a transform job to enrich text with an LLM
To enrich records with a large language model (classify tickets, extract fields, summarize documents), invoke Amazon Bedrock from your Glue or EMR Spark code and write the model output as a new column. Bedrock is serverless and exposes a single inference API across foundation models, so the transform engine still does the heavy data movement while Bedrock supplies the per-record intelligence.
- Use Glue Flex execution for non-urgent jobs to cut cost when start latency is fine
Glue's Flex execution class runs non-urgent jobs (pre-production, testing, one-time loads) on spare capacity at lower cost, with the trade-off of slower and less predictable start times. Reach for it when a job is not time-sensitive; keep the standard execution class for time-sensitive workloads that need fast startup and dedicated resources.
Trap Putting a time-sensitive, SLA-bound job on Flex to save money; Flex trades fast startup for spare-capacity scheduling and can delay the run unpredictably.
- Know the Firehose Lambda transformation limits: 5-minute invoke, 0.2-3 MB buffer, and source backup
Amazon Data Firehose buffers records before invoking the transformation Lambda using a buffer size hint between 0.2 MB and 3 MB (default 1 MB); raising it toward 3 MB batches more records per invocation while staying under Lambda's 6 MB synchronous payload limit. A transformation invocation may run up to 5 minutes, after which Firehose times out and retries. Enable source record backup to deliver the original untransformed records to a separate S3 location concurrently with the transformed output.
Trap Confusing this Lambda buffering hint with the delivery buffering that flushes to the destination on size or interval.
9 questions test this
- A financial services company uses Amazon Data Firehose with Lambda transformation to process trading data and deliver it to Amazon S3.…
- A company is using Amazon Data Firehose to stream IoT sensor data to Amazon S3. The company has configured an AWS Lambda function to enrich…
- A financial services company uses Amazon Kinesis Data Firehose to stream transaction data to Amazon S3. The company implements a Lambda…
- A data engineering team uses Amazon Kinesis Data Firehose with AWS Lambda transformation to convert streaming sensor data to Apache Parquet…
- A company uses Amazon Data Firehose to deliver streaming data to Amazon S3 with Lambda transformation enabled. The data engineer wants to…
- A data engineer is configuring an Amazon Kinesis Data Firehose delivery stream with AWS Lambda transformation to process streaming data…
- A company is using Amazon Data Firehose to deliver log data to Amazon S3 with Lambda transformation. During peak traffic periods, the data…
- A financial services company uses Amazon Kinesis Data Firehose with Lambda transformation to process transaction data and deliver it to…
- A data engineer is configuring an Amazon Kinesis Data Firehose stream to deliver transformed clickstream data to Amazon S3. The Lambda…
- Set maximizeResourceAllocation so EMR sizes Spark executors to the core nodes
maximizeResourceAllocation is an Amazon EMR-specific Spark configuration that, when true, calculates the maximum compute and memory available for an executor on an instance in the core instance group and sets the spark-defaults executor, driver, and parallelism settings accordingly. It best fits a cluster running one large Spark application at a time so the job can use the whole cluster instead of leaving executors underutilized.
Trap Enabling it alongside other distributed applications such as HBase, whose custom YARN configurations conflict with maximizeResourceAllocation and can cause Spark jobs to fail.
4 questions test this
- A data engineer is optimizing an Amazon EMR Spark cluster that runs a single data transformation application at a time. The cluster uses…
- A company is running Spark batch jobs on Amazon EMR. The data engineer notices that the Spark executors are not fully utilizing the…
- A company is using Amazon EMR with Apache Spark to process terabytes of data daily. The EMR cluster uses uniform instance groups with…
- A company is running large-scale batch processing jobs using Apache Spark on Amazon EMR. The data engineering team wants to ensure that…
- Submit a Spark job to a running EMR cluster as a step, not over SSH
Use aws emr add-steps with command-runner.jar to submit a Spark application to an existing EMR cluster; pass spark-submit and your options (including --deploy-mode cluster) in Args so the driver runs on the ApplicationMaster inside the cluster rather than on the client. Running the job as a step makes it independent of any SSH session and records its status and logs under the cluster's Steps section.
Trap SSHing in to run spark-submit in client mode, which ties the driver to the terminal session and loses the step's tracking and logs.
8 questions test this
- A data engineer needs to submit a Spark application to an Amazon EMR cluster as part of an automated data pipeline. The application should…
- A data engineering team runs a Spark data transformation job on an Amazon EMR cluster. The job processes large datasets from Amazon S3 and…
- A data engineer needs to submit a PySpark application stored in Amazon S3 to a running Amazon EMR cluster. The application requires custom…
- A data engineering team runs PySpark data transformation jobs on Amazon EMR clusters. The jobs are submitted programmatically using the AWS…
- A data engineer is submitting a Spark application to an Amazon EMR cluster. The application needs to be highly available and continue…
- A data engineering team is running Apache Spark jobs on an Amazon EMR cluster. The team needs to submit a Spark application…
- A company runs data transformation jobs on Amazon EMR clusters using Apache Spark. The data engineering team submits Spark applications as…
- A data engineer needs to submit Spark applications to an Amazon EMR cluster as automated batch jobs without directly connecting to the…
References
- What is AWS Glue?
- What is Amazon EMR?
- What is Amazon EMR Serverless?
- What is AWS Lambda?
- Lambda quotas (15-minute runtime, 10 GB memory)
- AWS Glue FAQs (per-second billing, 1-minute minimum, DPU defaults) FAQ
- Tracking processed data using AWS Glue job bookmarks
- What is AWS Glue Studio?
- Use columnar storage formats (Athena: Parquet/ORC, column pruning, predicate pushdown)
- Connecting to data (AWS Glue JDBC connections)
- What is Amazon Bedrock?
- Instance purchasing options in Amazon EMR (EC2 Spot Instances)
- Reduce AWS Glue Spark ETL job start times (Flex execution class)