Data Ingestion & Storage

File formats: row vs columnar by access pattern

A model that trains on three numeric features out of a 400-column table should never pay to read the other 397. That single idea drives almost every file-format question on this exam: store data the way it will be read.

Columnar formats store all values of one column together. Apache Parquet^[1] and Apache ORC^[2] are the two AWS-native columnar formats. Because a query or a training read touches only the columns it needs, the engine performs column pruning (read just those columns) and predicate pushdown (skip row groups whose statistics rule them out). Both formats also compress well, since a column holds one data type. The result is dramatically less data scanned, which is exactly what Amazon Athena bills for^[3] at $5 per TB scanned. Use columnar for analytical scans, feature extraction across wide tables, and anything queried repeatedly at rest.

Row formats store whole records together. CSV and JSON are plain-text row formats: human-readable, universally supported, but uncompressed and slow to scan at volume. Apache Avro^[4] is a compact binary row format whose defining strength is schema evolution, the schema travels with the data so producers can add or change fields without breaking older readers. Row formats fit streaming ingestion and write-heavy pipelines, where you append or read entire records and the schema is still changing.

RecordIO-protobuf is the special case. SageMaker's built-in algorithms expect data in application/x-recordio-protobuf, a record-oriented binary format that pairs with Pipe input mode to stream batches efficiently into the training container. When a question shows a built-in algorithm plus Pipe mode, RecordIO-protobuf is the format being described.

The reconciling rule when these seem to conflict: choose the format for the dominant access pattern of that stage. Land streaming data as Avro or JSON because writes and schema changes dominate, then convert it to Parquet during transformation because reads dominate downstream. The two are not in tension; they sit at different points in the pipeline.

Columnar layout lets a query read one column and skip the rest; row layout reads whole records, fitting streaming and schema evolution.

SageMaker input modes and storage backends

The same S3 dataset can reach a training job three different ways, and the right one depends on whether the data must sit on the instance before training starts.

All three input modes share one model: deliver an S3 dataset to the training container. They differ only in when the bytes arrive, per the SageMaker training-data access docs^[5].

File mode (the default)

SageMaker downloads the entire dataset to a local directory on the instance, then starts training. The instance must have enough storage for the whole set, and startup time grows with dataset size and file count. File mode is the simplest choice and the right one for small datasets. For distributed training you shard with the ShardedByS3Key option so each instance downloads a slice.

Fast file mode

Fast file mode exposes S3 objects through a POSIX file-system interface but streams content on demand as the script reads it. Training starts almost immediately because SageMaker only identifies the files up front rather than downloading them, and the dataset no longer has to fit on the volume. It supports random access but works best when data is read sequentially, and it supports S3 prefixes only (no manifest or augmented-manifest files). Fast file mode is the current default streaming recommendation.

Pipe mode

Pipe mode pre-fetches S3 data at high concurrency and streams it into a named FIFO pipe; each pipe is read by a single process. Like fast file mode it needs only enough disk for the model artifacts. AWS describes fast file mode as the newer, simpler mechanism that largely replaces pipe mode, so reach for pipe mode mainly when you need its managed sharding and shuffling or the RecordIO-protobuf path the built-in algorithms use.

Beyond input modes you also choose the storage backend. Amazon S3 needs no VPC and is the default. Amazon FSx for Lustre^[6] mounts in seconds regardless of dataset size and scales to hundreds of GB/s and millions of IOPS, making it the choice for large datasets read repeatedly across epochs; it requires a VPC and uses a single Availability Zone to avoid cross-AZ data-transfer cost. Amazon EFS^[7] is the choice when the data already lives in EFS and is shared across instances, and it also requires VPC access. The trade is throughput-and-reuse against operational simplicity, not capacity.

For the latency-sensitive case, S3 Express One Zone directory buckets deliver single-digit-millisecond access and are supported as a training input for file, fast file, and pipe mode.

Feature Store as the feature backbone

Amazon SageMaker Feature Store^[8] stores curated features in feature groups. A feature group can have an online store, an offline store, or both. The online store keeps only the latest record per identifier for low-millisecond reads at real-time inference; the offline store keeps the full history in your S3 bucket as Parquet, queryable with Athena, for training and batch inference. Using one feature group for both training and serving is how Feature Store reduces training-serving skew, the accuracy gap that appears when training features and serving features are computed differently.

All three input modes feed the same S3 dataset to the training container; File mode downloads first, Fast file and Pipe mode stream on demand.

Streaming ingestion: choosing among Kinesis, Firehose, MSK, Flink

Four AWS streaming services answer four different questions. The fastest way to pick is to ask what you actually need: a buffer to read from, a managed pipe to a destination, the Kafka ecosystem, or computation inside the stream.

Amazon Kinesis Data Streams: a durable, replayable buffer

Kinesis Data Streams is a sharded stream you write to and read from with your own consumers. It retains records (24 hours by default, up to 365 days) so multiple consumers can read independently and replay, and it preserves order within a shard. A single record can carry up to 10 MiB^[9], while sustained per-shard write throughput is 1 MB/s or 1,000 records/s. When several consumers each need full-rate reads, enhanced fan-out gives each registered consumer its own dedicated 2 MB/s per shard; the limit is 20 registered consumers per stream^[10] (per stream, not per shard). Reach for Streams when you need ordering, replay, or multiple independent readers.

Amazon Data Firehose: zero-management delivery

Amazon Data Firehose^[11] (formerly Kinesis Data Firehose) is the serverless path that buffers records by size or time and delivers them to S3, Amazon Redshift, or OpenSearch, with optional conversion to Parquet or ORC on the way. There is nothing to provision and no consumers to write. The trade is that Firehose is a delivery pipe, not a buffer you read from with custom logic, and it adds buffering latency. Choose Firehose when the requirement is simply to land a stream in a store with no operational overhead.

Amazon MSK: managed Apache Kafka

Amazon MSK runs Apache Kafka so teams already standardized on Kafka topics, partitions, and the existing consumer ecosystem keep their tooling without operating brokers. It is the answer when a question explicitly names Kafka or an existing Kafka workload, not when the requirement is generic streaming.

Amazon Managed Service for Apache Flink: in-stream computation

Managed Service for Apache Flink runs Apache Flink for stateful, sub-second processing: windowed aggregations, joins, and on-the-fly feature computation over the stream. It is the choice when you must transform or compute features inside the stream rather than after landing, and it can write engineered features straight into a Feature Store online store.

What the exam stem looks like

The phrasing telegraphs the answer. "Multiple consumers, must replay, ordered" selects Kinesis Data Streams. "Just deliver to S3/Redshift/OpenSearch, no servers, convert to Parquet" selects Amazon Data Firehose. "Existing Kafka workload" selects Amazon MSK. "Aggregate or compute features in the stream in real time" selects Managed Service for Apache Flink. A common distractor is offering Firehose when the scenario needs replay or multiple independent readers; Firehose delivers and discards, it is not a readable, replayable buffer.

Pick the streaming service top-down: in-stream compute selects Flink, Kafka selects MSK, replay or multiple readers selects Kinesis Data Streams, plain delivery selects Firehose.

Extracting and merging data from many sources

Training data rarely starts in one clean S3 bucket. The exam tests how you pull it out of operational stores and combine it without melting a production database.

Extraction starts with the storage service the data already sits in. From S3 you accelerate large or long-distance transfers with S3 Transfer Acceleration^[12], which routes uploads over the CloudFront edge network. For block storage that backs a database or a self-managed store, EBS Provisioned IOPS (io1/io2)^[13] supplies the sustained, consistent IOPS a throughput-bound extract needs. Operational stores each have their own pull path: a full or incremental query against Amazon RDS, a Scan or Query against Amazon DynamoDB, or an export to S3.

The load-bearing rule for extraction is to read from a replica or an export, not the primary. A heavy analytical pull against a transactional primary steals capacity from live traffic; point the extract at an RDS read replica, a DynamoDB export to S3, or an S3 snapshot so production stays healthy.

Merging multiple sources scales up in three steps that share one model: combine records keyed on a common identifier. For modest joins you do it programmatically in a notebook or processing job. As data grows, AWS Glue^[14] gives you serverless Apache Spark plus a crawled Data Catalog so a Glue ETL job joins cataloged sources without managing a cluster. When you already run a Spark or Hadoop estate, Amazon EMR runs the same Apache Spark joins on a cluster you control. Glue is the serverless default; EMR is the choice when you need cluster-level control or an existing big-data stack. The figure below walks this escalation: a modest join stays in a notebook or processing job, terabyte-scale joins across sources move to serverless Spark on AWS Glue, and only an existing Spark estate or a need for cluster-level control justifies Amazon EMR.

The recurring trap in merge questions is reaching for a hand-written per-record loop or a single oversized instance when the volume clearly calls for distributed Spark. If the scenario says "terabytes" and "join across sources," the answer is Glue (serverless) or EMR (managed cluster), not a script.

Escalate the merge by scale: notebook or processing job for a modest join, AWS Glue for terabyte-scale joins, Amazon EMR only for cluster control.

Training-data delivery: storage backend and input mode tradeoffs

Criterion	S3 + File mode	S3 + Fast file / Pipe mode	FSx for Lustre	EFS
How data reaches the job	Full download to local disk before start	Streamed on demand from S3	Mounted file system, read in place	Mounted file system, read in place
Training start latency	Grows with dataset size	Near-immediate; data fetched as read	Seconds; independent of dataset size	Seconds; data must already be in EFS
Local disk must hold whole dataset	Yes	No	No	No
Throughput ceiling	EBS-bound	High (concurrent prefetch)	Hundreds of GB/s, millions of IOPS	Moderate to high by throughput mode
VPC setup required	No	No	Yes (single AZ)	Yes
Best fit	Small datasets, simplicity	Large datasets that need not fit on disk	Large datasets reused across epochs	Data already shared in EFS

Decision tree

Cheat sheet

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Use columnar Parquet or ORC when you scan a few columns over many rows

Apache Parquet and Apache ORC store data column-by-column, so an analytical query or a feature extract that reads a subset of columns prunes the rest and only scans what it needs. That cuts both runtime and Amazon Athena cost, which is billed per terabyte scanned, and the columnar layout compresses well because each column holds one data type. Reach for a row format like CSV or JSON instead when you genuinely read whole records or need human-readable output.

Trap Choosing CSV for a wide analytical table because it is simple; the full-row scan reads every column and inflates the data scanned and the Athena bill.

Use a row format like Avro or JSON for streaming and evolving schemas

Row-oriented formats keep each record's fields together, which suits append-heavy streaming ingestion where you write and read whole records. Apache Avro adds compact binary encoding and carries its schema with the data, so producers can add or change fields without breaking older readers, which is schema evolution. Convert the landed data to columnar Parquet later for the analytical reads downstream rather than forcing columnar at ingest.

Trap Forcing Parquet at the streaming ingest point where the schema is still changing; columnar files are awkward to append to and rewrite on every schema change.

RecordIO-protobuf pairs with Pipe mode for built-in algorithms

SageMaker's built-in algorithms expect training data in the record-oriented application/x-recordio-protobuf format, which streams efficiently through Pipe input mode in batches. When a question shows a built-in algorithm fed by Pipe mode, RecordIO-protobuf is the format being described. For your own scripts and frameworks you are not bound to it and usually keep Parquet, CSV, or the framework's native format.

File mode downloads the whole dataset before training starts

File mode is the default SageMaker input mode: it copies the entire dataset from S3 to the instance's local disk, then begins training, so the instance must have storage for the whole set and startup time grows with dataset size and file count. It is the simplest correct choice for small datasets. For distributed training, shard with ShardedByS3Key so each instance downloads only its slice rather than the full dataset.

Trap Picking File mode for a dataset larger than the training volume; the up-front full download has nowhere to land and the job fails on disk space.

Fast file mode streams S3 on demand so the data need not fit on disk

Fast file mode exposes S3 objects through a POSIX file-system interface but streams their content as the training script reads it, so training starts almost immediately and the dataset no longer has to fit on the EBS volume. It supports random access yet works best on sequential reads, and it accepts S3 prefixes only, not manifest or augmented-manifest files. AWS now positions it as the default streaming choice over the older Pipe mode.

Trap Assuming fast file mode accepts augmented manifest files; it supports S3 prefixes only, so a manifest-driven dataset needs File mode or Pipe mode.

Pipe mode pre-fetches S3 into a FIFO pipe and is largely superseded by fast file mode

Pipe mode streams S3 data at high concurrency into a named FIFO pipe read by a single process, so like fast file mode it needs only enough local disk for the model artifacts. AWS describes the newer fast file mode as the simpler mechanism that largely replaces it, so Pipe mode stays relevant mainly for its managed sharding and shuffling and for the RecordIO-protobuf path built-in algorithms use.

Choose FSx for Lustre for large datasets reread across many epochs

Amazon FSx for Lustre mounts to the training instance in seconds regardless of dataset size and scales to hundreds of GB/s and millions of IOPS, so it shines when the same large dataset is read repeatedly across epochs and S3 download or re-streaming would dominate. The cost is operational: it requires a VPC and uses a single Availability Zone, which you map to the training subnet to avoid cross-AZ data-transfer charges. For a small or read-once dataset, plain S3 File mode is simpler and cheaper.

Trap Standing up FSx for Lustre and its VPC plumbing for a small or one-off dataset; S3 File mode downloads it in seconds with no networking to configure.

Use Amazon EFS as a training source only when the data already lives there

SageMaker can mount an existing Amazon EFS file system to the training instance and start the script in place, which fits data already shared across instances in EFS. The data must already reside in EFS before training and the job must connect to a VPC to reach it. When data sits in S3, stream it with fast file or Pipe mode rather than first copying it into EFS.

Read extracts from a replica or export, not the production primary

A large analytical extract against a transactional primary competes with live traffic and can degrade the application, so point the pull at an Amazon RDS read replica, a DynamoDB export to S3, or an S3 snapshot instead. The replica or export absorbs the heavy scan while the primary keeps serving. This is the recurring right answer whenever a stem describes pulling training data from an operational database under load.

Trap Running the full training extract directly against the RDS primary; the analytical scan steals IOPS and connections from production traffic.

S3 Transfer Acceleration speeds large or long-distance S3 uploads

S3 Transfer Acceleration routes uploads through the nearest CloudFront edge location and then over the AWS backbone to the bucket, which raises throughput for large objects or clients far from the bucket Region. It helps when distance or object size is the bottleneck, not when the client is already close to the Region. It does not change how training reads data; it is an ingestion-speed option for getting data into S3.

Trap Enabling Transfer Acceleration for clients in the same Region as the bucket; with no distance to overcome it adds cost without improving throughput.

EBS Provisioned IOPS volumes give sustained consistent IOPS for I/O-bound extracts

When an extract or a self-managed data store is throughput-bound on its block storage, EBS Provisioned IOPS SSD volumes (io1/io2) deliver the sustained, consistent IOPS the workload specifies, which general-purpose gp3 may not hold under steady heavy load. io2 also supports higher durability and, with Block Express, very high IOPS ceilings. Use Provisioned IOPS when the requirement is a guaranteed IOPS floor rather than lowest cost.

Kinesis Data Streams is a durable, replayable, multi-consumer buffer

Amazon Kinesis Data Streams retains records (24 hours by default, extendable up to 365 days) so multiple independent consumers can read the same stream and replay it, and it preserves order within a shard. Reach for it when you need ordering, replay, or several readers, rather than a one-way delivery pipe. Amazon Data Firehose, by contrast, delivers and discards and cannot be re-read with custom logic.

Trap Choosing Amazon Data Firehose when the requirement is replay or multiple independent consumers; Firehose only delivers to a destination and is not a re-readable buffer.

4 questions test this

A Kinesis shard sustains 1 MB/s or 1,000 records/s on writes, with 10 MiB max per record

Each Kinesis Data Streams shard sustains writes of 1 MB/s or 1,000 records per second, whichever it hits first, so you scale write throughput by adding shards. A single record's payload can be up to 10 MiB (the data blob plus partition key must stay within 10,485,760 bytes), not 1 MB. Size the shard count to the aggregate ingest rate, and remember the record ceiling is 10 MiB, well above the per-second throughput figure.

Trap Stating the maximum Kinesis record size as 1 MB; the per-shard sustained write rate is 1 MB/s, but a single record can be up to 10 MiB.

1 question tests this

An ML engineer is troubleshooting a real-time data ingestion pipeline that uses Amazon Kinesis Data Streams in provisioned mode with 10…

Enhanced fan-out gives each consumer 2 MB/s per shard, capped at 20 consumers per stream

Enhanced fan-out (EFO) gives each registered consumer its own dedicated 2 MB/s read throughput per shard via a push model, instead of sharing the standard 2 MB/s per shard across all consumers. The registration limit is 20 registered consumers per stream, not per shard, for standard On-Demand and Provisioned modes. Use EFO when several consumers each need full-rate, low-latency reads; standard polling is cheaper when readers are few.

Trap Claiming enhanced fan-out allows 20 consumers per shard; the limit is 20 registered consumers per stream.

2 questions test this

Use Amazon Data Firehose to land streaming data with no servers to manage

Amazon Data Firehose (formerly Kinesis Data Firehose) buffers records by size or time and delivers them to Amazon S3, Amazon Redshift, or OpenSearch, with optional in-flight conversion to Parquet or ORC, and there is nothing to provision. The trade is added buffering latency and that it is a delivery pipe, not a buffer you read with custom consumers. Choose it when the goal is simply to land a stream in a store with minimal operational overhead.

Trap Reaching for Amazon MSK or self-managed Kafka when you only need buffered delivery to a store; Firehose is serverless and removes broker and shard management.

3 questions test this

Choose Amazon MSK when the workload is already on Apache Kafka

Amazon MSK runs managed Apache Kafka so teams already invested in Kafka topics, partitions, and the existing consumer ecosystem keep their tooling without operating brokers. It is the right answer when a stem explicitly names Kafka or an existing Kafka estate, not when the requirement is generic streaming that Kinesis or Firehose covers more simply.

Trap Picking Amazon MSK for net-new generic streaming with no Kafka requirement; you take on Kafka's operational model for nothing Kinesis would not handle.

Use Managed Service for Apache Flink to compute features inside the stream

Amazon Managed Service for Apache Flink runs Apache Flink for stateful, sub-second stream processing: windowed aggregations, joins, and on-the-fly feature computation, and it can write the engineered features straight into a SageMaker Feature Store online store. Reach for it when transformation or feature creation must happen in the stream in real time, rather than after the data has landed in S3.

Use AWS Glue for serverless Spark joins across cataloged sources

AWS Glue runs serverless Apache Spark with a crawled Data Catalog, so a Glue ETL job joins multiple cataloged sources on a common key at scale without you managing a cluster. It is the default for combining terabyte-scale data from many sources. Amazon EMR runs the same Apache Spark joins on a cluster you control, which is the choice when you need cluster-level tuning or already operate a Spark or Hadoop estate.

Trap Writing a hand-rolled per-record loop or using one oversized instance to join terabytes across sources; the volume calls for distributed Spark on Glue or EMR.

Custom Grok or CSV classifiers let a Glue crawler parse non-standard formats

Built-in Glue classifiers cover JSON/CSV/Parquet/Avro but return an UNKNOWN certainty on fixed-width or oddly-delimited data; attach a custom Grok classifier (regex patterns mapping text to field names) for fixed-width or unusual text files, or a custom CSV classifier (specifying delimiter, quote, header, or the Open CSV SerDe for quoted commas). The crawler runs custom classifiers before the built-ins.

Trap Assuming the built-in CSV classifier handles pipe-delimited or fixed-width files it cannot actually parse.

6 questions test this

Glue crawler schema-change policy controls how discovered schema changes are applied

A Glue crawler's schema-change policy governs catalog updates: AddOrUpdateBehavior=MergeNewColumns adds newly discovered columns while preserving existing definitions; UpdateBehavior=LOG records changes without overwriting manual edits (UPDATE_IN_DATABASE is the default, which does overwrite); RecrawlBehavior=CRAWL_NEW_FOLDERS_ONLY only scans new partitions and leaves existing schema untouched; and 'Create a single schema for each S3 path' merges compatible files into one table instead of many.

Trap Leaving UpdateBehavior at UPDATE_IN_DATABASE, so the next crawl overwrites schema corrections data scientists made by hand.

6 questions test this

Kinesis on-demand mode auto-scales for unpredictable traffic with no shard management

On-demand capacity mode makes Kinesis Data Streams manage shards automatically, accommodating up to double the previous 30-day peak write throughput, which suits highly variable or unpredictable workloads with the least operational overhead. You can switch an existing stream between provisioned and on-demand (up to twice per 24 hours) without disrupting producers or consumers.

Trap Manually resharding a provisioned stream for spiky traffic when switching to on-demand mode removes the capacity-planning burden entirely.

5 questions test this

S3 Intelligent-Tiering fits unknown or changing access patterns with no retrieval fees

S3 Intelligent-Tiering automatically moves objects between Frequent, Infrequent (after 30 days), and Archive Instant Access (after 90 days) tiers based on actual access, charging no retrieval fees, which makes storage costs predictable for datasets whose access pattern is unpredictable. Optional Archive Access and Deep Archive Access tiers add deeper savings for data untouched for months (a small per-object monitoring fee applies).

Trap Hand-authoring lifecycle transition rules for data whose access pattern is genuinely unknown, where Intelligent-Tiering adapts automatically.

5 questions test this

S3 Glacier Instant Retrieval archives rarely-used data while keeping millisecond access

S3 Glacier Instant Retrieval is the lowest-cost class for long-lived data accessed only about once a quarter that still needs millisecond retrieval (e.g. audit or compliance datasets), saving up to ~68% versus S3 Standard-IA with no restore step (it has a 90-day minimum storage duration and higher per-GB retrieval fees). Lifecycle NoncurrentVersionTransition can move older object versions into it while current versions stay in S3 Standard.

Trap Choosing S3 Glacier Flexible or Deep Archive for rarely-accessed data that must still be readable in milliseconds, forcing a multi-hour restore.

7 questions test this

Match S3 storage class to access frequency and cascade lifecycle transitions

Cost-optimize S3 by transitioning objects down the waterfall as access drops: S3 Standard for active data, Standard-IA for infrequent-but-fast access, then Glacier Flexible Retrieval (3-5 hour standard restore) or Deep Archive for cold data, with an expiration action to delete at end of retention. S3 Standard has no minimum storage duration, so short-lived raw data can be expired without a minimum-duration charge (Standard-IA has a 30-day minimum).

Trap Parking data needed within milliseconds in Glacier Flexible Retrieval, whose standard restore takes 3-5 hours.

8 questions test this

File formats: row vs columnar by access pattern

SageMaker input modes and storage backends

File mode (the default)

Fast file mode

Pipe mode

Feature Store as the feature backbone

Streaming ingestion: choosing among Kinesis, Firehose, MSK, Flink

Amazon Kinesis Data Streams: a durable, replayable buffer

Amazon Data Firehose: zero-management delivery

Amazon MSK: managed Apache Kafka

Amazon Managed Service for Apache Flink: in-stream computation

What the exam stem looks like

Extracting and merging data from many sources

Training-data delivery: storage backend and input mode tradeoffs

Decision tree

Cheat sheet

Also tested in

References