Data Ingestion

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Pick the ingestion lane from the freshness requirement first

Sort every ingestion question by how fresh the data must be before comparing services: second-level freshness means streaming (Kinesis Data Streams or Amazon MSK), while a latency budget of minutes or hours means batch (a scheduled read from S3 with Glue or EMR). Streaming buys continuous, ordered, low-latency delivery; batch trades freshness for cheaper throughput on a bounded data set. Choosing the wrong lane, like a streaming pipeline for a nightly load, is the most common ingestion mistake.

Trap Reaching for a streaming service to satisfy a once-a-night load; batch from S3 is simpler and cheaper when second-level freshness is not required.

Choose Kinesis Data Streams when you need a replayable, multi-consumer log

Kinesis Data Streams is a raw shard log that many independent consumers can read at their own offsets and re-read within the retention window, so it fits fan-out to several processors and reprocessing after a bug. Amazon Data Firehose only delivers and keeps no re-readable buffer, and MSK is the answer instead only when a team specifically needs Kafka APIs. Pick Data Streams whenever the design must replay records or feed more than one downstream application from the same stream.

Trap Choosing Amazon Data Firehose when the requirement is replay or multiple independent consumers; Firehose is a one-way delivery pipe with no re-readable retention.

Use Amazon Data Firehose to land streaming data with no servers to manage

Amazon Data Firehose (formerly Kinesis Data Firehose) is a fully managed, serverless delivery stream with no shards to size and no consumers to run; you point producers at it and it loads to S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, or a generic HTTP endpoint. It can optionally invoke a Lambda function to transform records and convert their format in flight. Reach for it when the whole job is buffer-and-deliver and you do not need to re-read the data.

Firehose flushes on buffer size or buffer interval, whichever comes first

Amazon Data Firehose buffers incoming records by a buffer size in MB and a buffer interval in seconds, then delivers a batch when either threshold is reached. A larger interval produces fewer, bigger objects in S3 and adds delivery latency, while a smaller interval delivers sooner in smaller files; tune the two to trade freshness against object count. This buffering is why Firehose is near-real-time rather than truly real-time.

1 question tests this

• A data engineer is configuring an Amazon Data Firehose delivery stream to ingest IoT sensor data and deliver it to Amazon S3. The sensor…

A Kinesis shard caps at 1 MB/s or 1,000 records/s on writes

Each Kinesis Data Streams shard ingests up to 1 MB/s or 1,000 records per second on writes, whichever it reaches first, while a single record's data payload can be up to 10 MiB (the shard sustains 1 MB/s and absorbs the occasional large record with burst capacity). Exceed the write ceiling and the API returns ProvisionedThroughputExceededException, so size shard count to peak throughput in provisioned mode. Producers should aggregate or batch records and retry with exponential backoff to stay under the per-shard cap.

Classic shard reads share 2 MB/s; enhanced fan-out gives each consumer its own 2 MB/s

A shard's classic (shared) read path serves 2 MB/s total split across all polling consumers, capped at five GetRecords calls per second, so adding consumers starves them. Enhanced fan-out (EFO) gives each registered consumer a dedicated 2 MB/s per shard pushed over HTTP/2 via SubscribeToShard, which also lowers latency. Use EFO when several applications must each read the full stream at full speed; stay on shared reads when one or two consumers fit inside 2 MB/s, since EFO bills per consumer-shard hour plus data retrieved.

Trap Assuming adding a third or fourth classic consumer scales read throughput; they all divide the same shared 2 MB/s and throttle, which is exactly what EFO's dedicated 2 MB/s per consumer solves.

2 questions test this

• A financial services company ingests trading transactions into Amazon Kinesis Data Streams. The stream is configured in on-demand mode and…
• A financial services company uses Amazon Kinesis Data Streams to ingest transaction data from multiple banking applications. The stream has…

Enhanced fan-out is limited to 20 consumers per stream, not per shard

You can register up to 20 enhanced fan-out consumers for each data stream on On-Demand Standard and Provisioned modes (On-demand Advantage mode allows 50). The limit is per stream, not per shard, and each registered EFO consumer still gets its own dedicated 2 MB/s on every shard. Confusing the unit is a classic distractor.

Trap Reading the EFO limit as 20 consumers per shard; it is 20 registered consumers per stream (50 in On-demand Advantage mode), independent of shard count.

Use on-demand mode when load is spiky or unknown to skip shard math

Kinesis on-demand capacity mode removes shard sizing entirely: a new on-demand stream starts at 4 MB/s write and 8 MB/s read and scales automatically with traffic. It is the right call for spiky or unpredictable workloads where you would otherwise over- or under-provision shards. Provisioned mode is cheaper at steady, well-understood throughput where you can right-size shard count. You can switch a stream between on-demand and provisioned modes twice within 24 hours.

1 question tests this

• A company processes IoT sensor data using Amazon Kinesis Data Streams. The data volume varies significantly, with baseline traffic of 2…

Kinesis retention is what makes a stream replayable: default 24h, up to 365 days

A Kinesis data stream keeps records for a configurable retention period, defaulting to 24 hours and extendable to a maximum of 8,760 hours (365 days). Within that window a consumer can reset its iterator to an earlier sequence number and re-read, which is the mechanism behind replayability after a downstream bug or outage. A Firehose-only pipeline has no such window, so to combine managed delivery with replay you put Kinesis Data Streams in front of Firehose.

Use AWS DMS for database sources, including ongoing change data capture

AWS Database Migration Service (DMS) connects to relational, NoSQL, and warehouse sources and does both a one-time full load and ongoing change data capture (CDC) that replicates inserts, updates, and deletes to keep the target in sync. It supports fully heterogeneous migrations, so pair it with the AWS Schema Conversion Tool (SCT) when source and target engines differ, for example Oracle to Aurora PostgreSQL. DMS targets include S3, Kinesis, Amazon Redshift, and DynamoDB, so it is also how you stream a database's change feed into a data lake.

Trap Hand-rolling a polling job against the source database to capture changes; DMS CDC reads the database's transaction log and replicates inserts, updates, and deletes natively.

1 question tests this

• A company is migrating an on-premises Oracle database to Amazon Aurora PostgreSQL using AWS Database Migration Service (DMS). The data…

Use Amazon AppFlow to ingest from SaaS applications

Amazon AppFlow is the no-code, fully managed connector for SaaS APIs such as Salesforce, ServiceNow, Zendesk, and Slack, running flows on a schedule, on an event, or on demand and landing data in S3 or Amazon Redshift. When a question says to pull data from a named SaaS app, AppFlow is the answer rather than a custom API client or a generic file transfer service.

Trap Writing a custom API client (or using DataSync) to pull from Salesforce; AppFlow is the managed SaaS connector built for exactly that source.

Use DataSync for bulk file movement and Transfer Family for partner uploads

AWS DataSync moves large volumes of files or objects between on-premises storage (NFS, SMB, HDFS, object stores) and AWS (S3, EFS, FSx) with scheduling, verification, and high throughput, so it fits recurring bulk file ingestion you control. AWS Transfer Family is the inbound counterpart: it provides managed SFTP, FTPS, or FTP endpoints so external partners can push files straight into S3 or EFS without you running a file server. The deciding question is who initiates the transfer and over which protocol.

Trap Standing up a Transfer Family SFTP server to pull files you control from on-premises NAS; that is DataSync's job, while Transfer Family exists for partners pushing files in over SFTP/FTPS/FTP.

Use the Snow Family when data is too big to move over the network in time

When transferring the data set over the available bandwidth would take an unacceptable amount of time, ship it physically with AWS Snow Family (Snowball Edge) devices. Snowmobile is retired, so for multi-petabyte transfers answer with Snowball Edge fleets, AWS Direct Connect, or DataSync over a high-bandwidth link. The trigger phrase is a large volume plus limited bandwidth or a hard deadline.

Trap Answering a multi-petabyte transfer with Snowmobile; Snowmobile is retired, so the offline answer is Snowball Edge fleets (or Direct Connect / DataSync for an online link).

Trigger ingestion on events with S3 Event Notifications or EventBridge

Event-driven ingestion fires the moment data lands: an S3 Event Notification on s3:ObjectCreated:* can target Lambda, SNS, SQS, or EventBridge, which then starts a Lambda function, a Glue job, or a Step Functions workflow. Use events when latency matters and arrivals are irregular. Use a schedule (EventBridge Scheduler, a Glue trigger, or MWAA) instead when work is periodic and batched.

Call a Lambda from Kinesis with an event source mapping, not an S3 notification

To process a Kinesis (or DynamoDB) stream with Lambda, configure an event source mapping: Lambda polls the stream on your behalf and invokes the function per batch of records, with the batch size and parallelization you set. This is distinct from an S3 Event Notification, which fires on object creation in a bucket. When a stem says to call a Lambda from a Kinesis stream, the answer is the event source mapping.

Trap Wiring an S3 Event Notification to invoke Lambda from a Kinesis stream; S3 notifications fire on bucket object events, while a Kinesis-to-Lambda integration is an event source mapping that polls the shards.

Design ingestion to be replayable so a downstream bug never loses data

Replayability is the ability to re-process records after a failure without data loss, and the exam rewards designs that have it. A Kinesis stream is replayable within its retention window by resetting the iterator, and an S3 landing zone is replayable by construction because a failed batch job just re-runs against the same immutable objects. DynamoDB Streams gives a 24-hour replay window over item-level changes for the same reason; Amazon Data Firehose alone cannot replay, so front it with Kinesis Data Streams when you need both managed delivery and replay.

Keep related records ordered by routing them with a partition key

A stateless transaction processes each record independently, so retries and parallel consumers are safe; a stateful transaction depends on order or accumulated context. Kinesis preserves order only within a shard, so to process stateful records in sequence you assign a partition key that keeps related records (for example, all events for one customer) on the same shard. Spreading them across shards parallelizes throughput but loses cross-shard ordering.

Trap Assuming a Kinesis stream preserves global ordering across all shards; ordering is guaranteed only within a shard, so stateful, order-sensitive records need a partition key that pins them to one shard.

1 question tests this

• A data engineer is developing a real-time analytics application that processes clickstream data from an e-commerce website. The application…

Handle producer-side rate limits with backoff, batching, and aggregation

Downstream targets throttle on their own limits: a Kinesis shard at 1 MB/s or 1,000 records/s, and DynamoDB and RDS on their provisioned capacity. Producers absorb this by catching ProvisionedThroughputExceededException and retrying with exponential backoff, by batching with PutRecords, and by aggregating small records so fewer, larger writes stay under the cap. Throttling is a signal to scale capacity (more shards or on-demand mode) or smooth the producer, not to retry tighter.

DynamoDB Streams emits item-level changes retained for 24 hours

DynamoDB Streams captures a time-ordered, exactly-once log of item-level inserts, updates, and deletes and retains it for 24 hours. The StreamViewType controls what each record carries: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES, and it cannot be changed after the stream is created. Consume it with a Lambda trigger, Apache Flink, or the Kinesis adapter to drive change-driven ingestion such as replicating updates into a data lake.

Trap Expecting to switch a DynamoDB stream's StreamViewType (for example KEYS_ONLY to NEW_AND_OLD_IMAGES) in place; the view type is fixed at creation, so you must disable the stream and create a new one.

Start a CDC-only DMS task from a native start point matching where the load ended

When a full load was done separately (native tools or a prior full-load task), create a CDC-only DMS task and set CdcStartPosition to a native start point so replication resumes exactly where the load finished with no gap or overlap. For Oracle the native start point is the System Change Number (SCN); for MySQL and MariaDB it is the binary log file name and position written as file:position (e.g. mysql-bin-changelog.000024:373).

Trap Starting the CDC task from a wall-clock timestamp, which can skip or replay transactions because one timestamp can map to multiple native points in the log.

4 questions test this

• A company completed a full load migration from an on-premises Oracle database to Amazon RDS for PostgreSQL using AWS Database Migration…
• A company has completed the initial full load migration of its on-premises MySQL database to Amazon Aurora MySQL using AWS DMS. The data…
• A company has completed the initial full load migration of an on-premises Oracle database to Amazon RDS for Oracle using Oracle Data Pump.…
• A company has an on-premises MySQL database that generates 500 GB of redo logs daily. The company is using AWS DMS to perform an initial…

Route S3 events to an SQS FIFO queue through EventBridge, not directly

Amazon S3 Event Notifications cannot target an SQS FIFO queue; standard queues, SNS, Lambda, and EventBridge are the only direct destinations. To preserve strict ordering, enable EventBridge notifications on the bucket and add a rule whose target is the FIFO queue, since EventBridge supports sending S3 events to a FIFO queue.

Trap Configuring an S3 event notification to point straight at a .fifo queue, which the bucket configuration rejects.

4 questions test this

• A company has an existing data ingestion pipeline where Amazon S3 Event Notifications invoke an AWS Lambda function directly for…
• A healthcare company stores patient documents in an Amazon S3 bucket. The company needs to implement an event-driven architecture where S3…
• A data engineering team needs to send Amazon S3 object creation events to an Amazon SQS FIFO queue to maintain strict message ordering for…
• A financial services company processes trade data files uploaded to Amazon S3 using AWS Lambda. The company requires strict ordering…

Stop a Lambda from re-triggering itself by scoping the S3 trigger

When a Lambda triggered by S3 writes its output back to the same bucket, the new PutObject fires another notification and creates a recursive invocation loop. Break the loop by scoping the trigger with a prefix or suffix filter so only input objects invoke the function, or by writing transformed output to a different bucket or prefix.

Trap Triggering on the whole bucket and writing results back into it, so each transformed object launches the function again.

4 questions test this

• A company has developed an AWS Lambda function to transform CSV files uploaded to an Amazon S3 bucket into Apache Parquet format. The…
• A company has an AWS Lambda function that transforms CSV files uploaded to an Amazon S3 bucket. The Lambda function reads the files,…
• A financial services company processes incoming transaction files that are uploaded to an Amazon S3 bucket. An AWS Lambda function reads…
• A company has an AWS Lambda function that transforms CSV files into Parquet format and stores the output in an Amazon S3 bucket. The Lambda…

Use EventBridge for S3 events when you need suffix, size, or fan-out filtering

Plain S3 Event Notifications match only on prefix and suffix and allow one destination per event type, whereas EventBridge can filter on rich metadata such as object key suffix and numeric object size and route each pattern to its own rule with its own targets. Enable EventBridge on the bucket to send all events to the default event bus, then add multiple independent rules to route different file types or sizes to different consumers.

9 questions test this

• A financial services company ingests trading data files into an Amazon S3 bucket. The files are organized by date and stock symbol in the…
• A company is designing an event-driven data transformation pipeline. When files are uploaded to an Amazon S3 bucket, an AWS Lambda function…
• A company has an existing data lake with data stored in an Amazon S3 bucket. The company wants to implement a data transformation pipeline…
• A data engineering team is building an event-driven pipeline to process files uploaded to multiple Amazon S3 buckets. The team needs to…
• A company has a data ingestion pipeline that processes multiple file types uploaded to an Amazon S3 bucket. The company needs to route…
• A data engineer is configuring an event-driven pipeline where files uploaded to an Amazon S3 bucket with a specific folder structure…
• A data engineering team is building an event-driven data pipeline where an AWS Lambda function processes files uploaded to an Amazon S3…
• A data analytics company processes log files uploaded to an Amazon S3 bucket. The company needs to filter events to process only files…
• A healthcare company is building a data lake on Amazon S3. Multiple independent data processing teams need to trigger their own AWS Lambda…

Browse all 9 questions →