Domain 3 of 4 · Chapter 5 of 5

High-Performing Data Ingestion and Transformation

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AWS Certified Solutions Architect – Associate premium course — no separate purchase.

Included in this chapter:

  • Kinesis Data Streams: shard math, enhanced fan-out, retention tiers
  • Kinesis Firehose: buffer / transformation / format conversion patterns
  • Glue jobs: Spark vs Python, bookmarks, partition projection
  • Athena cost optimization: partitioning + columnar formats + workgroups

Ingestion + transformation services

ServiceTypeLatencyBest for
Kinesis Data StreamsStreamSub-secondOrdered events with replay (24h–365d); custom consumers
Kinesis FirehoseStream → store60+ s bufferFire-and-forget delivery to S3 / Redshift / OpenSearch / Splunk
MSK (Managed Kafka)StreamSub-secondKafka-compatible workloads; existing Kafka tooling
AWS GlueServerless ETL (Spark + Python)MinutesRecurring ETL; schema discovery via crawlers; Glue Data Catalog
EMRHadoop / Spark / Hive / Presto / Flink clusterMinutes-hoursLarge-scale data processing; non-Glue frameworks
AthenaServerless SQL over S3Seconds-minutesAd-hoc SQL; per-TB-scanned billing
RedshiftColumnar MPP DWSeconds-minutesPetabyte analytics warehouse; BI dashboards
Lake FormationData lake governance—Centralized fine-grained access on S3 + Glue Catalog
DataSync / DMS / SnowballTransferHours-daysOn-prem or other cloud → S3 migration
Cheat sheet

Cheat sheet

  • Choose streaming vs batch by how fresh the consumer needs the data
  • Match the transformation tool to the data's shape and size
  • Firehose delivers as soon as either the time or the size buffer is hit
  • Size Kinesis Data Streams shards at 1 MB/s in and 2 MB/s out each
  • The Glue Data Catalog is the lake's single shared metastore
  • Need row, column, or tag-level lake permissions, reach for Lake Formation
  • Size Glue jobs by worker type, where a DPU is 4 vCPU plus 16 GB
  • DMS replicates source to target, with optional ongoing CDC
  • Offline bulk transfer means Snowball Edge, since Snowmobile is retired
  • An IoT Core rule's error action fires when its primary action fails
  • IoT Core Basic Ingest skips the broker, so there's no per-message charge
  • A Firehose Lambda transform must return recordId, status, and base64 data

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Amazon Kinesis Data Streams
  2. Amazon Data Firehose (formerly Kinesis Data Firehose)
  3. Amazon MSK (Managed Streaming for Apache Kafka)
  4. What is AWS Glue
  5. Amazon EMR
  6. What is AWS Batch
  7. What is Amazon Athena
  8. AWS Glue components (Data Catalog)
  9. What is AWS Lake Formation
  10. Kinesis Data Streams scaling and quotas (shard limits)
  11. Kinesis Data Streams enhanced fan-out
  12. AWS Database Migration Service
  13. https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html
  14. https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html
  15. https://docs.aws.amazon.com/athena/latest/ug/ctas-partitioning-and-bucketing-what-is-bucketing.html