Domain 3 of 4 · Chapter 2 of 4

Ingesting Data into Unity Catalog

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Implementing Data Engineering Solutions Using Azure Databricks premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • The bookkeeping model of ingestion
  • Streaming ingestion and the checkpoint
  • Auto Loader for files at scale
  • SQL file loads: COPY INTO and full-load CTAS
  • Managed ingestion with Lakeflow Connect
  • Change data capture with AUTO CDC
  • Exam-pattern recognition

Ingestion methods by interface, bookkeeping, and fit

CriterionCOPY INTOAuto LoaderNotebook Structured StreamingLakeflow Connect
InterfaceSQLPython or SQL (cloudFiles)Python or Scala codeManaged connector, no code
Bookkeeping it keepsFile-load ledger in the tableCheckpoint listing discovered filesCheckpoint of offsets, commits, and stateManaged cursor and change tracking
Incremental (skips loaded)YesYesYesYes (initial full load, then incremental)
Best fit by volumeBounded sets, up to ~thousands of filesContinuous or very high volume, millions of filesMessage buses such as Kafka and Event HubsSaaS applications and databases
Typical cadencePeriodic batchContinuous or scheduled batchContinuous or AvailableNow batchScheduled batch or continuous

Decision tree

Managed SaaS app ordatabase source?Lakeflow Connectmanaged connectorYesIngesting a CDCchange feed to upsert?NoAUTO CDCordered upserts / SCDYesSource is a message bus(Kafka / Event Hubs)?NoStructured Streamingkafka formatYesMillions of files orcontinuous arrival?NoAuto LoadercloudFilesYesCOPY INTOSQL, bounded file setNo
Cheat sheet

Cheat sheet

  • Lakeflow Connect lands source data in Unity Catalog streaming tables
  • Lakeflow Connect supports batch and continuous sync
  • Notebook batch ingestion reads then appends to a table
  • Notebook streaming ingestion uses readStream with a checkpoint
  • COPY INTO is idempotent and loads only new files
  • CTAS creates once; CREATE OR REPLACE fully rebuilds
  • COPY INTO can infer types and evolve schema
  • Choose Auto Loader vs COPY INTO by file volume and cadence
  • AUTO CDC applies ordered CDC upserts and deletes
  • A CDC feed needs an ordering key and operation column
  • Checkpoints persist offsets for fast exactly-once resume
  • Trigger mode trades latency against cost
  • Each streaming query needs its own checkpoint location
  • Ingest Event Hubs via its Kafka-compatible endpoint
  • startingOffsets and maxOffsetsPerTrigger bound consumption
  • Auto Loader incrementally detects and loads new files
  • SDP streaming tables ingest incrementally; materialized views recompute
  • Auto Loader offers directory-listing or file-notification mode

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Streaming tables
  2. Structured Streaming checkpoints
  3. Configure Structured Streaming trigger intervals
  4. Connect to Apache Kafka
  5. What is Auto Loader?
  6. COPY INTO
  7. Get started using COPY INTO to load data
  8. Managed connectors in Lakeflow Connect
  9. The AUTO CDC APIs: Simplify change data capture with pipelines