Domain 3 of 4 · Chapter 1 of 4

Data Modeling in Unity Catalog

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Implementing Data Engineering Solutions Using Azure Databricks premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Model the extraction before the table
  • Land as Delta, not CSV or Parquet
  • Lay out the table with liquid clustering
  • Choose the ingestion tool for a governed load
  • Model history: SCD, time travel, change feed
  • Managed vs external tables, and exam patterns

Choosing the ingestion tool for a governed load

ConsiderationLakeflow ConnectNotebook + Auto LoaderAzure Data Factory
Unity Catalog governanceNative; lands into UCNative; lands into UCExternal to UC governance
Coding effortLow or no-code connectorCustom code you maintainPipelines and activities
Incremental syncBuilt inAuto Loader tracks filesYou design it
Best fitSupported SaaS or DB sourcesArbitrary files, custom logicStaging or orchestration
MaintenanceFully managedYou own itYou own it

Decision tree

New table, or one youcan fully rewrite?Liquid clusteringCLUSTER BY, the defaultLegacy partitioned orfragmented, slow filters?OPTIMIZE + ZORDER BYcompact and co-locateOver 100 TB, low-cardinalitykey, proven partition win?Hive partitioningverify vs clustering firstLiquid clusteringconvert to CLUSTER BYYesNoYesNoYesNo
Cheat sheet

Cheat sheet

  • Incremental extraction reprocesses only new files
  • Source file type dictates schema and evolution handling
  • Snapshot-only sources are diffed, not appended
  • Choose Lakeflow Connect for governed managed-source ingestion
  • Choose notebooks for custom or unsupported-source ingestion
  • Azure Data Factory is an orchestration or landing option
  • Pick streaming vs batch by latency SLA and arrival pattern
  • One streaming pipeline can run in batch or continuous mode
  • Delta is the default and required format for lakehouse features
  • CSV and JSON are ingestion or landing formats, not query targets
  • Iceberg reads (UniForm) expose one Delta copy to Iceberg readers
  • Do not partition small or high-cardinality tables
  • Liquid clustering replaces partitioning and Z-order
  • OPTIMIZE plus ZORDER BY compacts files and co-locates filter columns
  • Deletion vectors avoid rewriting files on delete or update
  • SCD Type 1 overwrites; Type 2 preserves history
  • Implement SCD with MERGE or AUTO CDC INTO
  • Choose table grain from the finest reporting requirement
  • Delta time travel queries bounded historical versions
  • Change Data Feed records row-level change events
  • SCD Type 2 provides durable temporal history
  • DROP deletes data for managed but not external tables
  • Only managed tables get automatic predictive optimization
  • External tables require an external location and credential

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. What is Auto Loader?
  2. Get started using COPY INTO to load data
  3. Configure schema inference and evolution in Auto Loader
  4. The AUTO CDC APIs: Simplify change data capture with pipelines
  5. Configure Structured Streaming trigger intervals
  6. Unity Catalog managed tables for Delta Lake and Apache Iceberg
  7. Read Delta Lake tables with Iceberg clients using UniForm
  8. Use liquid clustering for tables
  9. When to partition tables on Azure Databricks
  10. Data skipping
  11. Deletion vectors in Databricks
  12. Managed connectors in Lakeflow Connect
  13. Work with table history
  14. Use change data feed on Azure Databricks
  15. Predictive optimization for Unity Catalog managed tables
  16. Work with external tables