Prepare and process data

A batch of order files lands in cloud storage every hour, and before a single query against them can be trusted, four decisions stand between those files and a governed table. That is the shape of the whole domain: every path here reads a source and writes rows into Unity Catalog (UC), the governance layer built into Azure Databricks, as a Delta Lake table, the default table format that adds ACID (atomic, all-or-nothing) transactions and time travel to plain Parquet files. Hold that one model and the domain's recurring trap dissolves, because most questions are really asking which of the four decisions you are on and what the governed default is for it, not which clever command happens to exist. The four decisions run in order: model the target, ingest the data, cleanse and transform and load it, then constrain its quality.

Walk the four in order and you have the through-line of the domain. First, Data Modeling in Unity Catalog decides the destination's shape before any data moves: Delta as the default and required format, a managed table (Databricks owns the files and their maintenance) versus an external one, liquid clustering as the default layout, and how history is kept, whether a durable slowly changing dimension (SCD) Type 2 table or bounded time travel. Second, Ingesting Data into Unity Catalog copies source rows in, and the bookkeeping each method keeps (a COPY INTO file-load ledger, an Auto Loader checkpoint of discovered files, a Lakeflow Connect managed cursor) is what lets a re-run load only new data. Third, Cleanse, Transform, and Load Data turns raw rows into correctly typed, deduplicated rows and loads them idempotently, so a retry cannot double-load. Fourth, Data Quality Constraints in Unity Catalog decides which rules are guaranteed at write time and which are only documented.

Across every stage the exam rewards the same instinct. Prefer the option that keeps data inside Unity Catalog governance over one that lands an ungoverned copy outside it. Prefer the option that processes only new data over one that reprocesses all history, because a full reload with CTAS (CREATE TABLE AS SELECT) or CREATE OR REPLACE keeps no record of what it loaded and so cannot be incremental. And prefer the option that acts at write time, an enforced NOT NULL or CHECK constraint or a MERGE that corrects rows in one atomic transaction, over one that only hides a problem at read time or merely advises, such as a filtering view or an informational PRIMARY KEY that Unity Catalog never enforces. Governed, incremental, write-time is the tie-breaker.