Domain 4 of 4 · Chapter 1 of 4

Designing and Implementing Data Pipelines

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Implementing Data Engineering Solutions Using Azure Databricks premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Order pipelines by data dependencies
  • Choosing a pipeline implementation approach
  • Authoring datasets in a declarative pipeline
  • Development mode and pipeline update mode
  • Orchestrating steps with Lakeflow Jobs
  • Notebook orchestration and error handling
  • Exam-pattern recognition

Choosing a pipeline implementation approach

Decision factorLakeflow Declarative PipelineNotebook task in a JobMulti-task Job DAG
Best-fit workIncremental dataset ETLProcedural / non-ETL logicOrchestrating heterogeneous steps
Compute lifecycleManaged automaticallyYou manage itPer-task compute
Checkpoints & target tablesRuntime maintains themYou write themDepends on task type
Execution orderDerived from dependenciesImperative in codeExplicit depends_on edges
Failure recoveryAutomatic flow/update retriesManual try/exceptPer-task retry policy

Decision tree

Work maps to streamingtables / materialized views?Procedural logicAPI, model, arbitrary Python?Low-latencycontinuous ingest?Notebook / Pythontask in a JobMulti-taskJob DAGSDP pipeline,continuous modeSDP pipeline,triggered modeNoYesYesNoYesNo
Cheat sheet

Cheat sheet

  • Pipeline order of operations follows the bronze-silver-gold medallion flow
  • Lakeflow SDP infers execution order from declared dataset dependencies
  • Separating ingest, transform, and serve stages lets each scale and recover independently
  • Choose SDP for declarative incremental ETL with built-in quality and orchestration
  • SDP manages infrastructure; a notebook task requires you to manage it imperatively
  • Notebooks suit procedural or non-ETL logic that cannot be expressed declaratively
  • A Lakeflow Job orchestrates a DAG of tasks linked by depends_on
  • Control-flow tasks add branching, iteration, and nested jobs
  • Pass data between tasks with task values and parameter references
  • Tasks come in several types beyond notebooks
  • A task retry policy automatically re-runs a failed task with a delay
  • Run-if conditions let downstream tasks run even when an upstream fails
  • Notebooks handle errors with try/except and surface status via dbutils.notebook.exit
  • %run inlines a notebook; dbutils.notebook.run executes it as a separate scope
  • Express precedence with Job tasks rather than deeply nested notebook calls
  • dbutils.notebook.run enables conditional branching and returns a string
  • SDP datasets are defined with pyspark.pipelines Python decorators
  • Streaming tables ingest incrementally; materialized views recompute results
  • Auto Loader is the recommended incremental file source inside SDP
  • SDP can also be authored in SQL with CREATE OR REFRESH statements
  • SDP development mode keeps compute warm; production mode restarts and retries
  • SDP pipelines update in triggered or continuous mode

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. What is the medallion lakehouse architecture?
  2. Develop Lakeflow Spark Declarative Pipelines code with SQL
  3. Develop pipeline code with Python
  4. Configure and edit tasks in Lakeflow Jobs
  5. Pipeline properties reference
  6. Triggered vs. continuous pipeline mode
  7. Configure task dependencies
  8. Use task values to pass information between tasks
  9. Orchestrate Databricks notebooks and modularize code