Domain 2 of 5 · Chapter 3 of 3

Deploying & Operationalizing

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

  • Orchestrate the runs, then ship the code
  • Composer DAGs and Cloud Scheduler triggers
  • CI/CD for data pipelines: build, parameterize, promote
  • Exam-pattern recognition

Orchestration choices for data pipelines

CapabilityCloud Composer (Airflow)WorkflowsCloud Scheduler
Primary jobOrchestrate many dependent tasksChain a few services/APIsTrigger one target on a timetable
Authoring modelPython DAG (Apache Airflow)YAML/JSON step definitionCron expression + target
Operates a clusterYes, a managed Airflow environment runs continuouslyNo, fully serverlessNo, fully serverless
Pricing shapePer environment (always-on)Per step executedPer job (flat, very low)
Retries and backfillBuilt in (task retries, catchup, backfill)Per-step retry and error handlingNone, only re-fires on schedule
Reach for it whenComplex multi-task data workflowsLightweight service orchestrationPure scheduled trigger

Decision tree

Many dependent tasks withretries, scheduling, backfill?YesNoCloud Composermanaged Airflow DAGChain a few services or APIswith branching logic?YesNoWorkflowsserverless, pay per stepJust fire one jobon a timetable?YesNoCloud Schedulercron trigger, coordinates nothingReconsider:it is a build/design step
Cheat sheet

Cheat sheet

  • Separate orchestration from deployment when operationalizing a pipeline
  • Reach for Cloud Composer when a workflow has many dependent tasks
  • Use Workflows to chain a few services with branching, serverless
  • Cloud Scheduler is the cron trigger, it orchestrates nothing
  • A DAG starts a task only after its upstream tasks succeed
  • Composer schedule_interval and backfill handle recurring and missed runs
  • Deploy DAGs by syncing the dags/ folder into the Composer bucket
  • Cloud Build is CI: it tests and builds the artifact on a source change
  • Flex Templates build the graph at launch, classic templates freeze it at build
  • Dataflow templates separate who writes a pipeline from who runs it
  • Dataform release configurations promote SQL pipelines across environments
  • Promote one parameterized artifact, do not edit code per environment
  • CI builds the artifact, CD promotes it; a deploy stops short if it only builds
  • Operationalizing is not choosing the engine or writing the transform
  • Chain one DAG to the next with TriggerDagRunOperator, not a guessed schedule
  • Use deferrable operators so long-running jobs stop holding a worker slot
  • Run long-waiting sensors in reschedule mode to release the worker between pokes
  • Group related Airflow tasks with TaskGroup, never SubDAGs
  • Spread DAG start times and shrink parse time to relieve the scheduler
  • Cross-environment DAG dependencies use Composer operators or Pub/Sub, not TriggerDagRun
  • Lower the worker utilization hint to make Dataflow autoscale up sooner
  • Adjust a running Streaming Engine job's worker range with an in-flight update
  • Use FlexRS for non-time-critical batch jobs to cut cost with preemptible VMs
  • Track latency percentiles with a distribution log-based metric and a regex extractor
  • Alert on a specific log message with a log-based alerting policy, no metric needed
  • Configure missing-data handling so log-metric gaps do not fire false alerts
  • Detect a falling-behind Dataflow pipeline with system_lag and data freshness metrics

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Cloud Composer overview
  2. Cloud Build overview
  3. Test, synchronize, and deploy your DAGs (Composer CI/CD)
  4. Cloud Scheduler overview
  5. Dataflow templates (classic and Flex)
  6. Dataform release configurations