Domain 5 of 5 · Chapter 1 of 5

Optimizing Resources

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

  • Three levers, applied in order
  • Lever 1 and 2: BigQuery and Cloud Storage cost
  • Lever 3: capacity floor and cluster persistence
  • Exam-pattern recognition

Three cost levers for data workloads on Google Cloud

Cost leverThe decisionSteady / predictable choiceVariable / one-off choice
Shape the workReduce bytes scanned and storage held before pricing mattersPartition + cluster hot tables; long-term storage for cold tablesSame layout helps any usage; always do it first
Pricing modelPay per use or commit to capacity for analytics computeBigQuery Editions reservation + 1/3-year slot commitmentBigQuery on-demand (per byte scanned), no commitment
Capacity & lifetimeSize the capacity floor and choose cluster persistenceBaseline sized to the busy floor; long-running cluster when utilization stays highSmall baseline + autoscale for spikes; ephemeral Dataproc + Spot for batch

Cheat sheet

  • Fix an expensive query with layout, not more slots
  • Partition a BigQuery table on one date, timestamp, or integer-range column
  • Cluster on up to four high-cardinality filter columns to prune blocks
  • Use a materialized view, not a logical view, to cut bytes on repeated aggregations
  • BigQuery auto-discounts long-term storage after 90 days unmodified
  • Pick a Cloud Storage class by access frequency, not by data age alone
  • Cold storage classes charge for the full minimum duration on early deletion
  • Automate storage tiering with lifecycle rules or Autoclass
  • On-demand for unpredictable load, Editions slots for steady analytics
  • Commit to capacity only for steady, predictable usage
  • Size the reservation baseline to the floor, autoscale for the peak
  • Default to per-job ephemeral Dataproc clusters
  • Don't lift-and-shift an on-prem Hadoop cluster's persistence model
  • Run fault-tolerant batch on Spot VMs for up to ~91% off
  • Let Dataflow autoscale workers instead of fixing a worker count
  • Autoscale only Dataproc secondary workers, and let graceful decommission outlast the longest job
  • Turn on Enhanced Flexibility Mode so preemptible secondary workers can't lose shuffle data
  • Lower the worker utilization hint to make Dataflow scale up faster on spikes
  • Streaming Engine lets a streaming job scale down to one worker
  • Change a running streaming job's worker range with an in-flight update, no restart
  • Dataflow Prime Vertical Autoscaling fixes OOM workers without code changes
  • Raise numberOfWorkerHarnessThreads for hot keys once workers are maxed out
  • On a lifecycle-rule tie, Delete beats SetStorageClass and the coldest class wins

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Introduction to reservations
  2. Introduction to partitioned tables
  3. Introduction to clustered tables
  4. Introduction to materialized views
  5. BigQuery pricing (storage and long-term storage)
  6. Storage classes
  7. Object Lifecycle Management
  8. Autoclass
  9. Committed use discounts overview
  10. Dataproc overview
  11. Spot VMs