Domain 1 of 5 · Chapter 4 of 4

Data Migrations

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Data Engineer premium course — no separate purchase.

Included in this chapter:

  • Choosing the migration service from source and downtime
  • Migrating databases: DMS and Datastream
  • Loading warehouses and bulk data into BigQuery and Cloud Storage
  • Planning the move and validating the result
  • Exam-pattern recognition

GCP managed migration services by source and movement type

ServiceSourceDestinationMovement typePick it when
Database Migration ServiceMySQL, PostgreSQL, Oracle, SQL ServerCloud SQL, AlloyDBSnapshot + CDC, minimal downtimeMigrate a live relational DB with little outage
BigQuery Data Transfer ServiceTeradata, Redshift, S3, SaaS feedsBigQueryScheduled / one-time managed loadLand a warehouse or SaaS dataset in BigQuery
DatastreamOracle, MySQL, PostgreSQL, SQL ServerBigQuery, Cloud StorageServerless CDC streamKeep analytics in near real time from an OLTP source
Storage Transfer ServiceS3, Azure Blob, HDFS, on-prem filesCloud StorageOnline bulk, scheduledMove large object data when the link is fast enough
Transfer ApplianceOn-prem files / objectsCloud StorageOffline shipped hardwareBandwidth cannot meet the deadline, or site is remote

Decision tree

What is the source?Relational DBMinimal downtime?Yes, migrateDatabase MigrationService (DMS)Stream changesDatastreamCDC to BQ / GCSWarehouse / SaaSBigQuery DataTransfer Service (to BigQuery)Bulk objectsLink fastenough?YesStorage TransferService (online to GCS)NoTransferApplianceSource type is the first cut. Downtime budget splits a database into migrate (DMS) vs stream (Datastream).Link speed vs deadline splits bulk objects into online (Storage Transfer) vs offline (Transfer Appliance).
Cheat sheet

Cheat sheet

  • Pick the migration service from source type and downtime budget, not dataset size
  • CDC turns a multi-day database outage into a short cutover
  • Use Database Migration Service to move a live relational database with minimal downtime
  • DMS supports heterogeneous moves, so Oracle-to-PostgreSQL is in scope
  • Use Datastream for serverless CDC into BigQuery or Cloud Storage
  • DMS lands a managed database; Datastream feeds a change stream
  • BigQuery Data Transfer Service always lands data in BigQuery
  • Teradata and Redshift warehouse migrations to BigQuery are the canonical DTS scenario
  • Use Storage Transfer Service for online bulk objects into Cloud Storage
  • Use Transfer Appliance offline when bandwidth cannot meet the deadline
  • Online vs offline transfer is a bandwidth-vs-deadline calculation, not a size threshold
  • Cloud Interconnect and HA VPN are the migration path, not the data mover
  • A small one-off copy needs gcloud storage, not a managed transfer service
  • Plan from current state to desired state before moving any data
  • Validation proves completeness and fidelity before cutover
  • Scale Transfer Appliance with multiple parallel appliances or serial captures
  • Transfer Appliance security: CMEK key control, attestation, lock/unlock, and a wipe certificate
  • For EU residency, Transfer Appliance ships from and uploads in Belgium with an EU destination bucket
  • Storage Transfer Service managed private network avoids AWS egress fees
  • Event-driven Storage Transfer keeps Cloud Storage in near-real-time sync with S3
  • Datastream append-only mode keeps full change history; merge mode plus max_staleness tunes freshness vs cost
  • Warehouse-to-BigQuery migrations use a custom schema file, batch SQL translation, and the Data Validation Tool

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Database Migration Service overview
  2. What is BigQuery Data Transfer Service?
  3. What is Datastream?
  4. Storage Transfer Service overview
  5. Transfer Appliance overview
  6. Dedicated Interconnect overview
  7. Cloud VPN (HA VPN) overview
  8. Transferring your large datasets to Google Cloud Well-Architected