Monitoring & Troubleshooting

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Match the symptom to metrics, logs, or job introspection

Observability gives three signal kinds and each answers a different question: metrics (numeric time series in Cloud Monitoring) tell you whether a rate or backlog is wrong now, logs (text events in Cloud Logging) tell you what failed and why, and per-service job introspection (BigQuery INFORMATION_SCHEMA.JOBS, the Dataflow job-metrics tab, the Airflow UI) tells you which run or stage is at fault. Reach for the source that fits the question instead of scrolling one console; a growing backlog is a metric, its stack trace is a log, and the straggler stage is job introspection.

You cannot alert on a raw log; turn it into a metric first

Cloud Monitoring alerting policies fire on metric thresholds, not on log text, so to page on a specific log event you create a log-based metric over a log filter and then attach an alerting policy to that metric. A counter log-based metric counts entries matching the filter; a distribution log-based metric extracts a number from each matching entry (such as a parsed latency) and builds a histogram. The flow is always log filter, then metric, then alert.

Trap Trying to attach an alerting policy directly to a log entry; Cloud Monitoring conditions evaluate metrics, so the log must become a log-based metric first.

5 questions test this

• Your organization runs business-critical BigQuery batch jobs nightly. You need to be automatically notified when any of these jobs fail so…
• Your Cloud Composer environment runs data pipeline DAGs that process critical business data. You need to implement alerting to notify your…
• Your organization runs multiple Dataproc clusters across several Google Cloud projects. You need to create a centralized monitoring…
• Your organization uses Cloud Composer in production and wants to implement a robust alerting system for DAG failures and SLA misses. The…
• Your team manages critical ETL pipelines running on Cloud Composer. You need to implement alerting so that operations staff receive…

Log retention is a property of the bucket, not the entry

Cloud Logging keeps entries for as long as the log bucket they land in is configured to, not per entry. The _Default bucket retains everything for 30 days, the _Required bucket holds Admin Activity and required audit logs for a fixed 400 days, and a user-defined bucket can be set from 1 day up to 3650 days (10 years). To keep audit logs past 400 days you route them through a Log Router sink to a custom bucket or to Cloud Storage.

Trap Assuming the _Required audit-log bucket's 400-day retention is configurable; it is fixed, so longer audit retention needs a sink to another bucket or Cloud Storage.

The Log Router sends matching entries to four sink destinations

Every Cloud Logging entry passes through the Log Router, which evaluates sinks; each sink has an inclusion filter and one destination. A sink can write to a Cloud Logging bucket (keep and search), a BigQuery dataset (SQL analysis), a Cloud Storage bucket (cheap long-term archive), or a Pub/Sub topic (stream to a third-party SIEM or custom processor). Choosing the destination by intent is how you both retain and analyze the right logs.

Trap Picking a Cloud Storage sink when the goal is to run SQL over the logs; Cloud Storage is cheap archive, but querying needs the BigQuery dataset destination.

1 question tests this

• Your data engineering team runs multiple Dataproc clusters across various projects. You need to centralize all Dataproc job failure logs…

Alert on user-felt symptoms, the four golden signals

Google SRE's four golden signals to monitor for any user-facing service are latency, traffic, errors, and saturation. Anchor alerts to these symptoms and to an SLO rather than to an internal cause such as a single CPU spike, because a CPU blip that no user feels is noise while a latency or error climb is the thing worth paging on.

Error Reporting groups recurring failures so one bug is one entry

Error Reporting reads log entries from running services, recognizes stack traces, and groups errors by exception type and the top stack frames, so a single recurring failure surfaces as one group rather than thousands of identical lines, and it can notify on a new error. It samples up to about 1,000 errors per hour and estimates counts beyond that, so treat the count as an estimate at high volume.

INFORMATION_SCHEMA.JOBS is the 180-day BigQuery job audit trail

Query INFORMATION_SCHEMA.JOBS (and its JOBS_BY_USER, JOBS_BY_FOLDER, JOBS_BY_ORGANIZATION views) with SQL to see running jobs plus job history for the past 180 days. The columns that drive cost and performance triage are total_bytes_billed (the on-demand cost driver), total_slot_ms (slot-time consumed), job_type/statement_type, state, and error_result for why a job failed. A scheduled query summing total_bytes_billed by user_email is the standard who-is-spending report.

Trap Reaching for Cloud Billing Reports to find which BigQuery query was expensive; Billing Reports attributes by service and SKU, but the per-query culprit lives in INFORMATION_SCHEMA.JOBS.

1 question tests this

• You are troubleshooting a BigQuery query job that failed. You need to programmatically identify the failure reason to implement automated…

Read the BigQuery query plan: each symptom names one fix

The query plan and execution details (Query Insights) break a query into stages reporting slot wait time, shuffle bytes, and records read versus written. High slot wait time means slot contention, a concurrency problem fixed by adding slots; large shuffle bytes mean a big join or data skew, fixed by denormalizing; records read far larger than the result means scanning too much, fixed by partitioning, clustering, or a tighter filter. The symptom in the plan picks the remedy.

1 question tests this

• Your team runs a daily scheduled query in BigQuery that has started timing out intermittently. When analyzing the query execution graph,…

Adding slots does not shrink the bytes a query scans

Slots are processing capacity, so they speed up running a query but never change how many bytes a poorly-laid-out query must read. A single slow query is therefore usually a layout problem (no partition or clustering, a wide scan) rather than a slot shortage, and the fix is to reduce bytes scanned, not to buy capacity. Slots address concurrency and many-query contention, not one query's data volume.

Trap Buying more slots to fix one slow query whose plan shows a huge scan; capacity does not reduce bytes read, so the scan stays just as large.

Streaming Dataflow is healthy when data freshness stays low

In the Dataflow job-metrics tab, data freshness is the gap between an element's event-time timestamp and when it is processed, and system latency is the current maximum seconds an element has been processing or waiting. Rising data freshness together with rising backlog seconds (the estimated time to drain the current backlog) is the classic can't-keep-up picture; check the autoscaling chart to see whether workers are scaling toward the target or have hit their ceiling.

Trap Reading high data freshness as stale source data; it measures how far behind processing is, so a rising value means the job cannot keep up, not that the input aged.

1 question tests this

• Your streaming Dataflow pipeline reads from Pub/Sub and writes to BigQuery. Users report that dashboard visualizations are showing stale…

Watch oldest_unacked_message_age for a Pub/Sub backlog

A slow or stuck subscriber shows up as a growing Pub/Sub subscription backlog, and the subscription/oldest_unacked_message_age metric reports how stale the oldest unacknowledged message is. Watching it tells you the source side is falling behind before the downstream pipeline visibly stalls, which complements the Dataflow data-freshness view of the processing side.

A quota blocks the request; a budget only notifies

A quota is a hard limit: when a request would exceed it the system blocks the request and the operation fails, so quotas can actually prevent consumption. A Cloud Billing budget is the opposite, notification-only, and does not cap or stop spending; its default thresholds alert at 50%, 90%, and 100% of the amount you set against actual or forecasted cost. If you must stop runaway usage the lever is a quota; a budget just tells you it is happening.

Trap Setting a Cloud Billing budget to cap or halt spending; a budget only sends alerts and never stops usage, so spend continues past the threshold.

A quota-exceeded error needs an increase, not a retry

A quota-exceeded error means a hard limit was reached, so retrying the same request keeps failing; the fix is to request a quota adjustment in Cloud Quotas, in most cases at the project level, and adjustment requests are subject to review with an email acknowledging receipt. This differs from a rate limit (a per-second or per-minute cap that smooths bursts), where client-side retry with exponential backoff is the correct response.

Trap Answering a hit absolute quota with retry-and-backoff; backoff handles transient rate limits, but an exhausted hard quota only clears with an increase request.

Budget alerts can drive automation through Pub/Sub

Because a budget cannot stop spend on its own, route its alerts to a Pub/Sub topic so a subscriber (for example a Cloud Function) can take an automated response such as disabling billing on a project. The budget still only notifies; the Pub/Sub-triggered code is what actually acts, which is the supported way to approximate a hard spending stop.

Attribute spend with Cloud Billing Reports by service and SKU

When a budget warns that cost rose, Cloud Billing Reports is the built-in visualization that breaks spend down by project, service, SKU, location, label, and time, with forecasted as well as actual figures. It answers which service or SKU drove the spend, while a per-query cost answer for BigQuery comes from INFORMATION_SCHEMA.JOBS, not from Billing Reports.

Watch BigQuery slot usage in the admin resource charts

When BigQuery runs on capacity (editions) pricing, queries draw slots from reservations, and the administrative resource charts show slot usage, job concurrency, and job execution over time. Use them to confirm whether queries are waiting on slots at the project level, which is the same slot-wait signal the per-query plan reports, viewed across the whole reservation.

Cancel a runaway job now, then prioritize with reservations

The immediate operational action for a job consuming far more than expected is to cancel that specific BigQuery or Dataflow job. To prevent recurrence you route heavy batch and interactive queries into separate reservations so one workload does not starve the other, but sizing those reservations is capacity planning; here the in-flight fix is the cancel and the routing is the observe-and-act follow-up.

A metrics scope views many projects from one pane

A Cloud Monitoring metrics scope lets one scoping project view metrics from many Google Cloud projects, and even AWS accounts, so you watch an entire data platform from a single dashboard rather than hopping per project. Pair it with alerting policies and notification channels (email, PagerDuty, Slack, Pub/Sub, the mobile app) to centralize both viewing and paging.

Attribute Dataflow cost per team with additionalUserLabels and billing export

The additionalUserLabels pipeline option attaches labels (department, environment, business unit) to each Dataflow job; the labels flow into the Cloud Billing export to BigQuery, where you filter and aggregate cost by label. This needs no extra billing configuration and the labels also filter jobs in the Dataflow monitoring UI.

Trap Trying to attribute spend per job from the billing console alone, without first stamping jobs with additionalUserLabels.

3 questions test this

• Your company runs multiple Dataflow batch pipelines across several departments and needs to track costs at a departmental level. Finance…
• Your company runs multiple Dataflow batch pipelines across different business units. Management requires cost attribution reports that show…
• Your data engineering team manages over 200 Dataflow batch jobs across multiple business units. The finance department needs to track costs…

Use INFORMATION_SCHEMA.JOBS_TIMELINE for per-second BigQuery slot analysis

JOBS_TIMELINE has one row per second of every job's execution, so it is the most granular view of slot usage: period_slot_ms shows slot-milliseconds consumed per job per second, and period_estimated_runnable_units greater than zero means the job wanted more slots than were available. The plain JOBS view only summarizes a job, not its second-by-second timeline.

Trap Querying INFORMATION_SCHEMA.JOBS for slot contention, which lacks the per-second timeline that JOBS_TIMELINE provides.

3 questions test this

• Your organization wants to proactively monitor BigQuery slot utilization across all projects and set up alerts when usage exceeds 90% of…
• Your organization uses BigQuery reservations for capacity-based pricing. A business analyst reports that their queries are running slower…
• You are troubleshooting slow query performance in BigQuery and suspect slot contention. You need to identify which jobs consumed the most…

Alert on slow BigQuery with the query/execution_times metric at the 99th percentile

The native bigquery.googleapis.com/query/execution_times metric feeds a Cloud Monitoring alerting policy directly. Aggregate it at the 99th percentile so an alert fires when consistently slow queries cross a threshold while ignoring occasional outliers.

Trap Building a log-based metric from query logs to track latency, when execution_times is already an alertable native metric.

2 questions test this

• Your company wants to receive automated alerts when BigQuery query execution times exceed acceptable thresholds. You need to implement…
• A financial services company runs critical overnight batch jobs in BigQuery that must complete before market opening. The operations team…

Enable Dataproc OSS metrics with --metric-sources, billed and found under VM Instance

Default Dataproc cluster and job metrics are collected free. Custom open-source metrics (yarn, spark, hdfs, and more) require the --metric-sources flag at cluster creation and are billed, so use --metric-overrides to collect only the critical ones for important clusters. These custom metrics carry a custom.googleapis.com/ prefix and appear in Metrics Explorer under the VM Instance resource, not under Cloud Dataproc Cluster.

Trap Hunting for custom OSS metrics under the Cloud Dataproc Cluster resource, where only the standard dataproc.googleapis.com metrics live.

8 questions test this

• Your company is concerned about Cloud Monitoring costs for Dataproc clusters. You want to monitor cluster health effectively while…
• Your organization runs production Spark workloads on Dataproc clusters. The operations team needs to monitor YARN resource utilization and…
• Your organization wants to set up monitoring alerts for Dataproc jobs that run longer than expected. Jobs submitted through spark-submit…
• Your organization runs multiple Dataproc clusters across several projects. You need to enable comprehensive monitoring of cluster resource…
• Your Dataproc cluster running image version 2.0 is experiencing intermittent performance issues. You want to collect additional…
• Your data engineering team manages multiple Dataproc clusters running Spark and Hadoop workloads. You need to enable custom metric…
• You are troubleshooting a Dataproc cluster performance issue and need to view custom Spark and YARN metrics that were enabled during…
• Your data engineering team runs critical Spark workloads on Dataproc clusters and needs to monitor YARN resource utilization, Spark driver…

Find Dataproc job and executor logs under the Cloud Dataproc Job resource

Jobs submitted through the Dataproc Jobs API write driver and YARN container logs to Cloud Logging under the Cloud Dataproc Job resource type; filter by job_id and select yarn-userlogs to read Spark executor output. Cluster-level events (agent, startup) live under the Cloud Dataproc Cluster resource instead.

Trap Looking for Spark executor logs under the Cloud Dataproc Cluster resource, where only cluster-level entries appear.

2 questions test this

• A Spark job running on your Dataproc cluster is failing intermittently. You need to troubleshoot the issue by examining detailed executor…
• You are troubleshooting intermittent Dataproc job failures and need to analyze the google.dataproc.agent logs from the cluster master node.…

Grant the log sink's writer identity a write role on the destination

A Cloud Logging sink writes through a unique writer-identity service account that has no permissions by default. After creating the sink you must grant that identity the destination's write role: Pub/Sub Publisher on a topic, or Storage Object Creator on a bucket (least privilege for compliance). Without it the sink errors and entries never arrive.

Trap Granting broad project roles or your own account access, instead of giving the sink's own writer identity the destination write role.

1 question tests this

• Your Dataproc clusters process sensitive financial data. You must export cluster initialization and job failure logs to Cloud Storage for…

Alert on DAG and SLA failures by logging from callbacks and matching with a log-based policy

The Google-recommended Airflow alerting pattern is to write structured messages from on_failure_callback (DAG-level for run failures, task-level via default_args for task failures) and sla_miss_callback, then create Cloud Monitoring log-based alerting policies that match those messages and route to channels like PagerDuty. The sla parameter is set on the task as a timedelta, but sla_miss_callback must be set at the DAG level, not in default_args.

Trap Putting sla_miss_callback in default_args or on the task, where Airflow never invokes it; only the DAG-level callback fires.

11 questions test this

• You are developing an Airflow DAG in Cloud Composer that processes hourly sales data. Business stakeholders require notification if the…
• You are configuring SLA monitoring for a Cloud Composer DAG that must complete within a specific timeframe. You need to be notified when…
• Your organization runs production data pipelines in Cloud Composer and needs comprehensive alerting for DAG failures and SLA misses. The…
• A data engineer is configuring failure notifications for a Cloud Composer DAG using callbacks. The callbacks need to log messages that…
• Your data engineering team uses Cloud Composer to orchestrate ETL pipelines. You want to receive notifications when any task within a…
• Your Cloud Composer environment runs data pipeline DAGs that process critical business data. You need to implement alerting to notify your…
• A data engineer is implementing failure notification callbacks for a Cloud Composer DAG. They want to send different notifications…
• Your organization uses Cloud Composer in production and wants to implement a robust alerting system for DAG failures and SLA misses. The…
• You are developing an Airflow DAG in Cloud Composer that must notify your team via Slack when individual tasks fail but should send an…
• Your data engineering team runs time-sensitive ETL jobs on Cloud Composer. Management requires alerts when tasks exceed their expected…
• Your team manages critical ETL pipelines running on Cloud Composer. You need to implement alerting so that operations staff receive…

Browse all 11 questions →

Stop Composer overeager scaling by raising minimum workers and worker_concurrency

When many short tasks finish on existing workers before freshly autoscaled workers warm up, the new workers sit idle and waste money. Raise the minimum worker count and tune [celery]worker_concurrency so existing workers absorb the queue instead of triggering new workers; a persistently high queued-task count with the environment pinned at its worker maximum instead means too little capacity, so raise the maximum workers.

Trap Lowering the maximum worker count to curb idle workers, which starves real bursts of capacity rather than fixing warm-up lag.

2 questions test this

• Your organization runs a Cloud Composer 2 environment with multiple DAGs that execute many short-lived tasks. You observe that autoscaling…
• You manage a Cloud Composer 2 environment that runs many quick, small tasks in parallel. You observe in the Monitoring dashboard that…

Composer tasks failing with no logs and SIGTERM mean worker OOM

Tasks marked Failed with no logs, alongside 'Warm shutdown' and 'Received SIGTERM' in worker logs, indicate the worker pod was evicted for exceeding memory (logs were buffered and lost). Give workers more memory, or reduce [celery]worker_concurrency so each task has more memory and raise maximum workers to keep throughput.

Trap Treating logless task failures as code bugs, when SIGTERM and warm-shutdown messages point to a memory-evicted worker.

1 question tests this

• Your Cloud Composer 2 environment is experiencing frequent Airflow worker pod evictions. The Cloud Monitoring dashboard shows that Total…

When Dataflow won't scale up despite backlog, read the autoscaling rationale and check quota

The Autoscaling rationale chart explains why the autoscaler scaled up, down, or held steady; for Streaming Engine, downscaling targets 75% CPU and is disabled when that can't be met. If workers are below maxNumWorkers with a growing backlog, the usual cause is hitting Compute Engine CPU or external-IP quota; if already at the maximum, raise maxNumWorkers with an in-flight update.

Trap Assuming a stalled scale-up is a code bottleneck, when the rationale chart and project quotas usually reveal the real cause.

2 questions test this

• Your Dataflow streaming pipeline using Streaming Engine is not scaling down despite having low backlog and low CPU utilization consistently…
• Your streaming Dataflow job uses Streaming Engine and reads from Pub/Sub. You notice the job is not scaling up despite a growing backlog…

Rising data freshness with normal backlog points to stuck work, not the source

When streaming data freshness climbs but backlog bytes stay normal, work items are stuck inside the pipeline rather than arriving too fast. Check the Parallel processing chart and processing_parallelism_keys for insufficient key parallelism (especially when CPU is low and even across workers), and scan worker logs for elements failing and retrying repeatedly, which holds the watermark back without a bottleneck alert.

Trap Scaling up workers when freshness rises with normal backlog, when the cause is stuck keys or retry loops, not throughput.

2 questions test this

• Your team operates a Dataflow streaming pipeline that processes customer transaction data in real-time. The pipeline has been running…
• Your streaming Dataflow pipeline reads from Pub/Sub and writes to BigQuery. Users report that dashboard visualizations are showing stale…