Cloud Observability
The suite and the four telemetry types
A user reports the checkout page is slow. Where do you look first? The answer depends on the kind of evidence you need, and Google Cloud Observability is organized around exactly that distinction. It is a single suite (its documentation still lives under the legacy Stackdriver[1] name, which is the rename you should expect on exam wording) that gathers four kinds of telemetry plus a derived view, each answered by its own product.
The three primary signals are metrics, logs, and traces, the industry-standard pillars of observability. A metric is a numeric measurement sampled over time, such as requests per second or CPU utilization; Cloud Monitoring[2] owns metrics, dashboards, uptime checks, and alerting. A log is a timestamped record of a discrete event (an error, an admin action, an HTTP request line); Cloud Logging[3] ingests, stores, and routes logs. A trace records the path of a single request as it fans out across services, broken into spans (one timed unit of work each); Cloud Trace[4] reconstructs that path so you can see which service ate the latency.
Two more products round out the suite. Cloud Profiler[5] continuously samples CPU and heap usage in production and attributes it to source code, answering "which function is expensive" rather than "is the service slow." Error Reporting[6] reads log entries, recognizes stack traces, and groups recurring crashes into counted issues, answering "how often is this exception firing."
The architect's skill here is routing the question to the signal, not memorizing every API. "Is it up and how full?" is a metric. "What happened on this exact request?" is a log. "Where did the time go across five services?" is a trace. "Which line of code burns the CPU?" is a profile. "Which exception keeps recurring?" is Error Reporting. The exam rarely asks you to configure a product; it asks you to pick the right one for the symptom.
Cloud Logging: the Log Router, buckets, and retention
Every log entry written to a Google Cloud project passes through one place first: the Log Router. The Router evaluates sinks, and each sink has an inclusion filter that decides which entries it captures and a destination it writes them to. This is the single mental model for logging: logs do not "land" anywhere by themselves, the Router copies matching entries to wherever a sink points.
A sink can write to four kinds of destination, and choosing among them is a frequent exam decision:
| Destination | Use it for |
|---|---|
| Log bucket (Cloud Logging storage) | Default keep-and-search; retention configured per bucket |
| BigQuery dataset | SQL analysis and joins across logs |
| Cloud Storage bucket | Cheap long-term archive / compliance hold |
| Pub/Sub topic | Streaming to third-party SIEM or custom processing |
Retention is a property of the log bucket, not of the log entry, and the three buckets behave differently. Every project starts with two reserved buckets. The _Required bucket[7] stores Admin Activity and other required audit logs for 400 days, and that period is not configurable: it is fixed by Google. The _Default bucket holds everything else and keeps it for 30 days by default, which you can change. For a user-defined bucket[8] (or the _Default bucket in a project), you can set retention anywhere from 1 day to 3650 days (ten years). A common point of confusion: a longer _Default retention does not move audit logs, which already sit in _Required; to keep audit logs beyond 400 days you route them via a sink to a custom bucket or to Cloud Storage.
Logs also feed metrics. A log-based metric[9] turns matching log entries into a Cloud Monitoring time series. A counter metric counts entries that match a filter (for example, lines containing a specific error string). A distribution metric extracts a number from each matching entry and accumulates a histogram (for example, parsing a latency value out of each request log). Once a log-based metric exists, you alert on it exactly like any other metric, which is how you page on "more than N of this error per minute" without building a custom pipeline.
SLIs, SLOs, error budgets, and alerting that fires on symptoms
Reliability is not "keep it at 100%"; it is a number you choose below 100% and then spend deliberately. That single idea drives the whole SRE side of this exam, and Cloud Monitoring implements it directly.
Start with the vocabulary, defined once. A service level indicator (SLI) is a quantitative measurement of one aspect of service behavior, typically a ratio of good events to total events (successful requests over all requests, or fast requests over all requests). A service level objective (SLO) is a target value for an SLI over a time window, expressed in "nines" and set below 100% on purpose. A service level agreement (SLA) is a contract that promises an SLO to a customer and carries a consequence (a refund or credit) if breached; the test for whether something is an SLA is "what happens if it is missed." An SLA is therefore looser than the internal SLO that protects it. These distinctions are straight from Google SRE[10].
The error budget falls out of the SLO arithmetic: it is (1 − SLO) × eligible events in the compliance period. A 99.9% SLO over a million eligible requests yields a budget of 1,000 failures. Per Google SRE[11], the budget is the acceptable amount of unreliability a team may spend on shipping features and taking risks until it is exhausted, at which point the policy is to freeze risky launches and pour effort into stability. This is the lever a case study is reaching for when it asks how to balance feature velocity against reliability.
Cloud Monitoring's service monitoring[12] computes SLOs and budgets for you. A compliance period is either rolling (a continuous trailing window of 1 to 30 days, where the oldest data drops out each day) or calendar-based (a fixed week or month that resets on a boundary). SLOs come in two shapes: request-based (the ratio of good to total requests, e.g. "latency under 100 ms for 95% of requests") and windows-based (the ratio of good time windows to total, e.g. "the 95th-percentile latency is under 100 ms for 99% of 10-minute windows").
The payoff is burn-rate alerting. Instead of paging on a raw CPU threshold (a cause the user never feels), you alert when the error budget is being consumed faster than the SLO allows (a symptom that maps to real pain). This is the antidote to alert fatigue and the reason to anchor alerts to the four golden signals from Google SRE[13]: latency (time to serve a request, tracked separately for successes and errors), traffic (demand on the system), errors (rate of failed requests), and saturation (how full the most-constrained resource is). When a stem describes pagers going off constantly with no user impact, the fix is symptom-based, SLO-anchored alerting, not more thresholds.
Exam-pattern recognition: matching the stem to the product
PCA questions on observability are almost always "choose the right service" or "choose the right design," so train the pattern rather than the configuration detail.
Signal-selection stems
When a stem describes latency that crosses service boundaries ("a request touches the API, two backends, and a database; find where the time goes"), the answer is Cloud Trace with its spans, not a Monitoring dashboard. A dashboard shows that the service is slow; only trace spans show where. If the stem instead says the service is slow but you cannot reproduce it locally and need to know which function burns CPU or allocates memory, the answer is Cloud Profiler, which runs continuously in production at under 5% overhead. When the stem is "the same exception keeps crashing the app and we want it grouped and counted automatically," that is Error Reporting, which groups by exception type and the top stack frames straight from the logs, no extra instrumentation.
Routing and retention stems
When logs must be queried with SQL or joined with other data, the design is a Log Router sink to BigQuery. When logs must be archived cheaply for years or held for compliance, the sink goes to Cloud Storage, or you raise a custom log bucket toward its 3650-day ceiling. When logs must stream into a third-party SIEM, the sink targets Pub/Sub. Watch for the audit-log trap: extending _Default retention does nothing for audit logs, which sit in the fixed-400-day _Required bucket; to keep them longer you must route them out via a sink.
Reliability and alerting stems
When a stem complains that on-call is drowning in pages that do not correspond to user impact, the wrong answers add more metric thresholds; the right answer is symptom-based alerting anchored to an SLO and its error-budget burn rate, using the golden signals. When a case study asks how to decide between shipping features and stabilizing, the answer names the error budget: spend it on features while it lasts, freeze risky launches once it is exhausted. When the requirement is a measurable reliability target across a fleet of projects, combine Cloud Monitoring service monitoring (SLOs) with a metrics scope so one scoping project sees every project's metrics. And when instrumentation longevity matters, prefer OpenTelemetry over product-specific SDKs so the telemetry survives a future tooling change.
Which Observability signal answers which question
| Question being asked | Signal type | Product | Stored / analyzed in |
|---|---|---|---|
| Is the service up and how full is it? | Metric | Cloud Monitoring | Time series, dashboards, uptime checks |
| What exactly happened on this event? | Log | Cloud Logging | Log buckets via the Log Router |
| Where did latency go across services? | Trace | Cloud Trace | Spans, latency heatmap (OpenTelemetry) |
| Which code consumes the most CPU/memory? | Profile | Cloud Profiler | Continuous statistical profiles |
| How often is this exception firing? | Aggregated error | Error Reporting | Grouped from log stack traces |
Decision tree
Sharp facts the exam loves — give these one last read before exam day.
Cheat sheet
Sharp facts the exam loves — scan these before test day.
- Match the symptom to the signal: metric, log, trace, profile, or grouped error
Google Cloud Observability is one suite of five products, each answering a different question. "Is it up and how full" is a metric in Cloud Monitoring; "what happened on this event" is a log in Cloud Logging; "where did latency go across services" is a trace in Cloud Trace; "which function burns CPU or memory" is a profile in Cloud Profiler; "how often is this exception firing" is Error Reporting. On the exam the skill is routing the scenario to the right product, not configuring it.
- Cloud Observability is the rebranded Stackdriver suite
Google Cloud Observability is the current name for what was branded Stackdriver; the documentation still lives under the cloud.google.com/stackdriver path. Stems may use either name for the same metrics, logging, and tracing suite, so treat Stackdriver Monitoring/Logging as today's Cloud Monitoring/Cloud Logging.
- Cloud Trace answers where latency goes across services, not whether a service is slow
When a request fans out across microservices and you need to find which hop ate the time, use Cloud Trace: it records the request as a trace of spans (one timed unit of work each) and shows aggregate latency on a heatmap. A Cloud Monitoring dashboard tells you the service is slow; only trace spans tell you where, so reach for Trace whenever the scenario crosses service boundaries.
Trap Reaching for a Cloud Monitoring latency dashboard to locate the slow hop; it shows that the service is slow in aggregate but not which downstream call is responsible.
12 questions test this
- Your company has deployed a microservices-based e-commerce application on Google Kubernetes Engine (GKE). Users are reporting intermittent…
- Your SRE team is investigating an incident where users report intermittent slow responses from your microservices application deployed on…
- Cymbal Retail is implementing SRE practices for their order processing system running on GKE. The system must meet a latency SLO where 95%…
- Your company operates a microservices-based e-commerce platform on Google Kubernetes Engine (GKE) with 15 services. Users are experiencing…
- Your development team deployed a Go microservice on GKE that processes financial transactions. After analyzing Cloud Profiler data, you…
- Cymbal Retail operates a microservices-based e-commerce platform on GKE. SRE teams report that customers experience intermittent high…
- Cymbal Retail is experiencing intermittent slowdowns in their checkout service running on Cloud Run. The SRE team needs to identify which…
- Your organization runs a distributed e-commerce application instrumented with OpenTelemetry. During a checkout service investigation using…
- Your company operates a microservices-based e-commerce platform running on GKE. Users are reporting intermittent slow response times during…
- Your organization has deployed a microservices application on Google Kubernetes Engine (GKE) where services communicate via HTTP and gRPC.…
- Your company runs a microservices application on GKE. Users report intermittent slowness, but overall average latency metrics appear…
- Your team manages a distributed e-commerce platform and has defined latency-based SLOs using Cloud Monitoring. After a deployment, you…
- Cloud Profiler finds CPU and memory hotspots in production at under 5% overhead
When a service is CPU- or memory-bound and you cannot reproduce it locally, attach Cloud Profiler: it continuously samples production using statistical profiling and attributes CPU and heap usage to source-code lines. Collection overhead is under 5%, and amortized across replicas commonly under 0.5%, so it is safe to leave on always. It supports Go, Java, Node.js, and Python.
5 questions test this
- A Java application running on Compute Engine is experiencing higher than expected CPU utilization, causing increased costs. You need to…
- Your SRE team notices that a Go-based API service deployed on Cloud Run is experiencing gradually increasing response times after each…
- Your SRE team has identified that a Java application running on Compute Engine is experiencing high CPU utilization during peak hours,…
- Altostrat Media is running a video processing pipeline on GKE. The SRE team notices that some requests take significantly longer than…
- A company running a Python application on GKE wants to implement continuous profiling in production to proactively identify performance…
- Error Reporting groups recurring crashes from logs automatically
When the same exception keeps crashing an app and you want it counted and grouped without extra instrumentation, use Error Reporting: it reads log entries, recognizes stack traces, and groups errors by exception type and the top stack frames, then can notify you of new ones. It samples up to 1,000 errors per hour and estimates counts beyond that, so very high-volume counts are approximate.
Trap Building a custom log-based counter to detect duplicate exceptions; Error Reporting already groups by exception type and top frames out of the box.
3 questions test this
- Your organization uses Error Reporting to monitor application errors across multiple services running on Compute Engine. The operations…
- Your company runs a microservices application on Cloud Run. The development team needs to receive immediate notifications when new types of…
- Your team runs a microservices-based application on Cloud Run and needs to implement an incident response workflow. You want to be notified…
- Every log entry passes through the Log Router before it is stored
Logs do not land anywhere by themselves: the Log Router evaluates sinks, and each sink has an inclusion filter plus a destination. A sink can write to a log bucket, a BigQuery dataset, a Cloud Storage bucket, or a Pub/Sub topic. This single model is the answer to most routing questions: to send logs somewhere new you create or edit a sink, not a per-log setting.
- Pick the sink destination by purpose: BigQuery for SQL, Storage for archive, Pub/Sub for streaming
When logs must be queried with SQL or joined with other data, route a sink to BigQuery. When they must be archived cheaply for years or held for compliance, route to a Cloud Storage bucket. When they must stream into a third-party SIEM or custom pipeline, route to a Pub/Sub topic. The default keep-and-search destination is a Cloud Logging log bucket.
- The _Required log bucket keeps audit logs 400 days and cannot be changed
Admin Activity and other required audit logs go to the _Required bucket, which retains them for 400 days, a period fixed by Google and not configurable. Everything else lands in the _Default bucket at 30 days by default, which you can change. Knowing the 400-day fixed value is the testable specific behind most audit-retention questions.
Trap Assuming you can shorten or extend the _Required bucket's 400-day retention; it is fixed, so to keep audit logs longer you route them out via a sink.
- Custom log buckets retain between 1 and 3650 days
A user-defined log bucket (and the _Default bucket in a project) can have its retention set anywhere from 1 day to 3650 days (ten years). Raise it toward 3650 for compliance holds, or lower it to control cost. The _Default starts at 30 days; only custom or _Default project buckets are configurable, while folder- and organization-level _Default buckets are not.
- Extending _Default retention does not move audit logs
Audit logs sit in the fixed-400-day _Required bucket, so changing the _Default bucket's retention does nothing for them. To keep audit logs beyond 400 days you must route them via a sink to a custom log bucket (up to 3650 days) or to Cloud Storage. This is the classic audit-retention trap on the exam.
Trap Raising _Default retention to keep audit logs longer; audit logs live in _Required, so they are unaffected and still expire at 400 days.
- Log-based metrics turn matching log entries into Cloud Monitoring time series
A log-based metric converts log entries that match a filter into a metric you can chart and alert on like any other. A counter metric counts matching entries (for example, lines containing a specific error). A distribution metric extracts a number from each entry and builds a histogram (for example, parsing a latency value). This is how you page on "more than N of this error per minute" without a custom pipeline.
4 questions test this
- Your company requires that the security operations team receives immediate notifications when specific security events are logged, such as…
- Your team at Cymbal Retail needs to track request latency patterns from application logs that contain response times in the format 'Request…
- An application running on Compute Engine generates structured JSON logs that include custom latency values for each API request. You need…
- Your organization runs a critical e-commerce application on Google Cloud. The operations team needs to track API response latency values…
- An alerting policy is a condition plus notification channels plus documentation
A Cloud Monitoring alerting policy bundles a condition (the metric, threshold, and aggregation), one or more notification channels (email, PagerDuty, Slack, Pub/Sub, Cloud Mobile App), and documentation that appears in the notification. When the condition is met it opens an incident and notifies the channels. Notification channels are reusable objects you attach to many policies.
- Alert on user-felt symptoms anchored to an SLO, not on every internal cause
Page on the four golden signals that map to user pain (latency, traffic, errors, saturation), not on raw causes like a single CPU spike. Tie the alert threshold to an SLO so a page means a real error-budget burn, which directly cures alert fatigue. Cause-level metrics belong on dashboards for diagnosis, not on pagers. This is the answer when a stem describes on-call drowning in pages with no user impact.
Trap Adding more per-resource metric thresholds to fix alert fatigue; that multiplies cause-level noise, while SLO-anchored symptom alerts reduce it.
- The four golden signals are latency, traffic, errors, and saturation
Google SRE's four golden signals are latency (time to serve a request, tracked separately for successes and errors), traffic (demand on the system), errors (rate of failed requests), and saturation (how full the most-constrained resource is). They are the default set of things to monitor and alert on for any user-facing service, and the SRE-flavored questions expect you to name all four.
- An SLI is a measurement, an SLO is a target you set below 100%
An SLI (service level indicator) is a quantitative measurement of one aspect of service behavior, usually a ratio of good events to total. An SLO (service level objective) is a target value for that SLI over a window, expressed in nines and set below 100% on purpose because perfect uptime is unattainable and not worth its cost. Cloud Monitoring service monitoring computes SLOs against the SLI.
- An SLA is a contract with a penalty; it is looser than the internal SLO
An SLA (service level agreement) promises an SLO to a customer and carries a consequence such as a refund or credit if breached; the test for whether something is an SLA is "what happens if it is missed." Because a breach costs money, the SLA target is deliberately set looser than the internal SLO that protects it, giving the team headroom before the contract is at risk.
Trap Treating the SLA and the internal SLO as the same number; the SLA is set looser so the team has margin before a contractual penalty triggers.
- Error budget is (1 minus the SLO) times eligible events
The error budget is the allowed failure margin that falls out of the SLO: (1 minus the SLO) times the eligible events in the compliance period. A 99.9% SLO over a million requests yields a budget of 1,000 failures. It is the acceptable amount of unreliability a team may spend on shipping features and taking risk; while budget remains the team ships, once it is exhausted the policy freezes risky launches and pours effort into stability. This is the lever for balancing velocity against reliability.
- Compliance periods are rolling (1 to 30 days) or calendar-based
A Cloud Monitoring SLO measures over a compliance period that is either rolling (a continuous trailing window of 1 to 30 days where the oldest data drops out each day) or calendar-based (a fixed week or month that resets on a boundary). Rolling gives a continuously updating view; calendar matches reporting cycles. Pick rolling for steady operational alerting, calendar when the SLO must align with monthly business reporting.
- Request-based SLOs measure request ratios; windows-based SLOs measure good-window ratios
A request-based SLO is the ratio of good requests to total (for example, latency under 100 ms for 95% of requests). A windows-based SLO is the ratio of good time windows to total (for example, the 95th-percentile latency is under 100 ms for 99% of 10-minute windows). Windows-based smooths over bursty traffic and is the choice when you care that the service was healthy over intervals rather than per individual request.
6 questions test this
- Cymbal Retail is implementing SLO monitoring for their product catalog API deployed on GKE. They want to measure that 99% of requests…
- You are designing SLIs for a web application running on GKE. The application serves API requests with varying latency requirements. Your…
- Your organization operates a Cloud Run service that handles customer checkout transactions. The service must respond to 99% of requests…
- Your development team runs a microservices application on Google Kubernetes Engine and wants to track service latency using SLOs. The…
- EHR Healthcare is deploying a patient records API and needs to establish SLIs that measure user experience. The API returns health data to…
- Your company runs a customer-facing e-commerce application on Google Kubernetes Engine (GKE). The operations team needs to implement SLO…
- Burn-rate alerts page when the error budget drains too fast
A burn-rate alert fires when the error budget is being consumed faster than the SLO allows, rather than at a fixed metric threshold. This makes the page mean a real, accelerating reliability problem the user feels, and lets a slow steady burn pass without paging. It is the SLO-native form of symptom-based alerting and the recommended alerting strategy in Cloud Monitoring service monitoring.
7 questions test this
- Your organization operates a Cloud Run service that handles customer checkout transactions. The service must respond to 99% of requests…
- Altostrat Media has a streaming service with a 99.5% availability SLO measured over a 30-day rolling compliance period. The operations team…
- KnightMotives Automotive is implementing SRE practices for their connected vehicle platform running on Cloud Run. The platform team needs…
- Your company operates a customer-facing API deployed on Cloud Run. You have defined an SLO requiring 99.5% availability over a rolling…
- Altostrat Media operates a video streaming service with an external HTTP(S) load balancer. They have defined a 99.5% availability SLO with…
- Your company runs a customer-facing e-commerce application on Google Kubernetes Engine (GKE). The operations team needs to implement SLO…
- Altostrat Media operates a video streaming service and uses Cloud Monitoring to track SLOs. They have configured an SLO with a 99%…
- A metrics scope lets one project see metrics from many projects and AWS accounts
To centralize observability across an estate, configure a Cloud Monitoring metrics scope: a scoping project can view metrics from many Google Cloud projects, and even AWS accounts, on shared dashboards. This is the standard answer when the requirement is one pane of glass across an organization's projects, rather than logging into each project separately.
Trap Creating a separate dashboard inside every project and switching between them; a single metrics scope already aggregates all of their metrics in one scoping project.
- Uptime checks probe HTTP, HTTPS, and TCP endpoints from outside
Cloud Monitoring uptime checks probe HTTP, HTTPS, and TCP endpoints from Google's global locations to confirm responsiveness, and you alert when a check fails. They are the answer for blackbox external availability monitoring of a public endpoint, complementing the whitebox metrics the service emits about itself.
- Instrument with OpenTelemetry, not product-specific SDKs
Cloud Trace and the Cloud Observability agents prefer OpenTelemetry, the vendor-neutral instrumentation standard, over product-specific tracing SDKs. Instrumenting with OpenTelemetry (it speaks OTLP) means the same code survives a future tooling or vendor change, which is the recommended choice whenever a design must avoid lock-in to one tracing backend.
5 questions test this
- Your company has deployed a microservices-based e-commerce application on Google Kubernetes Engine (GKE). Users are reporting intermittent…
- Cymbal Retail is implementing SRE practices for their order processing system running on GKE. The system must meet a latency SLO where 95%…
- Your company operates a microservices-based e-commerce platform on Google Kubernetes Engine (GKE) with 15 services. Users are experiencing…
- Your company operates a microservices-based e-commerce platform running on GKE. Users are reporting intermittent slow response times during…
- Your organization has deployed a microservices application on Google Kubernetes Engine (GKE) where services communicate via HTTP and gRPC.…
- A log-based alerting policy notifies on a single matching log entry, no metric required
A log-based alerting policy fires a notification when a log entry matching its query filter appears, evaluating individual entries in near real time without first creating a metric. It is the right tool for rare-but-important events such as a specific error message or audit-log security event that needs immediate per-occurrence notification, and it can extract fields from the entry as labels for context.
Trap Reaching for a log-based metric plus a metric-threshold alert to get per-occurrence notification of a rare event; a metric counts and trends over time, while the log-based alerting policy fires immediate notification that one specific entry occurred.
5 questions test this
- Your e-commerce application running on GKE experiences intermittent performance issues. You need to be notified immediately when a specific…
- Your company requires that the security operations team receives immediate notifications when specific security events are logged, such as…
- EHR Healthcare runs a patient portal application on Compute Engine. The operations team wants to receive immediate notifications whenever a…
- Your company's e-commerce application on GKE is experiencing intermittent 500 errors. The operations team needs to be notified immediately…
- KnightMotives Automotive wants to be notified when specific security events occur in their Google Cloud audit logs, such as when a human…
Also tested in
References
- Google Cloud Observability overview
- Cloud Monitoring overview
- Cloud Logging overview
- Cloud Trace overview
- About Cloud Profiler
- Error Reporting: how errors are grouped
- Cloud Logging quotas and limits (retention)
- Cloud Logging: configure log buckets
- Cloud Logging: log-based metrics
- SRE fundamentals: SLIs, SLOs and SLAs Blog
- Google SRE Workbook: error budget policy Whitepaper
- SLO monitoring in Cloud Monitoring
- Google SRE Book: monitoring distributed systems (four golden signals) Whitepaper