Domain 6 of 6 · Chapter 2 of 6

Cloud Observability

The suite and the four telemetry types

A user reports the checkout page is slow. Where do you look first? The answer depends on the kind of evidence you need, and Google Cloud Observability is organized around exactly that distinction. It is a single suite (its documentation still lives under the legacy Stackdriver[1] name, which is the rename you should expect on exam wording) that gathers four kinds of telemetry plus a derived view, each answered by its own product.

The three primary signals are metrics, logs, and traces, the industry-standard pillars of observability. A metric is a numeric measurement sampled over time, such as requests per second or CPU utilization; Cloud Monitoring[2] owns metrics, dashboards, uptime checks, and alerting. A log is a timestamped record of a discrete event (an error, an admin action, an HTTP request line); Cloud Logging[3] ingests, stores, and routes logs. A trace records the path of a single request as it fans out across services, broken into spans (one timed unit of work each); Cloud Trace[4] reconstructs that path so you can see which service ate the latency.

Two more products round out the suite. Cloud Profiler[5] continuously samples CPU and heap usage in production and attributes it to source code, answering "which function is expensive" rather than "is the service slow." Error Reporting[6] reads log entries, recognizes stack traces, and groups recurring crashes into counted issues, answering "how often is this exception firing."

The architect's skill here is routing the question to the signal, not memorizing every API. "Is it up and how full?" is a metric. "What happened on this exact request?" is a log. "Where did the time go across five services?" is a trace. "Which line of code burns the CPU?" is a profile. "Which exception keeps recurring?" is Error Reporting. The exam rarely asks you to configure a product; it asks you to pick the right one for the symptom.

Cloud Observabilityone suiteCloud MonitoringMetric: up & full?Cloud LoggingLog: what happened?Cloud TraceTrace: where slow?Cloud ProfilerProfile: which code?ErrorReportingPick the signal that matches the symptom, not every API
Google Cloud Observability: five products, each the answer to one kind of question.

Cloud Logging: the Log Router, buckets, and retention

Every log entry written to a Google Cloud project passes through one place first: the Log Router. The Router evaluates sinks, and each sink has an inclusion filter that decides which entries it captures and a destination it writes them to. This is the single mental model for logging: logs do not "land" anywhere by themselves, the Router copies matching entries to wherever a sink points.

A sink can write to four kinds of destination, and choosing among them is a frequent exam decision:

Destination Use it for
Log bucket (Cloud Logging storage) Default keep-and-search; retention configured per bucket
BigQuery dataset SQL analysis and joins across logs
Cloud Storage bucket Cheap long-term archive / compliance hold
Pub/Sub topic Streaming to third-party SIEM or custom processing

Retention is a property of the log bucket, not of the log entry, and the three buckets behave differently. Every project starts with two reserved buckets. The _Required bucket[7] stores Admin Activity and other required audit logs for 400 days, and that period is not configurable: it is fixed by Google. The _Default bucket holds everything else and keeps it for 30 days by default, which you can change. For a user-defined bucket[8] (or the _Default bucket in a project), you can set retention anywhere from 1 day to 3650 days (ten years). A common point of confusion: a longer _Default retention does not move audit logs, which already sit in _Required; to keep audit logs beyond 400 days you route them via a sink to a custom bucket or to Cloud Storage.

Logs also feed metrics. A log-based metric[9] turns matching log entries into a Cloud Monitoring time series. A counter metric counts entries that match a filter (for example, lines containing a specific error string). A distribution metric extracts a number from each matching entry and accumulates a histogram (for example, parsing a latency value out of each request log). Once a log-based metric exists, you alert on it exactly like any other metric, which is how you page on "more than N of this error per minute" without building a custom pipeline.

Log entriesLog Routersinks + filtersLog bucket_Required 400d fixed | _Default 30d | 1-3650dBigQuerySQL analysisCloud Storagecheap archivePub/Substream to SIEM
Cloud Logging: one Log Router, sinks to four destinations, retention configured on the log bucket.

SLIs, SLOs, error budgets, and alerting that fires on symptoms

Reliability is not "keep it at 100%"; it is a number you choose below 100% and then spend deliberately. That single idea drives the whole SRE side of this exam, and Cloud Monitoring implements it directly.

Start with the vocabulary, defined once. A service level indicator (SLI) is a quantitative measurement of one aspect of service behavior, typically a ratio of good events to total events (successful requests over all requests, or fast requests over all requests). A service level objective (SLO) is a target value for an SLI over a time window, expressed in "nines" and set below 100% on purpose. A service level agreement (SLA) is a contract that promises an SLO to a customer and carries a consequence (a refund or credit) if breached; the test for whether something is an SLA is "what happens if it is missed." An SLA is therefore looser than the internal SLO that protects it. These distinctions are straight from Google SRE[10].

The error budget falls out of the SLO arithmetic: it is (1 − SLO) × eligible events in the compliance period. A 99.9% SLO over a million eligible requests yields a budget of 1,000 failures. Per Google SRE[11], the budget is the acceptable amount of unreliability a team may spend on shipping features and taking risks until it is exhausted, at which point the policy is to freeze risky launches and pour effort into stability. This is the lever a case study is reaching for when it asks how to balance feature velocity against reliability.

Cloud Monitoring's service monitoring[12] computes SLOs and budgets for you. A compliance period is either rolling (a continuous trailing window of 1 to 30 days, where the oldest data drops out each day) or calendar-based (a fixed week or month that resets on a boundary). SLOs come in two shapes: request-based (the ratio of good to total requests, e.g. "latency under 100 ms for 95% of requests") and windows-based (the ratio of good time windows to total, e.g. "the 95th-percentile latency is under 100 ms for 99% of 10-minute windows").

The payoff is burn-rate alerting. Instead of paging on a raw CPU threshold (a cause the user never feels), you alert when the error budget is being consumed faster than the SLO allows (a symptom that maps to real pain). This is the antidote to alert fatigue and the reason to anchor alerts to the four golden signals from Google SRE[13]: latency (time to serve a request, tracked separately for successes and errors), traffic (demand on the system), errors (rate of failed requests), and saturation (how full the most-constrained resource is). When a stem describes pagers going off constantly with no user impact, the fix is symptom-based, SLO-anchored alerting, not more thresholds.

SLIgood / totalSLO targete.g. 99.9%, below 100%Error budget(1 - SLO) x eventsBurn-rate alertbudget draining too fastPage on symptomuser pain, not CPUAnchor to golden signals: latency | traffic | errors | saturation
SLI to SLO to error budget to burn-rate alert: page on user-felt symptoms, not internal causes.

Exam-pattern recognition: matching the stem to the product

PCA questions on observability are almost always "choose the right service" or "choose the right design," so train the pattern rather than the configuration detail.

Signal-selection stems

When a stem describes latency that crosses service boundaries ("a request touches the API, two backends, and a database; find where the time goes"), the answer is Cloud Trace with its spans, not a Monitoring dashboard. A dashboard shows that the service is slow; only trace spans show where. If the stem instead says the service is slow but you cannot reproduce it locally and need to know which function burns CPU or allocates memory, the answer is Cloud Profiler, which runs continuously in production at under 5% overhead. When the stem is "the same exception keeps crashing the app and we want it grouped and counted automatically," that is Error Reporting, which groups by exception type and the top stack frames straight from the logs, no extra instrumentation.

Routing and retention stems

When logs must be queried with SQL or joined with other data, the design is a Log Router sink to BigQuery. When logs must be archived cheaply for years or held for compliance, the sink goes to Cloud Storage, or you raise a custom log bucket toward its 3650-day ceiling. When logs must stream into a third-party SIEM, the sink targets Pub/Sub. Watch for the audit-log trap: extending _Default retention does nothing for audit logs, which sit in the fixed-400-day _Required bucket; to keep them longer you must route them out via a sink.

Reliability and alerting stems

When a stem complains that on-call is drowning in pages that do not correspond to user impact, the wrong answers add more metric thresholds; the right answer is symptom-based alerting anchored to an SLO and its error-budget burn rate, using the golden signals. When a case study asks how to decide between shipping features and stabilizing, the answer names the error budget: spend it on features while it lasts, freeze risky launches once it is exhausted. When the requirement is a measurable reliability target across a fleet of projects, combine Cloud Monitoring service monitoring (SLOs) with a metrics scope so one scoping project sees every project's metrics. And when instrumentation longevity matters, prefer OpenTelemetry over product-specific SDKs so the telemetry survives a future tooling change.

What does the stem ask?where slow?which code?recurring crash?where to send logs?Cloud TraceCloud ProfilerError ReportingSink to...by purposeSQLarchiveBigQueryCloud StoragePages with no user impact? SLO + error-budget burn-rate alertanchor on golden signals, page on symptoms
Stem-to-product decision tree for PCA observability scenarios.

Which Observability signal answers which question

Question being askedSignal typeProductStored / analyzed in
Is the service up and how full is it?MetricCloud MonitoringTime series, dashboards, uptime checks
What exactly happened on this event?LogCloud LoggingLog buckets via the Log Router
Where did latency go across services?TraceCloud TraceSpans, latency heatmap (OpenTelemetry)
Which code consumes the most CPU/memory?ProfileCloud ProfilerContinuous statistical profiles
How often is this exception firing?Aggregated errorError ReportingGrouped from log stack traces

Decision tree

What is the question?pick the signalup & how full?recurring crash?where slow?store logs?Cloud Monitoringmetrics, uptime, alertsError Reportinggroups by stack traceCloud Tracespans / heatmapLog Router sinkby purposenoisy pages?CPU/mem hotspot?SLO + budget burnalert on symptomsCloud Profiler<5% overheadSQLarchiveCloud StorageBigQueryAlways: instrument with OpenTelemetry; metrics scope across projectsgolden signals: latency | traffic | errors | saturation

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Match the symptom to the signal: metric, log, trace, profile, or grouped error

Google Cloud Observability is one suite of five products, each answering a different question. "Is it up and how full" is a metric in Cloud Monitoring; "what happened on this event" is a log in Cloud Logging; "where did latency go across services" is a trace in Cloud Trace; "which function burns CPU or memory" is a profile in Cloud Profiler; "how often is this exception firing" is Error Reporting. On the exam the skill is routing the scenario to the right product, not configuring it.

Cloud Observability is the rebranded Stackdriver suite

Google Cloud Observability is the current name for what was branded Stackdriver; the documentation still lives under the cloud.google.com/stackdriver path. Stems may use either name for the same metrics, logging, and tracing suite, so treat Stackdriver Monitoring/Logging as today's Cloud Monitoring/Cloud Logging.

Cloud Trace answers where latency goes across services, not whether a service is slow

When a request fans out across microservices and you need to find which hop ate the time, use Cloud Trace: it records the request as a trace of spans (one timed unit of work each) and shows aggregate latency on a heatmap. A Cloud Monitoring dashboard tells you the service is slow; only trace spans tell you where, so reach for Trace whenever the scenario crosses service boundaries.

Trap Reaching for a Cloud Monitoring latency dashboard to locate the slow hop; it shows that the service is slow in aggregate but not which downstream call is responsible.

12 questions test this
Cloud Profiler finds CPU and memory hotspots in production at under 5% overhead

When a service is CPU- or memory-bound and you cannot reproduce it locally, attach Cloud Profiler: it continuously samples production using statistical profiling and attributes CPU and heap usage to source-code lines. Collection overhead is under 5%, and amortized across replicas commonly under 0.5%, so it is safe to leave on always. It supports Go, Java, Node.js, and Python.

5 questions test this
Error Reporting groups recurring crashes from logs automatically

When the same exception keeps crashing an app and you want it counted and grouped without extra instrumentation, use Error Reporting: it reads log entries, recognizes stack traces, and groups errors by exception type and the top stack frames, then can notify you of new ones. It samples up to 1,000 errors per hour and estimates counts beyond that, so very high-volume counts are approximate.

Trap Building a custom log-based counter to detect duplicate exceptions; Error Reporting already groups by exception type and top frames out of the box.

3 questions test this
Every log entry passes through the Log Router before it is stored

Logs do not land anywhere by themselves: the Log Router evaluates sinks, and each sink has an inclusion filter plus a destination. A sink can write to a log bucket, a BigQuery dataset, a Cloud Storage bucket, or a Pub/Sub topic. This single model is the answer to most routing questions: to send logs somewhere new you create or edit a sink, not a per-log setting.

Pick the sink destination by purpose: BigQuery for SQL, Storage for archive, Pub/Sub for streaming

When logs must be queried with SQL or joined with other data, route a sink to BigQuery. When they must be archived cheaply for years or held for compliance, route to a Cloud Storage bucket. When they must stream into a third-party SIEM or custom pipeline, route to a Pub/Sub topic. The default keep-and-search destination is a Cloud Logging log bucket.

The _Required log bucket keeps audit logs 400 days and cannot be changed

Admin Activity and other required audit logs go to the _Required bucket, which retains them for 400 days, a period fixed by Google and not configurable. Everything else lands in the _Default bucket at 30 days by default, which you can change. Knowing the 400-day fixed value is the testable specific behind most audit-retention questions.

Trap Assuming you can shorten or extend the _Required bucket's 400-day retention; it is fixed, so to keep audit logs longer you route them out via a sink.

Custom log buckets retain between 1 and 3650 days

A user-defined log bucket (and the _Default bucket in a project) can have its retention set anywhere from 1 day to 3650 days (ten years). Raise it toward 3650 for compliance holds, or lower it to control cost. The _Default starts at 30 days; only custom or _Default project buckets are configurable, while folder- and organization-level _Default buckets are not.

Extending _Default retention does not move audit logs

Audit logs sit in the fixed-400-day _Required bucket, so changing the _Default bucket's retention does nothing for them. To keep audit logs beyond 400 days you must route them via a sink to a custom log bucket (up to 3650 days) or to Cloud Storage. This is the classic audit-retention trap on the exam.

Trap Raising _Default retention to keep audit logs longer; audit logs live in _Required, so they are unaffected and still expire at 400 days.

Log-based metrics turn matching log entries into Cloud Monitoring time series

A log-based metric converts log entries that match a filter into a metric you can chart and alert on like any other. A counter metric counts matching entries (for example, lines containing a specific error). A distribution metric extracts a number from each entry and builds a histogram (for example, parsing a latency value). This is how you page on "more than N of this error per minute" without a custom pipeline.

4 questions test this
An alerting policy is a condition plus notification channels plus documentation

A Cloud Monitoring alerting policy bundles a condition (the metric, threshold, and aggregation), one or more notification channels (email, PagerDuty, Slack, Pub/Sub, Cloud Mobile App), and documentation that appears in the notification. When the condition is met it opens an incident and notifies the channels. Notification channels are reusable objects you attach to many policies.

2 questions test this
Alert on user-felt symptoms anchored to an SLO, not on every internal cause

Page on the four golden signals that map to user pain (latency, traffic, errors, saturation), not on raw causes like a single CPU spike. Tie the alert threshold to an SLO so a page means a real error-budget burn, which directly cures alert fatigue. Cause-level metrics belong on dashboards for diagnosis, not on pagers. This is the answer when a stem describes on-call drowning in pages with no user impact.

Trap Adding more per-resource metric thresholds to fix alert fatigue; that multiplies cause-level noise, while SLO-anchored symptom alerts reduce it.

The four golden signals are latency, traffic, errors, and saturation

Google SRE's four golden signals are latency (time to serve a request, tracked separately for successes and errors), traffic (demand on the system), errors (rate of failed requests), and saturation (how full the most-constrained resource is). They are the default set of things to monitor and alert on for any user-facing service, and the SRE-flavored questions expect you to name all four.

An SLI is a measurement, an SLO is a target you set below 100%

An SLI (service level indicator) is a quantitative measurement of one aspect of service behavior, usually a ratio of good events to total. An SLO (service level objective) is a target value for that SLI over a window, expressed in nines and set below 100% on purpose because perfect uptime is unattainable and not worth its cost. Cloud Monitoring service monitoring computes SLOs against the SLI.

An SLA is a contract with a penalty; it is looser than the internal SLO

An SLA (service level agreement) promises an SLO to a customer and carries a consequence such as a refund or credit if breached; the test for whether something is an SLA is "what happens if it is missed." Because a breach costs money, the SLA target is deliberately set looser than the internal SLO that protects it, giving the team headroom before the contract is at risk.

Trap Treating the SLA and the internal SLO as the same number; the SLA is set looser so the team has margin before a contractual penalty triggers.

Error budget is (1 minus the SLO) times eligible events

The error budget is the allowed failure margin that falls out of the SLO: (1 minus the SLO) times the eligible events in the compliance period. A 99.9% SLO over a million requests yields a budget of 1,000 failures. It is the acceptable amount of unreliability a team may spend on shipping features and taking risk; while budget remains the team ships, once it is exhausted the policy freezes risky launches and pours effort into stability. This is the lever for balancing velocity against reliability.

1 question tests this
Compliance periods are rolling (1 to 30 days) or calendar-based

A Cloud Monitoring SLO measures over a compliance period that is either rolling (a continuous trailing window of 1 to 30 days where the oldest data drops out each day) or calendar-based (a fixed week or month that resets on a boundary). Rolling gives a continuously updating view; calendar matches reporting cycles. Pick rolling for steady operational alerting, calendar when the SLO must align with monthly business reporting.

1 question tests this
Request-based SLOs measure request ratios; windows-based SLOs measure good-window ratios

A request-based SLO is the ratio of good requests to total (for example, latency under 100 ms for 95% of requests). A windows-based SLO is the ratio of good time windows to total (for example, the 95th-percentile latency is under 100 ms for 99% of 10-minute windows). Windows-based smooths over bursty traffic and is the choice when you care that the service was healthy over intervals rather than per individual request.

6 questions test this
Burn-rate alerts page when the error budget drains too fast

A burn-rate alert fires when the error budget is being consumed faster than the SLO allows, rather than at a fixed metric threshold. This makes the page mean a real, accelerating reliability problem the user feels, and lets a slow steady burn pass without paging. It is the SLO-native form of symptom-based alerting and the recommended alerting strategy in Cloud Monitoring service monitoring.

7 questions test this
A metrics scope lets one project see metrics from many projects and AWS accounts

To centralize observability across an estate, configure a Cloud Monitoring metrics scope: a scoping project can view metrics from many Google Cloud projects, and even AWS accounts, on shared dashboards. This is the standard answer when the requirement is one pane of glass across an organization's projects, rather than logging into each project separately.

Trap Creating a separate dashboard inside every project and switching between them; a single metrics scope already aggregates all of their metrics in one scoping project.

2 questions test this
Uptime checks probe HTTP, HTTPS, and TCP endpoints from outside

Cloud Monitoring uptime checks probe HTTP, HTTPS, and TCP endpoints from Google's global locations to confirm responsiveness, and you alert when a check fails. They are the answer for blackbox external availability monitoring of a public endpoint, complementing the whitebox metrics the service emits about itself.

1 question tests this
Instrument with OpenTelemetry, not product-specific SDKs

Cloud Trace and the Cloud Observability agents prefer OpenTelemetry, the vendor-neutral instrumentation standard, over product-specific tracing SDKs. Instrumenting with OpenTelemetry (it speaks OTLP) means the same code survives a future tooling or vendor change, which is the recommended choice whenever a design must avoid lock-in to one tracing backend.

5 questions test this
A log-based alerting policy notifies on a single matching log entry, no metric required

A log-based alerting policy fires a notification when a log entry matching its query filter appears, evaluating individual entries in near real time without first creating a metric. It is the right tool for rare-but-important events such as a specific error message or audit-log security event that needs immediate per-occurrence notification, and it can extract fields from the entry as labels for context.

Trap Reaching for a log-based metric plus a metric-threshold alert to get per-occurrence notification of a rare event; a metric counts and trends over time, while the log-based alerting policy fires immediate notification that one specific entry occurred.

5 questions test this

Also tested in

References

  1. Google Cloud Observability overview
  2. Cloud Monitoring overview
  3. Cloud Logging overview
  4. Cloud Trace overview
  5. About Cloud Profiler
  6. Error Reporting: how errors are grouped
  7. Cloud Logging quotas and limits (retention)
  8. Cloud Logging: configure log buckets
  9. Cloud Logging: log-based metrics
  10. SRE fundamentals: SLIs, SLOs and SLAs Blog
  11. Google SRE Workbook: error budget policy Whitepaper
  12. SLO monitoring in Cloud Monitoring
  13. Google SRE Book: monitoring distributed systems (four golden signals) Whitepaper