Domain 6 of 6 · Chapter 6 of 6

Production Reliability

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Professional Cloud Architect premium course — no separate purchase.

Included in this chapter:

  • Reliability you can prove, not reliability you hope for
  • Chaos engineering and fault injection
  • Load and stress testing for capacity
  • Recovery testing, game days, and pen-test policy
  • Exam-pattern recognition

Which resilience-validation technique proves which reliability property

TechniqueReliability property it provesWhat you injectPass/fail measured against
Chaos engineering / fault injectionGraceful degradation and redundancyKilled instances, zones, or backends; injected latency or errors; misconfigurationsGolden signals stay inside the SLO error budget; no cascading failure
Load and stress testingHorizontal scalability and capacityTraffic beyond expected peak, escalated to a breaking pointAutoscaling responds in time; latency and error rate hold to target
Disaster-recovery testing / game daysRecovery from failure and data lossReal regional failover, release rollback, restore from backupRecovery time meets RTO; data loss meets RPO
Penetration testingSecurity of the controls that keep the system availableSimulated attacker activity against your own projectsNo exploitable path; runs under the AUP, no Google pre-authorization

Decision tree

What does the scenariodoubt?Penetration testingunder AUP, no pre-authsecurity of my deploymentCapacity under atraffic surge?elseLoad and stress testingdoes autoscaling keep up?yes, capacityRecovery time or dataloss vs RTO/RPO?noDR drill / game dayreal failover + restoreyes, recoveryChaos engineering /fault injectionno, a part failingAlways: measure the result against the SLO,RTO, and RPO using Cloud Monitoring
Cheat sheet

Cheat sheet

  • Reliability is proven by testing, not assumed from a redundant design
  • Match the resilience technique to the property the scenario doubts
  • Chaos engineering runs a controlled experiment, not random destruction
  • Escalate chaos blast radius gradually: instance, then zone, then region
  • Test misconfigurations and cascading failures, not just hardware faults
  • Run chaos and recovery tests in production or a production-like environment
  • Load testing proves capacity; stress testing finds the breaking point
  • A load test really tests autoscaling reaction time and where saturation hits first
  • Judge every reliability test against the SLO error budget, not a gut feel
  • A DR drill measures real recovery against RTO and RPO
  • A backup is unproven until you have restored it; set frequency to the RPO
  • Tighter RTO/RPO moves you up the cold-warm-hot DR scale and costs more
  • A game day rehearses the people and runbook, not just the technology
  • You can pen-test your own Google Cloud resources with no pre-authorization
  • Reliability validation consumes observability and SLOs; it does not build them
  • A snapshot-schedule resource policy meets RPO by frequency and survives disk deletion by its delete rule
  • Zero-downtime MIG rolling updates use maxUnavailable=0 with maxSurge>0

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Compute Engine: Instance groups (MIG)
  2. Well-Architected Framework: Reliability pillar principles Well-Architected
  3. Well-Architected Framework: Perform testing for recovery from failures Well-Architected
  4. Site Reliability Engineering: Monitoring distributed systems (the four golden signals)
  5. Compute Engine: Autoscaling groups of instances
  6. Architecture Center: Disaster recovery planning guide Well-Architected
  7. Well-Architected Framework: Perform testing for recovery from data loss Well-Architected
  8. Cloud Security FAQ: penetration testing (no pre-authorization) FAQ
  9. Google Cloud Platform Acceptable Use Policy