Domain 2 of 5 · Chapter 4 of 4

Monitor and Maintain Models

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Operationalizing Machine Learning and Generative AI Solutions premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Two questions monitoring must answer
  • Detecting model decay: drift and data quality
  • Operational health: endpoint and deployment metrics
  • Autoscaling on the right metric
  • Closing the loop: alerts and retraining triggers
  • Exam patterns: signal, scope, and trigger

Monitoring surfaces: what each one detects

ConcernModel monitoring signalsEndpoint Traffic metricsDeployment Resource metrics
Question answeredIs the model still accurate?Is request flow healthy?Are the instances saturated?
Example metricData drift (Jensen-Shannon distance)Requests Per Minute, Request Latency, Connections ActiveCPU / GPU Utilization Percentage, memory, disk
Azure scopeWorkspace monitoring jobOnline endpointOnline deployment, per instance
Data sourceCollected production inputs and outputsAzure Monitor platform metricsAzure Monitor platform metrics
Computed byScheduled serverless Spark jobEmitted automatically every minuteEmitted automatically every minute
Typical actionAlert, then retrainAlert or scale on latencyAutoscale on CpuUtilizationPercentage

Decision tree

Prediction quality?Model monitoringdata drift + data qualityYesNoLive serving health?YesNoRequest flow?Fixed cadence?YesNoYesNoEndpoint TrafficRequests Per Minute,Request LatencyDeployment ResourceCPU / GPUUtilization %Azure ML Schedulerecurrence or cron,time onlyEvent Gridsubscription to aLogic App / Function
Cheat sheet

Cheat sheet

  • Data drift compares production data against a reference
  • Monitoring requires production data collection on the deployment
  • Out-of-box monitoring tracks three signals; feature attribution drift is advanced
  • Feature importance focuses monitoring on the most influential features
  • Traffic-category metrics report endpoint health
  • Match the endpoint metric to the signal you need
  • Deployment-scope Resource metrics drive autoscale and reveal compute saturation
  • Monitoring alerts when a metric exceeds its threshold
  • Wire alerts to trigger a retraining pipeline
  • Azure ML Schedules run a pipeline job on a recurrence or cron cadence, with no event-based triggers
  • Azure ML publishes workspace lifecycle events to Event Grid for event-driven retraining

Unlock with Premium — includes all practice exams and the complete study guide.

Also tested in

References

  1. Model monitoring in production - Azure Machine Learning
  2. Monitor online endpoints - Azure Machine Learning
  3. Collect production data from models for real-time inferencing - Azure Machine Learning
  4. Monitor model performance in production - Azure Machine Learning
  5. Azure Machine Learning monitoring data reference - Azure Machine Learning
  6. Autoscale online endpoints - Azure Machine Learning
  7. Schedule pipeline jobs - Azure Machine Learning
  8. Trigger events in ML workflows - Azure Machine Learning