Model & Data Monitoring — MLA-C01 Study Guide

Why models drift, and the two kinds of drift

A credit model that scored cleanly at launch starts approving riskier applicants a year later, and not one line of its code changed. The world it predicts on moved, and the model did not. Monitoring exists to catch that gap between a frozen model and a moving reality before it costs you. The first thing to get straight is what moved, because the answer decides which signal can even see it.

Two distinct things can change after deployment, and they share one mental model: in both, the live data the model now sees no longer matches the training data it learned from. They differ only in which relationship broke.

Data drift (covariate shift)

Data drift means the distribution of the input features moves away from the training distribution. The mortgage applications arriving this month skew younger and higher-income than the ones the model trained on, so the inputs themselves have shifted even though the rule mapping inputs to risk is unchanged. The defining property of data drift is that you can detect it from the inputs alone: compare the live feature statistics against a baseline taken from training data, and a divergence is data drift. You do not need to know whether the predictions were right.

Concept drift

Concept drift means the relationship between the inputs and the correct answer changes. The same applicant profile that meant low risk during a stable economy now means high risk during a downturn; the inputs may look identical to training, but the right label for them is different. The defining property of concept drift is the opposite of data drift: you cannot see it from inputs alone. You only detect it by comparing the model's predictions against the real outcomes (ground-truth labels) once those outcomes are known, and watching a metric like accuracy fall.

That split is the whole reason there are different monitor types. Drift you can catch from inputs (data drift, shifting feature importance) is monitored continuously and cheaply. Drift you can only catch from outcomes (concept drift) needs labels merged back in, which arrive later and cost more to obtain. The AWS Well-Architected Machine Learning Lens^[1] makes monitoring a deployed model a first-class operational requirement for exactly this reason: a model is a perishable asset, and the lens's monitoring guidance is to track both data quality and model quality continuously and to automate the response when either degrades.

Drift splits in two: data drift moves the inputs and is visible from inputs alone; concept drift moves the input-to-label rule and is visible only against ground-truth outcomes.

SageMaker Model Monitor: the four types and the pipeline

Once you know which drift you fear, Amazon SageMaker Model Monitor^[2] is the managed service that watches for it on a schedule and alerts you through CloudWatch. (Note the operational caveat: AWS has closed Model Monitor to new customers effective 31 July 2026, with existing customers unaffected and the feature still squarely on the exam. Treat it as the canonical answer for SageMaker production monitoring questions.) Every monitor type runs the same loop, so learn the loop once.

Data Capture comes first

Nothing can be monitored until the endpoint records what it saw and what it answered. Data Capture^[3] is the prerequisite step: you enable it in the endpoint configuration for a real-time endpoint (or in the batch transform job for batch monitoring), and SageMaker AI logs each inference request input and prediction output to an S3 location. You set a sampling percentage rather than capturing everything, and for a persistent real-time endpoint you can toggle capture on and off or change the sampling frequency over time. Model Monitor automatically parses this captured data; without it, there is no data to analyze.

Baseline, schedule, compare, alarm

With capture flowing, the loop is: create a baseline from the dataset used to train the model, which computes metrics and suggests constraints; create a monitoring schedule that says what to collect, how often, and how to analyze it; let each scheduled job recompute the metrics on the captured live data and compare them to the baseline constraints, reporting anything outside the constrained values as a violation; and watch the metrics and notifications that land in Amazon CloudWatch^[4], where you set an alarm on the violation metric.

The four monitor types

Four types plug into that loop, each watching a different signal:

Data quality monitors drift in input feature statistics against the training baseline. This is the direct detector for data drift, and it needs only the inputs.
Model quality monitors drift in prediction metrics such as accuracy, precision, and recall. Because it grades predictions, it requires the real outcomes: you merge ground-truth labels with the captured predictions, which is how concept drift surfaces.
Bias drift monitors whether the live data has introduced bias the training data did not have, recomputing bias metrics such as DPPL on a rolling window and alerting when they leave an allowed range. It is powered by SageMaker Clarify^[5].
Feature attribution drift monitors whether the importance ranking of features in live data has diverged from the baseline, scored with NDCG (normalized discounted cumulative gain) over the Clarify SHAP^[6] attributions. A shift in which features drive predictions is an early warning of input drift, and it needs only inputs and predictions, not labels.

The Clarify link matters for the exam: Clarify run as a one-off job is point-in-time detection of bias or explanations on a dataset or trained model, while Clarify inside Model Monitor is the ongoing, over-time drift detector on a live endpoint.

Two scope limits worth memorizing

Model Monitor computes metrics on tabular data only. It can still monitor an image classifier's tabular label output, but not the raw image pixels going in, so monitor a tabular feature representation when the raw input is not tabular. And Model Monitor supports only endpoints that host a single model; it cannot monitor a multi-model endpoint.

The pipeline every monitor type shares: Data Capture and a training-data baseline feed scheduled jobs that compare live metrics to constraints and alarm in CloudWatch on a violation.

Validating a new model: A/B testing and shadow tests

When a retrain or a new feature set produces a candidate model, offline validation metrics are not enough to trust it with real users. SageMaker AI gives two ways to test a candidate against live production traffic, and the exam wants you to pick the right one from the wording of the scenario.

A/B testing with production variants

The building block is the production variant: a model plus its instance type and count, deployed behind a SageMaker endpoint. An endpoint can host several variants at once, and you split traffic by setting each variant's weight^[7] in the endpoint configuration. Two variants each with weight 1 each get 50% of requests; that is an A/B test, where both the old and new model serve real users and you compare their CloudWatch metrics (latency, invocations, and your own accuracy metrics) to see which wins. You can also bypass the random split and send a request to a named variant with the TargetVariant parameter on InvokeEndpoint, which is how you smoke-test one variant directly. Once a winner is clear, you shift traffic by calling UpdateEndpointWeightsAndCapacities, which changes the weights with no endpoint downtime: ramp the new variant 50/50, then 75/25, then 100/0, and finally delete the loser. The key property is that A/B testing exposes the candidate to real users, so a bad candidate does affect a slice of them.

Shadow testing

A shadow test^[8] removes that risk. You deploy the candidate as a shadow variant and SageMaker AI routes a copy of the live inference requests to it in real time within the same endpoint, but only the production variant's responses are returned to the calling application. The shadow's responses are discarded or logged for offline comparison, so you measure how the candidate would have behaved on real traffic, on real latency and error rates, without a single user ever receiving its output. Shadow testing is the answer when you must validate operational performance, a patched container, or a new instance type against live traffic with zero user impact. Note its incompatibilities: shadow tests do not work with serverless inference, asynchronous inference, multi-container or multi-model endpoints, or Inferentia (Inf1) instances.

Closing the loop: automated retraining

Monitoring is only useful if a breach leads to action. The production pattern wires the Model Monitor CloudWatch alarm to Amazon EventBridge^[9], whose rule then starts a retraining workflow, typically a SageMaker Pipelines execution or a Step Functions state machine, that retrains on fresh data, registers the new model version in the Model Registry for approval, and deploys it as a new variant. That turns drift detection into a self-healing system: drift crosses a threshold, CloudWatch fires, EventBridge triggers, the model retrains, and the cycle repeats, with a human approval gate at the registry if you want one.

Pick by user impact: A/B with production variants when the candidate may serve a slice of real users, a shadow test when it must see live traffic with none reaching users.

Reading the question: which monitoring answer

Monitoring scenarios on this exam usually describe a symptom and ask for the service or monitor type that addresses it. Read for the signal phrase, then check the trap.

Signal phrases for the monitor type

"Model accuracy is declining in production" or "predictions getting worse over time, we have the true labels" selects the Model quality monitor, because grading predictions needs ground truth. "The input data looks different from training" or "detect drift in incoming features" with no mention of labels selects the Data quality monitor. "Fairness/bias must be maintained after deployment" or "alert if bias exceeds a threshold over time" selects Bias drift (Clarify in Model Monitor). "Which features are driving predictions is changing" selects Feature attribution drift (Clarify, SHAP). A one-time "check this dataset/model for bias before launch" or "explain the model's predictions" is plain SageMaker Clarify, not Model Monitor.

Signal phrases for validation and automation

"Compare a new model against the current one on live traffic and split requests" selects production variants / A/B testing; "shift more traffic to the better model with no downtime" is UpdateEndpointWeightsAndCapacities. "Validate a new model on live traffic without affecting users" or "mirror production traffic to the candidate" selects a shadow test. "Automatically retrain when drift is detected" selects Model Monitor alarm to EventBridge to a retraining pipeline.

The distractors that catch people

The first trap is choosing Model quality monitoring when no ground-truth labels are available; without labels you cannot grade predictions, so the answer is data-quality or feature-attribution drift instead. The second is reaching for a one-off Clarify job when the scenario says bias is changing over time in production, which is the ongoing Bias drift monitor, not point-in-time detection. The third is forgetting Data Capture: a question that asks why a monitoring schedule has no data to analyze is missing the capture configuration on the endpoint. The fourth is confusing A/B testing with shadow testing; if the requirement is no user impact, A/B is wrong because it serves the candidate to real users, and shadow is the only option that withholds the candidate's responses. The fifth is trying to monitor a multi-model endpoint or raw image input, both of which Model Monitor does not support.

The four SageMaker Model Monitor types

Monitor type	What it detects	Needs ground-truth labels?	Powered by Clarify?
Data quality	Drift in input feature statistics versus the training baseline (data drift)	No, inputs only	No
Model quality	Drop in prediction metrics such as accuracy, precision, recall (concept drift shows here)	Yes, labels merged with predictions	No
Bias drift	Bias metrics (e.g. DPPL) leaving an allowed range as live data shifts	Yes, for label-aware metrics	Yes
Feature attribution drift	Change in feature-importance ranking (NDCG of SHAP values) versus baseline	No, inputs and predictions	Yes

Decision tree

Cheat sheet

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Data drift is detectable from inputs alone; concept drift is not

Data drift (covariate shift) means the live input feature distribution moved away from the training distribution, and you can detect it by comparing live feature statistics to a training baseline without knowing whether the predictions were right. Concept drift means the relationship between inputs and the correct label changed, so the same inputs now imply a different answer, and you can only catch it by comparing predictions against real outcomes and watching accuracy fall. The distinction decides which monitor can even see the problem.

Trap Assuming an input-statistics monitor will catch concept drift; if the inputs look unchanged but the input-to-label rule moved, only ground-truth comparison reveals it.

Enable Data Capture before anything can be monitored

Data Capture is the prerequisite for SageMaker Model Monitor: you turn it on in the endpoint configuration for a real-time endpoint, or in the batch transform job for batch monitoring, and SageMaker AI logs each inference request input and prediction output to S3. You set a sampling percentage rather than capturing every request, and on a persistent real-time endpoint you can change the sampling frequency or toggle capture over time. Model Monitor automatically parses this captured data and compares it to your baseline, so a monitoring schedule with capture disabled has nothing to analyze.

Trap Creating a monitoring schedule but never enabling Data Capture on the endpoint, so the scheduled jobs run against no captured data.

5 questions test this

Model Monitor has exactly four monitor types

SageMaker Model Monitor supports four and only four monitor types: Data quality (drift in input feature statistics), Model quality (drift in prediction metrics like accuracy), Bias drift, and Feature attribution drift. Bias drift and Feature attribution drift are the two powered by SageMaker Clarify. Knowing the set lets you eliminate distractors that invent a fifth type or attribute one to the wrong service.

1 question tests this

A financial services company has deployed a fraud detection model using Amazon SageMaker. The ML team wants to implement comprehensive…

Every monitor type runs one baseline-to-alarm pipeline

All four monitor types share the same loop: compute a baseline of metrics and constraints from the training dataset, run a monitoring schedule that recomputes those metrics on captured live data, flag any value outside the baseline constraints as a violation, and emit metrics to CloudWatch where an alarm notifies you. Learning the loop once covers all four; only the metric being computed differs by type.

12 questions test this

The baseline comes from the training dataset

Model Monitor builds its baseline from the dataset that was used to train the model, computing reference metrics and suggesting constraints; live predictions are then compared against those constraints and anything outside is reported as a violation. Baselining against training data is what makes a deviation meaningful, because it measures how far production has moved from what the model learned.

Trap Baselining on recent production data instead of the training set, which hides the very drift you are trying to detect by treating the drifted state as normal.

3 questions test this

Model quality monitoring needs ground-truth labels

The Model quality monitor grades predictions against real outcomes, so it requires ground-truth labels merged with the captured predictions; this is how concept drift, an accuracy decay, surfaces. Labels often arrive late or never, so when they are unavailable, fall back to data-quality or feature-attribution drift monitoring, which need only the inputs and predictions.

Trap Choosing Model quality monitoring in a scenario with no ground-truth labels; without outcomes you cannot grade predictions, so a data-quality or feature-attribution monitor is the answer.

1 question tests this

A data science team is setting up comprehensive monitoring for a credit scoring model deployed on Amazon SageMaker. The team needs to…

Data quality monitor watches input statistics for data drift

The Data quality monitor recomputes statistics on the captured input features and compares them to the training baseline, making it the direct detector for data drift. It needs only the inputs, no labels, so it runs continuously and cheaply and is the right pick when a question says the incoming data looks different from training and never mentions outcomes.

2 questions test this

Bias drift and feature attribution drift are powered by Clarify

Bias drift monitoring recomputes bias metrics such as DPPL on a rolling window of live data and alerts when they leave an allowed range, while Feature attribution drift uses NDCG over Clarify's SHAP attributions to detect when the importance ranking of features diverges from the baseline. Both are SageMaker Clarify integrated into Model Monitor, which is what makes them the over-time, on-a-live-endpoint detectors rather than point-in-time scans.

16 questions test this

Clarify as a one-off job is detection; Clarify in Model Monitor is over-time drift

A standalone SageMaker Clarify processing job is point-in-time detection: it measures bias on a dataset or trained model, or produces a SHAP explanation, once. Clarify running inside Model Monitor on a schedule is the ongoing drift detector that watches a live endpoint over time. Match the answer to whether the scenario is a pre-launch check or a production-over-time concern.

Trap Reaching for a one-off Clarify job when the scenario says bias is changing over time in production; ongoing bias monitoring is the Bias drift monitor, not point-in-time detection.

Model Monitor computes metrics on tabular data only

Model Monitor calculates statistics and metrics on tabular data only. It can still monitor an image classifier by watching its tabular label output, but it cannot monitor the raw image pixels going in, so to monitor non-tabular inputs you monitor an engineered tabular feature representation instead. This limit appears as a distractor whenever a question proposes monitoring raw images, audio, or text directly.

Trap Proposing Model Monitor to watch raw image or text inputs directly; it computes metrics on tabular data only, so monitor a tabular feature representation.

Model Monitor supports single-model endpoints only

Model Monitor works on an endpoint that hosts a single model; it does not support monitoring a multi-model endpoint, where many models share one container behind one endpoint. A scenario that needs both MME density and built-in drift monitoring has a conflict the exam expects you to spot.

Trap Assuming a multi-model endpoint can be watched by Model Monitor; MME is unsupported, so the design must use single-model endpoints to get built-in monitoring.

A/B test new models with production variants split by weight

Deploy the old and new model as two production variants behind one endpoint and split live traffic by setting each variant's weight; two variants with equal weight each take 50% of requests. Both serve real users, so you compare their CloudWatch metrics to pick the winner. This is the real-traffic comparison answer, distinct from a shadow test, because the candidate actually responds to a slice of users.

Shift traffic between variants with UpdateEndpointWeightsAndCapacities

Once an A/B test names a winner, call UpdateEndpointWeightsAndCapacities to change the variant weights, which reroutes traffic with no endpoint downtime: ramp the new variant 50/50, then 75/25, then 100/0, then delete the loser. There is no need to recreate or update the endpoint itself to move traffic.

Trap Tearing down and recreating the endpoint to shift traffic between variants; UpdateEndpointWeightsAndCapacities reweights live with zero downtime.

Invoke a specific variant directly with TargetVariant

To bypass the weighted random split and send a request to one named production variant, pass the TargetVariant parameter on InvokeEndpoint; SageMaker AI then routes that request to the variant you named and the targeting overrides the configured traffic distribution. This is how you smoke-test or directly compare a single variant without changing weights.

Shadow tests validate on live traffic with zero user impact

A shadow test deploys the candidate as a shadow variant and routes a copy of live inference requests to it in real time within the same endpoint, but only the production variant's responses are returned to the calling application. The shadow's responses are discarded or logged for offline comparison, so you measure real latency, error rate, and behavior on production traffic without any user receiving the candidate's output. Reach for it when the requirement is to validate on live traffic with no chance of affecting users.

Trap Using an A/B test when the requirement is zero user impact; A/B serves the candidate to a slice of real users, so only a shadow test withholds the candidate's responses.

Shadow tests are incompatible with several endpoint features

Shadow tests do not work with serverless inference, asynchronous inference, Marketplace containers, multi-container endpoints, multi-model endpoints, or Inferentia (Inf1) instances; requesting a shadow test on such an endpoint fails validation. When a scenario combines one of these features with a shadow-test requirement, the design has to change.

Wire drift alarms to EventBridge to trigger automated retraining

To turn drift detection into action, connect the Model Monitor CloudWatch alarm to an Amazon EventBridge rule that starts a retraining workflow, typically a SageMaker Pipelines execution or a Step Functions state machine, which retrains on fresh data and registers the new model version for approval. This closes the loop so a threshold breach automatically launches retraining rather than waiting on a human to notice.

1 question tests this

A data science team has deployed a classification model on an Amazon SageMaker endpoint and configured SageMaker Model Monitor to detect…

Register the retrained model and gate promotion through approval

An automated retraining loop should register each new model version in the SageMaker Model Registry, where versions start as PendingManualApproval and a transition to Approved can initiate the CI/CD deployment. This adds a controlled gate so a freshly retrained model is reviewed or auto-approved before it replaces the production variant, rather than deploying blind on every drift event.

Monitoring is the Well-Architected ML Lens operations requirement

The AWS Well-Architected Machine Learning Lens treats a deployed model as a perishable asset and makes continuous monitoring of both data quality and model quality, plus an automated response when either degrades, a core operational practice. When a question frames monitoring as a design-principle or best-practice choice, the ML Lens monitoring guidance is the reference being tested.

ModelLatency is the model container's response time; OverheadLatency is SageMaker's added time

To break down endpoint response time, watch two AWS/SageMaker invocation metrics: ModelLatency is the interval a model takes to respond to a SageMaker Runtime request — it includes the local communication to send the request to and fetch the response from the model container, plus the time to run the inference in the container. OverheadLatency is the extra time SageMaker adds, measured from when SageMaker receives the request until it returns the response to the client, minus ModelLatency. Both are published in microseconds, so a 500 ms threshold is 500000.

Trap Reading ModelLatency in milliseconds and setting the alarm threshold to 500 instead of 500000.

11 questions test this

An M-out-of-N CloudWatch alarm suppresses transient spikes

To ignore brief spikes, use an 'M out of N' alarm: set EvaluationPeriods to N (the window of periods examined) and DatapointsToAlarm to M (how many must breach). '2 of 3' or '3 of 5' fires only on sustained breaches. Because SageMaker latency metrics are in microseconds, convert the threshold (500 ms -> 500000).

Trap Treating EvaluationPeriods and DatapointsToAlarm as equal — that demands every period breach, defeating the noise tolerance.

5 questions test this

ConcurrentRequestsPerModel scales out faster than InvocationsPerInstance

For fast scale-out — especially generative-AI endpoints with long, streaming requests — target-track on the high-resolution predefined metric SageMakerVariantConcurrentRequestsPerModelHighResolution (CloudWatch metric ConcurrentRequestsPerModel), which counts the simultaneous requests a model container handles, including requests queued inside the container. It emits data every 10 seconds, versus the standard InvocationsPerInstance metric that emits once per minute, so the policy scales out much more quickly on rising concurrent traffic. (Scale-in still happens at standard speed.)

Trap Using SageMakerVariantInvocationsPerInstance for a bursty gen-AI endpoint — its one-minute resolution detects spikes too late.

11 questions test this

Clarify flags bias drift when the confidence interval falls outside the allowed range

In Clarify bias monitoring a metric value (e.g. DPPL) closer to zero is more balanced. You specify an allowed range A (for example (-0.1, 0.1)) the metric should stay within during deployment. To stay robust on small windows, Clarify builds a Normal Bootstrap Interval C around the computed value; it interprets an overlap of C with A as 'the live bias is likely within the allowed range,' and raises a bias_drift_check alert only when C and A are disjoint (Clarify is confident the metric is outside the allowed range). constraint_violations.json records the facet, facet_value, metric short code, and check type.

Trap Assuming a value numerically near zero is safe — if its confidence interval falls entirely outside the allowed range, it still violates.

7 questions test this

The baseline job emits statistics.json and constraints.json; tune monitoring_config.comparison_threshold to cut false positives

DefaultModelMonitor.suggest_baseline() runs a Deequ-on-Spark job (the sagemaker-model-monitor-analyzer container) on the training data that outputs statistics.json (per-feature stats) and constraints.json (suggested constraints). Data drift is the baseline_drift_check, which fires when the distribution distance between current and baseline data exceeds monitoring_config.comparison_threshold. To make drift detection less sensitive, raise that threshold in constraints.json and pass the modified file to the monitoring schedule; violations then appear in constraint_violations.json.

Trap Deleting or re-running the baseline to quiet false positives instead of just raising monitoring_config.comparison_threshold in constraints.json.

8 questions test this

Why models drift, and the two kinds of drift

Data drift (covariate shift)

Concept drift

SageMaker Model Monitor: the four types and the pipeline

Data Capture comes first

Baseline, schedule, compare, alarm

The four monitor types

Two scope limits worth memorizing

Validating a new model: A/B testing and shadow tests

A/B testing with production variants

Shadow testing

Closing the loop: automated retraining

Reading the question: which monitoring answer

Signal phrases for the monitor type

Signal phrases for validation and automation

The distractors that catch people

The four SageMaker Model Monitor types

Decision tree

Cheat sheet

References