Deployment Infrastructure — MLA-C01 Study Guide

The four SageMaker AI inference options

A fraud model scoring card swipes in 50 ms, a document model that takes four minutes on a 400 MB PDF, and a nightly job that scores ten million rows all need different infrastructure, and Amazon SageMaker AI^[1] gives each its own deployment option. Getting this choice right is the single highest-value decision in this domain. Four options exist, and they sort cleanly along three axes: how traffic arrives, how big each request is, and how long inference takes.

All four share one model: you register a trained model plus a serving container, then choose how requests reach it. They differ only in whether an endpoint stays up, how large a request can be, and how the caller gets the answer.

Real-time endpoint

A real-time endpoint^[2] is a persistent HTTPS endpoint that stays provisioned and warm, so it answers interactive requests with consistent low latency. You call it with the synchronous InvokeEndpoint^[3] API, which caps the request body at 6 MB and requires the model container to respond within 60 seconds. Because the instances run continuously, you pay for them even when no requests arrive. This is the default for online prediction behind an application: recommendations, fraud scoring, search ranking.

Serverless inference

Serverless inference^[4] removes instance management: SageMaker AI launches compute on demand and scales it to zero when no requests arrive, so you pay only while requests are processed. The trade is a cold start, the extra latency to spin up compute when traffic resumes after idle, which you can monitor with the OverheadLatency CloudWatch metric and minimize with Provisioned Concurrency. You configure memory from 1024 MB to 6144 MB rather than picking an instance type. Serverless explicitly does not support GPUs, multi-model endpoints, VPC, network isolation, or Model Monitor, so a workload that needs any of those cannot run serverless. Reach for it when traffic is spiky or unpredictable and idle cost would dominate.

Asynchronous inference

Asynchronous inference^[5] queues each request and processes it in the background, which lets it accept payloads up to 1 GB and run inference for up to one hour, far beyond the synchronous limits. The caller submits a request that points at an S3 input object and gets back a location where the result will land; SageMaker AI writes the prediction to S3 and can send an SNS notification on completion. Like serverless, an async endpoint can scale to zero between bursts. Use it for large inputs or long-running models, such as high-resolution computer vision or large-document NLP, where the request is too big or too slow for a real-time endpoint but you still want near-real-time turnaround.

Batch transform

Batch transform^[6] is the only option with no endpoint at all. You start a transform job that reads a dataset from S3, spins up instances, distributes the records across them, writes one .out prediction file per input file back to S3, and then tears everything down. The per-record payload is capped by MaxPayloadInMB, which must not exceed 100 MB. There is nothing left running and nothing to pay for once the job ends. This is the answer whenever the task is to score a whole fixed dataset offline rather than serve live requests.

The reconciling rule when async and batch both seem to fit: async serves individual large or slow requests that arrive over time and need a per-request answer back, while batch transform processes a whole dataset at once with no persistent endpoint. One is request-shaped, the other is job-shaped.

Pick top-down: offline dataset selects batch transform, large or slow requests select asynchronous, spiky traffic selects serverless, steady traffic selects a real-time endpoint.

Single, multi-model, and multi-container endpoints

Once you have chosen a real-time or asynchronous endpoint, a second question follows: how many models sit behind it, and how do they share the hardware. The cost difference between one endpoint per model and one endpoint for thousands of models is enormous, so the exam tests when each pattern fits.

A single-model endpoint dedicates the fleet to one model. It gives the most predictable latency and is the right call when a model has high traffic or strict latency requirements and should not compete for memory.

Multi-model endpoints (MME)

A multi-model endpoint^[7] hosts a large number of models on one shared serving container and one fleet. Instead of loading every model up front, SageMaker AI dynamically loads a model into the container's memory on the first request that targets it (named with the TargetModel header), keeps it cached for subsequent calls, and unloads the least-recently-used models when memory fills; an unloaded model stays on the instance's storage volume so it can reload without re-downloading from S3. The catch is a cold-load penalty: the first call to a model that is not in memory waits while SageMaker AI downloads and loads it. MME is ideal for many models that use the same ML framework and have a mix of frequent and infrequent traffic, the classic per-customer or per-region model fleet in a SaaS application. MME supports both CPU- and GPU-backed instances.

Multi-container endpoints and inference pipelines

When models need different frameworks, an MME does not fit, because every model must run in the one shared container. A multi-container endpoint^[8] instead hosts up to 15 distinct containers behind a single endpoint. You can invoke one container directly with the TargetContainerHostname header, or chain the containers as an inference pipeline^[9], a linear sequence of 2 to 15 containers where each container's output feeds the next. A pipeline is how you co-locate preprocessing, prediction, and post-processing on the same instances so feature transforms run with low latency right next to the model, often reusing the Spark ML or scikit-learn container built during training.

The distinction the exam leans on: MME is many models, one framework, shared container, cost savings through density; multi-container is a few models, different frameworks, separate containers, invoked directly or in sequence. They solve different problems and are not interchangeable.

Single-model dedicates the fleet to one model; MME packs many same-framework models into one shared container with on-demand loading; multi-container hosts up to 15 different-framework containers.

Choosing containers and compute for inference

The endpoint type decides how requests arrive; the container and instance decide what actually runs the model and how much it costs. Both are graded decisions on this exam, and both follow from the model rather than from preference.

Provided versus bring-your-own containers

SageMaker AI ships managed containers for its built-in algorithms and for popular frameworks such as TensorFlow, PyTorch, and MXNet, available as prebuilt Deep Learning Containers. Use a provided container whenever your model fits a supported framework, because there is nothing to build or maintain. When you need a custom runtime, an unusual dependency, or your own serving stack, you bring your own container^[10] (BYOC): you package your code into a Docker image that honors the SageMaker AI serving contract (responding to /invocations and /ping) and push it to Amazon ECR. The rule is to reach for BYOC only when a provided container cannot do the job, since you take on the maintenance.

CPU, GPU, and AWS Inferentia

Match the instance family to the model's compute profile. CPU instances (the c and m families) are the cheapest choice and handle classical models, tree ensembles, and lightweight inference well. GPU instances (the g and p families) provide the parallel matrix math that deep neural networks need, so a large vision or language model that is too slow on CPU belongs on a GPU. AWS Inferentia^[11] (the inf1 and inf2 instances, programmed through the AWS Neuron SDK) is purpose-built for high-throughput, low-cost deep-learning inference and often beats general-purpose GPUs on price-performance once a model is compiled for it. When the right instance is genuinely unclear, SageMaker Inference Recommender^[12] runs load tests across candidate instance types and reports latency, throughput, and cost so the pick is measured.

SageMaker Neo for the edge

SageMaker Neo^[13] compiles a trained model into a hardware-optimized form for a specific target and pairs it with a compact runtime, so the model runs faster and smaller without being retrained. Neo compiles models from frameworks such as PyTorch, TensorFlow, MXNet, and ONNX for two kinds of target: cloud instances (including Inferentia) and edge devices built on processors from ARM, Intel, Nvidia, and others. The compiler can also quantize parameters to INT8 or FP16 to shrink the model further. When a scenario puts inference on a constrained edge device, a camera, a sensor gateway, a vehicle, with limited CPU and memory and no dependable network, Neo is the answer: compile for the device and deploy there, rather than round-tripping every prediction to a cloud endpoint.

Pick inference compute by the model: edge devices need a Neo-compiled build, deep nets need a GPU (or Inferentia for cost per throughput), and classical models run cheapest on CPU.

Reading the question: which deployment when

Deployment-infrastructure questions are usually a short scenario whose wording points straight at one answer. Train yourself to read for the signal phrase, then check the trap.

Signal phrases for the inference option

"Interactive, low latency, steady traffic" or "a web application calls the model per request" selects a real-time endpoint. "Unpredictable or intermittent traffic", "idle periods", or "avoid paying for idle capacity" selects serverless inference, but only if no GPU is required, because serverless has no GPU support. "Large payload", "payload up to 1 GB", "inference takes several minutes", or "process up to an hour" selects asynchronous inference. "Score an entire dataset", "no persistent endpoint needed", "nightly", or "offline predictions to S3" selects batch transform.

Signal phrases for endpoint topology and compute

"Thousands of models", "a model per customer", or "many similar models, same framework, cost-efficient" selects a multi-model endpoint. "Models in different frameworks behind one endpoint" or "preprocess, predict, post-process in sequence" selects a multi-container endpoint or inference pipeline. "Deploy to an IoT or edge device with limited resources" selects SageMaker Neo. "Which instance type for this model" with no obvious answer selects SageMaker Inference Recommender.

The distractors that catch people

The most common wrong answer is offering a real-time endpoint for a large or slow request; it fails the 6 MB and 60-second synchronous limits, and asynchronous inference is the intended answer. The second is reaching for serverless when the scenario mentions a GPU or Model Monitor, both of which serverless does not support. The third is choosing a multi-model endpoint for models in different frameworks, which MME cannot host because all models share one container; that case is multi-container. The fourth is standing up a persistent endpoint to score a fixed dataset once, where batch transform is cheaper because it leaves nothing running. Reading for these four traps resolves most deployment questions before you weigh the remaining options.

SageMaker AI inference options by traffic shape, payload, latency, and cost

Criterion	Real-time endpoint	Serverless inference	Asynchronous inference	Batch transform
Best-fit traffic	Steady, interactive	Spiky, intermittent	Large or long-running requests	Offline, whole dataset
Persistent endpoint	Yes, always on	No, scales to zero	Yes, can scale to zero	No endpoint at all
Payload limit	6 MB per request	Synchronous request	1 GB (via S3)	100 MB per record (S3)
Processing time	60 s per request	60 s per request	Up to 1 hour	Whole job, no per-request cap
Cold start	None (kept warm)	Possible on scale-up	Possible on scale-up	Job spins up per run
Pay for idle	Yes, always provisioned	No	No when scaled to zero	No, job ends

Decision tree

Cheat sheet

Sharp facts the exam loves — give these one last read before exam day.

Cheat sheet

Sharp facts the exam loves — scan these before test day.

Use a real-time endpoint for steady, interactive, low-latency traffic

A SageMaker AI real-time endpoint is a persistent endpoint that stays provisioned and warm so it answers synchronous requests with consistent low latency, which fits online prediction behind an application. The synchronous InvokeEndpoint call caps the request body at 6 MB and the model must respond within 60 seconds. Because the instances run continuously you pay for them even during idle, so it suits sustained traffic rather than occasional bursts.

Trap Choosing a real-time endpoint for a payload over 6 MB or inference that runs minutes; it exceeds the synchronous limits and asynchronous inference is the right option.

13 questions test this

Use serverless inference for spiky traffic that should scale to zero

Serverless inference launches compute on demand and scales to zero when no requests arrive, so you pay only while requests are processed, which fits unpredictable or intermittent traffic with idle periods. You configure memory from 1024 MB to 6144 MB instead of choosing an instance type. The tradeoff is a cold start, the extra latency to spin compute back up after idle, which you measure with the OverheadLatency metric and reduce with Provisioned Concurrency.

Trap Keeping a real-time endpoint running for traffic that arrives in occasional bursts, paying for idle instances when serverless would scale to zero.

6 questions test this

Serverless inference does not support GPUs, MME, VPC, or Model Monitor

Serverless inference excludes several real-time features: GPUs, multi-model endpoints, VPC configuration, network isolation, data capture, multiple production variants, Model Monitor, and inference pipelines. A workload that needs any of these cannot run serverless. The most common case is a deep-learning model that needs a GPU, which forces a real-time or asynchronous endpoint instead.

Trap Selecting serverless inference when the scenario mentions a GPU or Model Monitor, neither of which serverless supports.

9 questions test this

Use asynchronous inference for large payloads or long-running requests

Asynchronous inference queues each request and processes it in the background, which lets it accept payloads up to 1 GB and run inference for up to one hour, well past the synchronous 6 MB and 60-second limits. The caller submits a request pointing at an S3 input and gets a location where the result will be written; SageMaker AI writes the prediction to S3 and can fire an SNS notification on completion. Like serverless, an async endpoint can scale to zero between bursts, which suits high-resolution vision or large-document NLP.

Trap Reaching for batch transform when each large request needs its own near-real-time answer back; batch transform is for scoring a whole dataset, not individual queued requests.

Use batch transform to score a whole dataset offline with no endpoint

Batch transform is the only inference option with no persistent endpoint: a transform job reads a dataset from S3, spins up instances, distributes records across them, writes one .out prediction file per input file back to S3, then tears down. The per-record payload is set by MaxPayloadInMB, which must not exceed 100 MB. Nothing is left running afterward, so it is the cheapest answer when the task is a fixed offline scoring run rather than live serving.

Trap Standing up a persistent endpoint to score a fixed dataset once, paying to keep it running when batch transform leaves nothing behind.

Use a multi-model endpoint for many same-framework models on one fleet

A multi-model endpoint (MME) hosts a large number of models on one shared serving container and one fleet, loading each model into memory on first request (named with the TargetModel header) and evicting the least-recently-used models when memory fills. It cuts cost sharply for many similar models, the classic per-customer or per-region SaaS fleet, and supports CPU- and GPU-backed instances. All models must use the same ML framework because they share one container.

Trap Putting models that need different frameworks on an MME; they all share one container, so different-framework models require a multi-container endpoint.

An MME pays a cold-load penalty on the first call to an uncached model

Because an MME loads models on demand rather than up front, the first request for a model not currently in the container's memory waits while SageMaker AI downloads it from S3 and loads it. An unloaded model stays on the instance's storage volume so it can reload without re-downloading. MME therefore works best when the application tolerates occasional cold-start latency on infrequently used models and reserves dedicated single-model endpoints for high-traffic or strict-latency models.

Trap Hosting a high-TPS, strict-latency model on a shared MME where it competes for memory; a dedicated single-model endpoint gives predictable latency.

Use a multi-container endpoint for different-framework models behind one endpoint

A multi-container endpoint hosts up to 15 distinct containers on a single endpoint, which is how you serve models that use different frameworks together. You can invoke one container directly with the TargetContainerHostname header, or chain the containers as an inference pipeline so each container's output feeds the next. This is the different-framework counterpart to an MME, which can only host same-framework models in its one shared container.

Trap Using a multi-model endpoint when the requirement is models in different frameworks behind one endpoint; that case needs a multi-container endpoint.

An inference pipeline chains 2 to 15 containers on the same instances

An inference pipeline is a SageMaker AI model made of a linear sequence of 2 to 15 containers, where the first container handles the request and each intermediate response feeds the next container. Because the containers are co-located on the same instances, feature processing and prediction run with low latency, often reusing the Spark ML or scikit-learn container built during training. A pipeline can serve real-time predictions or run inside a batch transform job.

Trap Standing up separate endpoints and network calls for preprocessing and prediction when an inference pipeline co-locates them on one set of instances for lower latency.

1 question tests this

A logistics company needs to deploy three related ML models for route optimization, demand forecasting, and fleet allocation. The models…

Use a provided container unless the model needs a custom runtime

SageMaker AI ships managed containers for its built-in algorithms and prebuilt Deep Learning Containers for frameworks such as TensorFlow, PyTorch, and MXNet, so a model that fits a supported framework needs nothing built. You bring your own container (BYOC) only when you need a custom runtime, an unusual dependency, or your own serving stack: package the code into a Docker image that honors the SageMaker AI serving contract and push it to Amazon ECR.

Trap Building a custom container for a model that a provided framework container already supports, taking on maintenance for no benefit.

A BYOC inference image must answer /invocations and /ping

A bring-your-own inference container must implement the SageMaker AI serving contract: it responds to POST /invocations with predictions and to GET /ping for health checks, and the image is stored in Amazon ECR. Honoring this contract is what lets SageMaker AI host any custom image the same way it hosts a provided one.

1 question tests this

A data science team is developing a custom inference container for Amazon SageMaker real-time endpoints. The team has created a Flask…

Serve classical and lightweight models on CPU instances

CPU instances such as the c and m families are the cheapest inference compute and handle classical models, tree ensembles, and lightweight workloads well. They are the default choice unless the model genuinely needs hardware acceleration, so paying for a GPU for a small tabular model is wasted cost.

Trap Provisioning a GPU instance for a lightweight or classical model that runs fine and far cheaper on CPU.

Serve deep neural networks on GPU instances

GPU instances in the g and p families provide the parallel matrix math that deep neural networks need, so a large vision or language model that is too slow on CPU belongs on a GPU. The g family targets cost-effective inference and the p family targets the heaviest training and inference, so match the family to the model's size and throughput need.

2 questions test this

Use AWS Inferentia for cost-efficient high-throughput deep-learning inference

AWS Inferentia, the inf1 and inf2 instances programmed through the AWS Neuron SDK, is purpose-built for deep-learning inference and often beats general-purpose GPUs on price-performance once a model is compiled for it. Reach for Inferentia when the goal is the lowest inference cost per throughput at scale, after compiling the model with the Neuron toolchain.

Trap Assuming a model runs on Inferentia unchanged; it must first be compiled with the Neuron SDK to target the inf1/inf2 hardware.

Compile with SageMaker Neo to run a model on edge or constrained hardware

SageMaker Neo compiles a trained model into a hardware-optimized form for a specific target and ships a small-footprint runtime, giving faster inference and a smaller package without retraining the model. It compiles models from frameworks such as PyTorch, TensorFlow, MXNet, and ONNX for cloud instances (including Inferentia) and for edge devices on ARM, Intel, Nvidia, and similar processors, and can quantize parameters to INT8 or FP16. When inference must run on a resource-limited edge device with no reliable network, Neo is the deployment answer rather than a cloud endpoint.

Trap Routing every prediction from a disconnected edge device back to a cloud endpoint when Neo can compile the model to run on the device itself.

Use SageMaker Inference Recommender to choose the instance type by load test

SageMaker Inference Recommender runs load tests across candidate instance types and reports latency, throughput, and cost, so the instance choice for an endpoint is measured rather than guessed. Reach for it when a scenario asks which instance type to deploy a model on and the answer is not obvious from the model's size alone.

Asynchronous and serverless endpoints can scale to zero; real-time cannot

Among endpoint types, serverless inference and asynchronous inference both scale their compute to zero during idle periods so you stop paying when no requests arrive, while a real-time endpoint stays provisioned and bills continuously. This is why a steady-traffic model uses real-time and an intermittent one uses serverless or async: the cost model follows the traffic pattern.

Trap Assuming a standard real-time endpoint scales to zero on idle; only serverless and asynchronous endpoints do, so an idle real-time endpoint keeps billing.

Async serves individual large requests; batch transform processes a whole dataset

Asynchronous inference and batch transform both handle large or slow inference, but they differ in shape: async queues individual requests that arrive over time and returns a per-request answer to S3, while batch transform runs one job over an entire dataset with no persistent endpoint. Pick async when callers submit requests and need each answer back; pick batch transform when the task is to score a fixed dataset in one offline run.

Split a single input file into many objects so batch transform uses every instance

SageMaker Batch Transform partitions S3 objects by key and maps whole objects to instances, so a single input file is processed by exactly one instance while the rest sit idle no matter how many you provision. Split the data into at least as many S3 objects (under a common prefix) as instances to distribute the work in parallel.

Trap Adding more instances to a one-file job — the extra instances stay idle because one object can only map to one instance.

7 questions test this

Tune SplitType, BatchStrategy, and MaxPayloadInMB to control batch-transform record packing

For batch transform, SplitType=Line breaks input on newline boundaries into records, BatchStrategy=MultiRecord packs as many records as fit per request up to MaxPayloadInMB (default 6 MB) while BatchStrategy=SingleRecord sends one record per request, and MaxConcurrentTransforms sets parallel requests per instance. AWS requires MaxPayloadInMB be no greater than 100 MB, and if MaxConcurrentTransforms is set, MaxConcurrentTransforms × MaxPayloadInMB must also not exceed 100 MB.

Trap Setting MaxConcurrentTransforms × MaxPayloadInMB above 100 MB — the job configuration is rejected.

9 questions test this

Use DataProcessing (InputFilter/JoinSource/OutputFilter) to keep an ID column out of inference but in the output

Batch transform's DataProcessing parameter filters columns with JSONPath: InputFilter selects which columns reach the model (e.g. $[1:] to exclude a leading ID column), JoinSource=Input joins the original record back to the prediction, and OutputFilter chooses what lands in the output file. This carries a passthrough key like a customer ID alongside predictions without feeding it to the model.

3 questions test this

Host a small model on Lambda: load the model in init and add provisioned concurrency for cold starts

AWS Lambda can serve lightweight ML inference with the model on an EFS mount or baked into an ECR container image, and CPU scales in proportion to memory (one vCPU at 1,769 MB, up to 6 vCPUs near the 10,240 MB maximum). Because loading a model can take tens of seconds, put the load outside the handler so it runs during init, and enable provisioned concurrency on a version/alias to pre-initialize environments and eliminate cold-start latency.

Trap Loading the model inside the handler — every cold start pays the full load time on the first request.

8 questions test this

Lambda reaches EFS only inside a VPC with mount targets per AZ and NFS port 2049 open

A Lambda function must be attached to the EFS VPC (use subnets in at least two Availability Zones for resilience), EFS must have a mount target in every AZ the function uses, an access point connects the function (you can't mount the file-system root directly), and the security groups must allow NFS traffic on port 2049 between the function and the mount targets. A missing 2049 rule or absent mount target shows up as connection timeouts, often only when the function scales out.

Trap Granting the elasticfilesystem IAM permissions but leaving port 2049 blocked in the security group — the mount still times out.

6 questions test this

Automate ECR image cleanup with a lifecycle policy on tag pattern or count

An ECR lifecycle policy expires images automatically with no custom code: rules use tagStatus (tagged/untagged) plus tagPatternList (e.g. prod*) and countType=imageCountMoreThan to keep the N most recent matching images, or countType=sinceImagePushed to expire images older than a set number of days. Use the lifecycle-policy preview to see what a rule would delete before applying it.

Trap Combining tagPatternList and tagPrefixList in one rule — a rule may specify only one of them (both may only be used when tagStatus is tagged).

4 questions test this

Choose ECR enhanced scanning (Inspector) for continuous OS + language CVE coverage

ECR basic scanning uses AWS-native technology to scan for operating-system package vulnerabilities on push (or manually); ECR enhanced scanning integrates with Amazon Inspector to continuously re-scan for both operating-system and programming-language package vulnerabilities as new CVEs appear, and Inspector maps images to the ECS tasks and EKS pods running them. Enhanced scanning emits an event to EventBridge, which you can route to SNS to alert on critical/high severity.

Trap Relying on basic scan-on-push when the requirement is continuous monitoring — basic scanning does not re-scan for newly disclosed CVEs.

4 questions test this

Use an ECR pull-through cache to dodge Docker Hub rate limits on EKS

An ECR pull-through cache rule mirrors images from an upstream registry (e.g. Docker Hub) into your private ECR and keeps them synced (checking upstream at most once per 24 hours per tag); after the first cached pull, subsequent pulls are served from your private registry so they no longer contact the upstream. For authenticated upstreams like Docker Hub, store credentials in a Secrets Manager secret whose name uses the ecr-pullthroughcache/ prefix and point the EKS manifests at the ECR registry URI.

4 questions test this

Also tested in

CLF-C02 AWS Certified Cloud Practitioner

The four SageMaker AI inference options

Real-time endpoint

Serverless inference

Asynchronous inference

Batch transform

Single, multi-model, and multi-container endpoints

Multi-model endpoints (MME)

Multi-container endpoints and inference pipelines

Choosing containers and compute for inference

Provided versus bring-your-own containers

CPU, GPU, and AWS Inferentia

SageMaker Neo for the edge

Reading the question: which deployment when

Signal phrases for the inference option

Signal phrases for endpoint topology and compute

The distractors that catch people

SageMaker AI inference options by traffic shape, payload, latency, and cost

Decision tree

Cheat sheet

Also tested in

References