Optimize RAG Performance and Accuracy

A retrieval-augmented generation (RAG)^[1] system answers a question in two stages: it retrieves passages from your own documents, then generates an answer from them. Optimizing one stage is a different job from optimizing the other, and the fastest way to waste a tuning cycle is to fix the wrong one.

Think of the pipeline as two halves. The retrieval half turns documents into searchable vectors and pulls back candidates: chunk the text, embed each chunk, store the vectors in an index, then run a query that retrieves and optionally reranks. The generation half is the prompt and the model that read those candidates and write the reply. Every lever on this page lives in one half or the other, and the figure traces a query through both, previewing the concrete tool at each retrieval step — the Text Split skill, a text-embedding-3 model, and an HNSW index — that the sections below unpack.

The three built-in RAG evaluators^[2] turn "which half is broken" from a guess into a measurement, each scoring a response from 1 to 5. Retrieval rates how well the retrieved chunks rank the relevant context; Groundedness rates whether the answer is supported by that context without fabricating; Relevance rates whether the answer addresses the question. Read them together: a low Retrieval score is a retrieval-half problem, while a high Retrieval score paired with low Groundedness or Relevance is a generation-half problem, because the right context was fetched and the answer still missed it.

Fix retrieval first

When Retrieval is the weak metric, tune retrieval before you touch the prompt. No prompt rewrite can ground an answer on context that was never retrieved, so a prompt change layered on top of a retrieval failure just moves noise around. The rest of this page walks the retrieval half in pipeline order (chunking, embeddings, vector search, then hybrid and semantic ranking) and returns to measurement and A/B testing at the end, which is where generation-half fixes get validated too.

Every embedding model has a hard input ceiling, and that ceiling is why chunking is the first retrieval lever. The Azure OpenAI text-embedding-3 models accept at most 8,191 tokens per call^[3], so a document longer than that cannot be embedded whole; it has to be split first. In Azure AI Search integrated vectorization^[4], an indexer pulls documents from a data source and runs a skillset that does this automatically: the Text Split skill^[5] breaks each document into chunks, and an embedding skill (for example the Azure OpenAI Embedding skill) vectorizes each chunk. The figure shows that indexing pipeline, from data source through indexer and skillset to the index.

Sizing the chunk: precision versus context

Chunk size is a trade-off, and it is the single setting that most changes retrieval quality. Smaller chunks return more precise matches but can strip the surrounding context a passage needs to make sense; larger chunks keep the context but dilute relevance, since one vector now has to represent several ideas, and they cost more to embed. Microsoft recommends starting near a 512-token chunk with about 25 percent overlap, roughly 128 tokens, then tuning from there per the chunking guidance^[3].

Overlap keeps passages whole

Overlap repeats a slice of text at each chunk boundary so a sentence that straddles two chunks is not cut in half. Setting overlap to 0 to avoid duplicate text is a false economy: a relevant passage split across two chunks lowers recall, because neither chunk now embeds the whole idea. In the Text Split skill, the pageOverlapLength parameter sets that repeated slice, and it must be less than half of maximumPageLength.

Text Split skill parameters (integrated vectorization)

{
  "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
  "textSplitMode": "pages",       // "pages" = fixed-size chunks (default); "sentences" = one sentence each
  "maximumPageLength": 512,        // max characters or tokens per chunk
  "pageOverlapLength": 128,        // repeated text at each boundary; must be < half of maximumPageLength
  // ... input (document text) and output (chunked pages) field bindings omitted
}

The textSplitMode key chooses the strategy: pages (the default) packs multiple sentences up to maximumPageLength, while sentences emits one sentence per chunk. maximumPageLength caps the chunk size, and pageOverlapLength is the overlap slice named above. The // ... comment marks the input and output field bindings left out for brevity. Because the whole split-then-embed flow runs inside one skillset, the chunk size you set here is exactly what the embedding model sees on the next step.

The embedding model decides how much meaning each vector can carry, and unlike a chat model you tune it by selection, not training. text-embedding-3-large^[6] outputs 3,072-dimensional vectors by default and text-embedding-3-small outputs 1,536; the older text-embedding-ada-002 is fixed at 1,536. A larger, higher-dimensional model captures more semantic nuance and tends to raise domain accuracy, at the price of more vector storage and slower, costlier queries.

The dimensions parameter and Matryoshka truncation

The v3 models add a dimensions parameter that shortens the output vector, for example from 3,072 down to 1,536 or 256, trading a small accuracy loss for lower storage and faster search. This works because the models are trained with Matryoshka Representation Learning, which front-loads the most important information into the first components of the vector, so a truncated vector still represents the concept. One rule makes or breaks it: the same dimensions value must be used at indexing time and at query time, or the index and query vectors have different lengths and cannot be compared. Reducing dimensions only on the query side silently breaks the similarity comparison.

You select and tune; you do not fine-tune

Azure OpenAI embedding models cannot be fine-tuned; fine-tuning applies to chat and completion models only. To raise domain-specific retrieval accuracy you pick the right model and dimension count, improve chunking, and add hybrid and semantic ranking, rather than training the embedder on your data. Fine-tuning as a customization path for generation models is a separate topic, covered in advanced fine-tuning.

Match the similarity metric to the model

A vector field's similarity metric must match how the model was trained. Azure OpenAI embeddings are trained for cosine similarity^[7], so cosine is the default and correct choice; dotProduct and euclidean are the alternatives, and switching to euclidean to "improve" OpenAI-embedding relevance actually degrades it. Cosine compares the angle between two vectors and is unaffected by their length, which is why documents of very different sizes still compare fairly.

Model	Default dimensions	Fine-tunable
text-embedding-3-large	3,072 (adjustable via dimensions)	No
text-embedding-3-small	1,536 (adjustable via dimensions)	No
text-embedding-ada-002	1,536 (fixed)	No

Once chunks are embedded and indexed, retrieval quality comes down to the search algorithm and two query knobs. The vector field's algorithm sets the strategy. HNSW (Hierarchical Navigable Small World) runs an approximate nearest-neighbor (ANN) search^[8]: it builds a layered proximity graph at indexing time and navigates it at query time, which scales to large indexes with low latency. Exhaustive KNN (k-nearest neighbors) instead scans every vector for the exact nearest neighbors, which is more accurate but does not scale. The default for a large production index is HNSW; reach for exhaustive KNN only on small or medium data, or to build a ground-truth set for measuring an ANN index's recall. Choosing exhaustive KNN for a large index just because it is "exact" is the classic scaling mistake. An HNSW field can still be forced to run an exact scan for a single query with "exhaustive": true.

HNSW exposes three tuning parameters: m (the number of neighbor links per node), efConstruction (how many candidates are considered while building the graph, default 400, range 100 to 1,000), and efSearch (how many candidates are held while searching). Higher values raise recall at the cost of index size, latency, or both.

k sets recall, and always returns k

The k parameter is the number of nearest neighbors a vector query returns. Raising k hands the reranker or the model more candidates, which raises recall but also latency and noise. The behavior that surprises people: a pure vector query always returns k results^[9], even when the nearest neighbors are barely similar, because nearest-neighbor search just finds the closest vectors regardless of how close they actually are. Do not confuse k with the embedding dimension count; k is the number of documents returned, unrelated to the 1,536 or 3,072 vector dimensions. When a vector query feeds semantic ranking, Microsoft recommends setting k to 50 so the reranker receives a full candidate set.

A similarity threshold is the only weak-match filter

Because k always returns k, neither a smaller k nor a different similarity metric will remove off-topic chunks; the only lever that does is a per-query similarity threshold. Setting a threshold of kind vectorSimilarity (in preview) excludes any result scoring below the minimum even if fewer than k results survive, so weakly related chunks never reach the generator. This is a precision lever distinct from top-k and from the metric. One boundary: apply it only to single-vector queries. Hybrid queries fuse ranks with Reciprocal Rank Fusion (RRF) — covered in Hybrid search and semantic ranking below — and those fused rank ranges are too small and volatile for a raw score cutoff, so the threshold does not apply to them.

A single-vector query with k, exhaustive, and a similarity threshold

{
  "vectorQueries": [
    {
      "kind": "vector",
      "fields": "contentVector",
      "vector": [ /* ... query embedding, elided ... */ ],
      "k": 50,                 // return 50 nearest neighbors (full input for semantic ranking)
      "exhaustive": false,     // false = use the HNSW graph; true = exact scan for this query
      "threshold": { "kind": "vectorSimilarity", "value": 0.8 }  // drop matches below 0.8 (single-vector only, preview)
    }
  ]
}

Here k caps the result count at 50, exhaustive chooses approximate versus exact search for this one query, and the threshold block drops weak matches by score. The /* ... */ marks the embedding array elided for brevity. This is a single-vector query, which is the only query shape where threshold is valid.

Vector search alone misses exact-keyword hits (a product code, an error string, a rare name) because those tokens may not sit close in embedding space. Hybrid search fixes that by running the vector (HNSW) query and a BM25 keyword query in parallel over one index, then merging their two ranked lists. BM25 is the classic keyword-ranking function; it scores on term frequency, so it nails the literal matches the vector side can paraphrase past. Hybrid retrieval^[10] usually beats either method alone, especially for queries that mix exact terms with semantic intent. The figure traces a query through both stages.

RRF merges by rank, not by score

The merge uses Reciprocal Rank Fusion (RRF)^[11], and the key idea is that it combines results by rank position, not raw score. BM25 scores have no upper bound while cosine scores sit in a small range, so adding or averaging the two would let one scale dominate. Instead RRF gives each document a score of 1 / (rank + k) in each list (where this k is a small RRF constant, about 60, entirely separate from the k-nearest-neighbors count) and sums those across the lists, so a document ranked highly by both the vector and the keyword query rises to the top. RRF never adds or averages the BM25 and cosine scores themselves.

Semantic ranker: an L2 rerank of the top 50

Setting queryType=semantic adds a semantic ranker^[12] pass on top. It takes the first-stage BM25 or RRF result and applies a second-level (L2) rerank using Microsoft's language models, rescoring the top 50 candidates and returning @search.rerankerScore from 4.0 (highly relevant) down to 0.0 (irrelevant). The load-bearing limit: it reranks only those existing top 50 candidates and never re-queries the corpus, so first-stage recall still depends entirely on the vector and keyword query. Expecting the reranker to surface a document the first-stage query missed is the most common mistake here; raise recall in the retrieval query (a bigger k, hybrid search, better chunks), not in the reranker.

Captions, answers, and query rewrite

Beyond reranking, the semantic ranker returns verbatim semantic captions (usually under 200 words) and optional semantic answers extracted from indexed text, and with query rewrite enabled it expands a query into up to 10 semantically similar variants that each run and are rescored. One caveat readers get wrong: captions and answers are always verbatim from your index, so no generative model composes them, and if the indexed text contains no answer-like passage, none is produced.

Every lever above is only worth pulling if you can measure its effect, so RAG tuning ends where it began: with the same three RAG evaluators^[2] — Retrieval, Groundedness, and Relevance — that How RAG optimization works defined at the top of this page. What that section left for here is where they run and how to read the number: the Azure AI Evaluation SDK and Foundry evaluations host them, each returns its 1-to-5 score against a default pass threshold of 3, and you run them together, often alongside content-safety evaluators, for a full quality picture. Groundedness and Relevance are the same RAG pair introduced in generative AI evaluation and validation.

The evaluators localize the fault

The scores are a map to the broken component. A low Retrieval score points at the retrieval half (chunking, the embedding model or dimensions, k, or hybrid search). A high Retrieval score with low Groundedness or Relevance points at the generation half (the prompt or the model), because the right context was retrieved and the answer failed to use it. So the fix order is fixed: when Retrieval is the weak metric, repair retrieval first. Rewriting the prompt while Retrieval is low cannot help, since no prompt change grounds an answer on context that was never fetched.

A/B test one variable on a fixed dataset

To actually improve the system, run A/B tests^[13] that change one variable at a time (chunk size, overlap, k, embedding model or dimensions, hybrid on or off, semantic ranker on or off) and score every variant on the same evaluation dataset, then keep the configuration with the higher aggregate scores. A shared, controlled test set is what makes the comparison valid; comparing two configurations on different query sets proves nothing. If production queries are unavailable, a synthetic-but-consistent test set still works, because consistency, not realism, is what the comparison requires. The evaluate() API takes the dataset as a file path: JSON Lines (JSONL) is the standard format, and the SDK reference also lists CSV as accepted.

Exam-pattern recognition

AI-300 questions on this objective usually hinge on one distinction. Read the stem for the signal:

• A low Retrieval score in the stem means fix indexing or the query (chunking, embedding model or dimensions, k, hybrid), never the prompt.
• High Retrieval, low Groundedness or Relevance means fix generation (the prompt or the model); the context was already retrieved.
• "Irrelevant chunks reach the model" points to a vector-similarity threshold on a single-vector query, not a smaller k or a different metric.
• "Queries mix exact terms and meaning" points to hybrid search with RRF; "reorder for intent" or "the top results are in the wrong order" points to the semantic ranker.
• "Surface a document the query missed" is a recall problem in retrieval; the semantic ranker only reorders the top 50 it was handed.
• "Raise domain accuracy on the embedding model" means select a larger model or tune the dimension count; it never means fine-tune the embedder.
• "Compare two configurations fairly" means change one variable at a time and score on one fixed evaluation dataset.