Domain 4 of 5 · Chapter 2 of 2

Observability for Generative AI Applications

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Operationalizing Machine Learning and Generative AI Solutions premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Three observability pillars on one backend
  • Production monitoring: quality and operational health
  • Cost optimization is token telemetry
  • Distributed tracing with OpenTelemetry
  • Exam-pattern recognition

Foundry observability: three pillars on one backend

DimensionEvaluationMonitoringTracing
Question it answersIs the output good and safe?Is production healthy right now?Which step caused this response?
Runs whenIn development, plus sampled or scheduled in productionContinuously in productionPer request, on every run
Primary signalsQuality and safety scores (groundedness, coherence, violence)Latency, throughput, error rate, token cost, quality scoresSpans: LLM calls, tool calls, retrievals, token usage
Backing storeAzure Monitor Application InsightsAzure Monitor Application InsightsAzure Monitor Application Insights
Setup gateEnable evaluators or eval rulesConnect Application Insights, enable metricsAssociate Application Insights, instrument the SDK

Decision tree

Response bad or unsafe?Aggregate signal, orone specific response?Slow / unavailable,or costly?Evaluationcontinuous +scheduledTracingopen span-by-spanMonitoringlatency, throughput,error rateMonitoringtoken usage + cost,prompt cachingAll three store telemetry in Azure Monitor Application Insights (query with KQL)YesNoAggregateOne responseSpeedCost
Cheat sheet

Cheat sheet

  • Foundry observability = evaluation + monitoring + tracing
  • Continuous evaluation samples live traffic; scheduled evaluation detects drift
  • Azure Monitor alerts fire on quality-threshold or harmful-content breaches
  • Monitoring tracks latency, throughput, and error rate in real time
  • Metrics are queryable in Application Insights / Log Analytics
  • Token usage is captured as gen_ai.usage input/output tokens
  • Cost signal differs for Standard versus Provisioned deployments
  • Prompt caching discounts repeated prompt prefixes to cut token cost
  • Tracing is OpenTelemetry-based and needs an associated Application Insights
  • Instrument with configure_azure_monitor and an OpenTelemetry instrumentor
  • Tracing supports major agent frameworks
  • Prompt/response content is captured only when you opt in
  • Add custom spans and attributes; export to console for CI/CD
  • Tracing connects to the evaluation loop for root-cause debugging

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Observability in Generative AI - Microsoft Foundry
  2. Application Insights OpenTelemetry observability overview - Azure Monitor
  3. Microsoft Foundry Tracing and Data Handling - Microsoft Foundry
  4. Monitor agents with the Agent Monitoring Dashboard - Microsoft Foundry
  5. Overview of Azure Monitor alerts - Azure Monitor
  6. View Trace Results for AI Applications using OpenAI SDK (classic) - Microsoft Foundry (classic) portal
  7. Prompt caching with Azure OpenAI in Microsoft Foundry Models - Microsoft Foundry
  8. Provisioned throughput for Foundry Models - Microsoft Foundry
  9. Set Up Tracing for AI Agents in Microsoft Foundry - Microsoft Foundry