Generative AI & LLMs

LLM Observability & Tracing

9 practice questions. Free questions open a full answer guide; the rest unlock with Pro.

  • Your RAG service is over budget on inference spend and its p95 latency is too high, but nobody can say which step is responsible. How would you use tracing to attribute cost and latency, and then decide where to cut? Senior level
  • An LLM feature that retrieves documents and then answers is giving a wrong answer. How does tracing the request help you tell whether the problem is in retrieval or in the generation step?Go Pro Junior level
  • Offline evals pass before every release, but once an LLM feature is live nobody knows whether quality is holding up day to day. How would you build continuous quality measurement into your production observability rather than waiting for user complaints?Go Pro Senior level
  • What does "LLM observability" mean, and why isn't a normal application request log enough once an LLM feature is in production?Go Pro Junior level
  • Why are teams standardizing LLM tracing on the OpenTelemetry GenAI semantic conventions instead of each tool inventing its own trace format?Go Pro Junior level
  • You're putting a multi-step RAG pipeline into production and the team can only debug failures by re-running prompts by hand. How would you design the observability and tracing layer so you can actually diagnose a bad answer after the fact?Go Pro Senior level
  • You want to score the quality of your LLM feature's real production traffic, not just a fixed test set. How would you set up online evaluation on top of your tracing, and what would you watch out for?Go Pro Mid level
  • A user reports that an agent gave a wrong final answer on a long, multi-step task. Walk me through how you'd use traces to find the root cause, and why metrics alone usually aren't enough here.Go Pro Mid level
  • You're adding tracing to an LLM feature so you can debug it in production. What would you instrument in a trace, and how would you structure the spans for a request that does retrieval, a tool call, and a model generation?Go Pro Mid level
Want questions matched to your role? Paste a job title, job description, or CV for a personalized set, or go Pro to unlock the full bank.