Domain 3 of 5 · Chapter 4 of 4

FM Evaluation

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AWS Certified AI Practitioner AIF-C01 premium course — no separate purchase.

Included in this chapter:

  • Evaluation approaches: human, automated, benchmark
  • Metrics in depth + Amazon Bedrock model evaluation
  • Exam-pattern recognition: metric→task stems & traps

Automated FM evaluation metrics compared

MetricWhat it measuresBest-fit taskHigher is better?
ROUGERecall-oriented n-gram overlap between generated and reference textSummarizationYes
BLEUPrecision-oriented n-gram overlap with a brevity penaltyMachine translationYes
BERTScoreSemantic similarity via contextual embeddings (not exact tokens)Tasks where meaning matters more than exact wordingYes
PerplexityHow well a language model predicts a text sample (fluency, not correctness)Language-model fluency / fit on a datasetNo (lower is better)

Decision tree

Summarization task? how much reference content captured Yes ROUGE recall-oriented n-gram overlap No Machine translation? precision overlap vs reference Yes BLEU precision + brevity penalty No Semantic similarity to a reference? meaning matters, not exact words Yes BERTScore contextual-embedding match No Model fluency / uncertainty? how well it predicts a text sample Yes Perplexity lower = better; not correctness No Human evaluation subjective quality, safety, helpfulness Always: shortlist on benchmarks, then evaluate on your own task data Amazon Bedrock model evaluation runs both automatic and human jobs; tie every technical score to a business KPI Never: accuracy / precision / recall / F1 for open-ended generated text those are predictive-ML metrics; generative tasks use overlap, embeddings, perplexity, or human review
Cheat sheet

Cheat sheet

  • Automated metrics give scalable, objective, repeatable FM scoring
  • Human evaluation judges subjective and high-stakes quality
  • Automated and human evaluation are complementary, not interchangeable
  • ROUGE measures recall-oriented overlap, the metric for summarization
  • BLEU measures precision-oriented overlap, the metric for translation
  • BERTScore uses embedding similarity, so it credits valid paraphrases
  • Perplexity scores fluency, not correctness, and lower is better
  • Overlap metrics are higher-is-better; perplexity is lower-is-better
  • Classification metrics don't apply to open-ended generated text
  • Benchmarks shortlist models; representative data picks the winner
  • Tie technical evaluation scores to business metrics
  • Amazon Bedrock model evaluation compares and selects FMs on your data
  • Bedrock automatic evaluation scores objective task types at scale
  • Bedrock human evaluation: your own work team or an AWS-managed team
  • Evaluate RAG retrieval quality separately from generation quality
  • Evaluate agents on tool choice, ordering, and task completion
  • Amazon A2I adds managed human review with a sensitivity-driven workforce

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Evaluate the performance of Amazon Bedrock resources
  2. Automatic model evaluation jobs in Amazon Bedrock
  3. Human-based model evaluation jobs in Amazon Bedrock
  4. Amazon Bedrock Evaluations