AIP-C01 Domain 5: Testing, Validation, and Troubleshooting (11%)
Domain 5 is worth 11% of the AIP-C01 exam. It tests whether you can evaluate the quality of FM outputs with systematic methods, test retrieval and agents, report results, and troubleshoot context, integration, prompt and retrieval problems.
Weight on the exam: 11%. It is one of the five domains of the AWS Certified Generative AI Developer – Professional AIP-C01 exam. Below is a summary of its tasks and skills according to the official AWS guide.
Tasks and skills tested
Task 5.1: Implement evaluation systems for GenAI
- Assess FM output quality beyond traditional ML metrics: relevance, factual accuracy, consistency and fluency.
- Identify the best configuration with Bedrock Model Evaluations, A/B and canary testing, multi-model evaluation and cost-performance analysis.
- Improve continuously from user feedback with feedback interfaces, ratings and annotation workflows.
- Maintain standards with continuous evaluation, regression testing of outputs and automated quality gates for deployments.
- Evaluate from several perspectives with RAG evaluation, LLM-as-a-judge and human feedback collection.
- Test retrieval quality with relevance scoring, context matching verification and retrieval latency measurements.
- Measure agent performance with task completion rate, tool usage effectiveness and reasoning quality in multi-step workflows.
- Report results with visualization tools, automated reporting and model comparison visualizations.
- Validate deployments with synthetic user workflows, hallucination and semantic drift checks and automated consistency checks.
Task 5.2: Troubleshoot GenAI applications
- Resolve content handling problems with context window overflow diagnostics, dynamic chunking, prompt design optimization and truncation error analysis.
- Diagnose FM integration issues with error logging, request validation and response analysis.
- Fix prompt engineering problems with prompt testing frameworks, version comparison and systematic refinement.
- Troubleshoot retrieval with response relevance analysis, embedding quality diagnostics, drift monitoring, vectorization fixes, chunking remediation and vector search tuning.
- Troubleshoot prompt maintenance with template testing, CloudWatch Logs, X-Ray prompt observability pipelines, schema validation and systematic refinement workflows.
Services and tools to master
- Amazon Bedrock Model Evaluations
- Amazon Bedrock Agent evaluations
- LLM-as-a-judge
- Amazon Bedrock Knowledge Bases
- Amazon Bedrock Prompt Management
- Amazon CloudWatch
- Amazon CloudWatch Logs
- AWS X-Ray
- AWS CodePipeline
- AWS Step Functions
- AWS Lambda
- Amazon OpenSearch Service
Key facts for domain 5
Short summaries; each links to the study topic with the full note and its official source.
- Types of Bedrock evaluation jobs — Bedrock evaluations include programmatic (automatic) jobs, jobs that use a judge model (a second LLM scores and explains each response), jobs that use human workers, and RAG evaluations for knowledge bases and other RAG sources.
- RAG evaluation needs ground truth — A RAG evaluation dataset holds the prompts plus ground truth: the expected retrieved text and responses.
- What can be evaluated — Model evaluation jobs support foundation models, Marketplace models, customized and imported models, prompt routers and models that have Provisioned Throughput.
- Metadata filters need the faiss engine — If metadata filtering fails on an OpenSearch Serverless vector index, check the engine: the index must use faiss, not nmslib.
- Fewer results than requested — numberOfResults is a maximum. With hierarchical chunking it counts child chunks, and children that share a parent are replaced by that one parent, so you can get fewer results.
- Guardrail trace shows the original PII — In the guardrail trace (for example in a Converse response) the match field contains the original detected PII, not the masked value.
- Hallucination testing — Task 4.3 names GenAI-specific failure modes: golden datasets to detect hallucinations, output diffing for response consistency, and reasoning path tracing to find logical errors.
Guides that go deeper
Practice Domain 5
Answer practice questions, read the explanation and review your mistakes with spaced repetition.