You cannot improve what you cannot see. These notes cover model invocation logging, how to attribute token usage to callers, the kinds of Bedrock evaluation jobs and what a RAG evaluation dataset must contain.
Model invocation logging is disabled by default. When enabled it sends full request and response data to CloudWatch Logs, S3 or both, for Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream calls made through the bedrock-runtime endpoint.
Official source: docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
Log entries hold input and output bodies up to 100 KB. Larger bodies and binary data such as images go to S3 as separate objects, and only S3 supports large payloads; Converse images and documents are logged to S3 only if delivery and image logging are enabled.
Official source: docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
Every log record carries identity.arn, so a CloudWatch Logs Insights query grouped by identity.arn shows token usage per IAM principal. Add requestMetadata tags to break usage down by team or environment. Blocked guardrail content also appears as plain text in these logs.
Official source: docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
Bedrock evaluations include programmatic (automatic) jobs, jobs that use a judge model (a second LLM scores and explains each response), jobs that use human workers, and RAG evaluations for knowledge bases and other RAG sources.
Official source: docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html
A RAG evaluation dataset holds the prompts plus ground truth: the expected retrieved text and responses. That lets the evaluation judge whether the knowledge base retrieves relevant information and generates useful answers.
Official source: docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html
Model evaluation jobs support foundation models, Marketplace models, customized and imported models, prompt routers and models that have Provisioned Throughput. Evaluation datasets and results live in S3, which needs the required CORS permissions.
Official source: docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html
Task 4.3 names GenAI-specific failure modes: golden datasets to detect hallucinations, output diffing for response consistency, and reasoning path tracing to find logical errors. These are not part of traditional ML monitoring.
Official source: docs.aws.amazon.com/aws-certification/latest/ai-professional-01/ai-professional-01-domain4.html
Answer practice questions, then review your mistakes with spaced repetition.