AI World 5 · Serve and Secure · Stage 5

Test in Production

A new container with no user impact, a 90/10 split, QA on the new variant, a canary that rolls back, yesterday's model back in a minute, and a serverless model's candidates.

Shadow tests
Production variants and A/B
Deployment guardrails

Domain: AI and machine learning
World: AI World 5: Serve and Secure (Associate)
Game type: Solution board: Drop services into each row's slots, then run records through them.
Time: About 25 minutes on a first attempt (estimated)
Exam guides: MLA-C02Skill 2.3.3: Compare the performance of shadow variants to production variants; Skill 3.3.1: Implement automated deployment strategies and rollback actions; Skill 4.1.4: Monitor model performance in production by using A/B testing
AIP-C01Skill 5.1.2: Create systematic model evaluation systems to identify optimal configurations (for example, by using Amazon Bedrock Model Evaluations, A/B testing and canary testing of FMs, multi-model evaluation, cost-performance analysis); Skill 5.1.9: Create deployment validation systems to maintain reliability during FM updates (for example, by using synthetic user workflows, AI-specific output validation for hallucination rates and semantic drift, automated quality checks to ensure response consistency)