AI summarized from verified sources
Easier to predict model behavior using real deployment data beforehand
Streamlines pre-release risk assessment, making it easier to safely adopt new models in work.
SOURCE CHECK
1 sources
Sources
Key Points
- 1Simulates with production-like conversations
- 2Improved accuracy on 20 behavior types
- 3Supports agentic tool-use scenarios
OpenAI released Deployment Simulation. It replays past conversations with candidate models to predict rates of undesired behaviors. Provides signals closer to real usage than traditional evals. Uses anonymized data for privacy.
Key Points
Deployment Simulation removes original responses from past user conversations and regenerates them with the new model for analysis. It is closer to real deployment distribution and harder for models to detect as tests than traditional evals.
Impact
Higher accuracy in pre-release predictions makes it easier to understand real-world risks beyond rare events. This could reduce the effort needed to verify safety for business use.
What changed
OpenAI released Deployment Simulation. It replays past conversations with candidate models to predict rates of undesired behaviors. Provides signals closer to real usage than traditional evals. Uses anonymized data for privacy.