Start natural voice conversations anytime with GPT-Live Talk naturally while answers are still coming Track the latest safety rules for bigger models Translate naturally during calls, meetings, and travel Easily automate multi-step daily tasks at lower cost Make Claude easier to deploy through AWS Claude Fable 5 is usable again after the pause Keep research tools and analysis in one place Keep research tools in one place and move faster Delegate more everyday coding work to Claude Measure how well AI agents handle ambiguous biology research judgments Claude Sonnet 5 is built for heavier coding and work tasks HP partnership makes enterprise rollout easier Tag Claude in Slack to delegate tasks with your whole team Hand Slack tasks to Claude more easily Confidential AI gets stronger for sensitive workloads Gemini API key management is moving to safer auth keys Google Home Speaker makes home control feel natural Claude expands more easily into Korean businesses and research Anthropic expands Claude adoption and research in Korea Start natural voice conversations anytime with GPT-Live Talk naturally while answers are still coming Track the latest safety rules for bigger models Translate naturally during calls, meetings, and travel Easily automate multi-step daily tasks at lower cost Make Claude easier to deploy through AWS Claude Fable 5 is usable again after the pause Keep research tools and analysis in one place Keep research tools in one place and move faster Delegate more everyday coding work to Claude Measure how well AI agents handle ambiguous biology research judgments Claude Sonnet 5 is built for heavier coding and work tasks HP partnership makes enterprise rollout easier Tag Claude in Slack to delegate tasks with your whole team Hand Slack tasks to Claude more easily Confidential AI gets stronger for sensitive workloads Gemini API key management is moving to safer auth keys Google Home Speaker makes home control feel natural Claude expands more easily into Korean businesses and research Anthropic expands Claude adoption and research in Korea

Official sources only. Rumors, leaks, and get-rich schemes are excluded.

← Back to top

AI BriefingOpenAIPolicy21:41

AI summarized from verified sources

AI coding evaluation reliability drops, making alternatives easier to consider

Easier to choose evaluation methods that fit real work instead of relying on outdated benchmarks.

SOURCE CHECK

1 sources

VERIFIED

Sources

Primary / x.com

Official X

Key Points

130% of tasks have issues
2Retracted recommendation for researchers
3Need for stricter evaluations

OpenAI audited SWE-Bench Pro and found 30% of tasks broken due to hidden requirements, contradictory instructions, or flawed tests. It retracted its prior recommendation as a leading coding benchmark. Researchers and developers are likely to seek more reliable alternatives.

What happened

OpenAI audited SWE-Bench Pro tasks using model agents and 5 expert engineers, finding 30% broken or with incomplete grading that could distort results.

Impact

The benchmark's reliability for comparing frontier models has declined. The retraction right after new results from xAI has sparked industry discussion on benchmarks.