ChatGPT Voice feels more natural in live conversation Translate naturally during calls, meetings, and travel Easily automate multi-step daily tasks at lower cost Make Claude easier to deploy through AWS Claude Fable 5 is usable again after the pause Keep research tools in one place and move faster Delegate more everyday coding work to Claude Measure how well AI agents handle ambiguous biology research judgments Claude Sonnet 5 is built for heavier coding and work tasks HP partnership makes enterprise rollout easier Tag Claude in Slack to delegate tasks with your whole team Hand Slack tasks to Claude more easily Confidential AI gets stronger for sensitive workloads Helps defenders validate and fix vulnerabilities Gemini API key management is moving to safer auth keys Google Home Speaker makes home control feel natural Claude expands more easily into Korean businesses and research Anthropic expands Claude adoption and research in Korea Domain knowledge helps intermediate users succeed with Claude Code Easier to predict model behavior using real deployment data beforehand ChatGPT Voice feels more natural in live conversation Translate naturally during calls, meetings, and travel Easily automate multi-step daily tasks at lower cost Make Claude easier to deploy through AWS Claude Fable 5 is usable again after the pause Keep research tools in one place and move faster Delegate more everyday coding work to Claude Measure how well AI agents handle ambiguous biology research judgments Claude Sonnet 5 is built for heavier coding and work tasks HP partnership makes enterprise rollout easier Tag Claude in Slack to delegate tasks with your whole team Hand Slack tasks to Claude more easily Confidential AI gets stronger for sensitive workloads Helps defenders validate and fix vulnerabilities Gemini API key management is moving to safer auth keys Google Home Speaker makes home control feel natural Claude expands more easily into Korean businesses and research Anthropic expands Claude adoption and research in Korea Domain knowledge helps intermediate users succeed with Claude Code Easier to predict model behavior using real deployment data beforehand

Official sources only. Rumors, leaks, and get-rich schemes are excluded.

← Back to top

AI BriefingOpenAIFeature Updates20:45

AI summarized from verified sources

Stricter benchmarks needed to accurately track coding AI progress

Avoid overestimating AI performance for better model selection.

SOURCE CHECK

1 sources

VERIFIED

Sources

Primary / x.com

Official X

Key Points

1Many task issues found in SWE-Bench Pro
2Correct solutions failing due to hidden requirements
3Independent review by 5 engineers

OpenAI audited SWE-Bench Pro and found it saturated with noise, no longer reliable. They retracted their recommendation. Developing fairer benchmarks is now urgent.

What happened

OpenAI audited SWE-Bench Pro in detail, identifying issues like contradictory instructions and incomplete grading.

Impact

Prompts a review of AI coding evaluation methods and encourages development of more reliable benchmarks.