An evergreen filter for AI news: four questions that tell an executive whether an announcement changes anything inside their company.
Most AI announcements describe what a system can do in a demo or on a benchmark. Executives need to know what it will change in their own company, and the gap between the two is large.
An NBER survey of nearly 6,000 executives in the US, UK, Germany, and Australia, fielded between November 2025 and January 2026, found that about 70% of firms use AI. Yet, nine in 10 executives report no impact on employment or productivity over the past three years. The same executives forecast that AI will raise their firms’ productivity by 1.4% over the next three years. Four questions can help close that gap before an announcement becomes a budget line.
1. Who Measured It?
Start with the source of the number. Announcements often rest on vendor-reported results or on how users feel, and METR’s randomized trial shows why that matters. Experienced open-source developers worked in their own repositories, with AI tools allowed for some tasks but not others. They expected AI to speed them up by 24%.
After finishing, they still believed it had sped them up by 20%. The measured result was a 19% slowdown. A claim built on perception, or on a vendor’s own test, deserves less weight than one that compares outcomes with and without the tool.
2. How Old Is the Evidence?
Evidence has a shelf life, and it can expire in either direction. In February 2026, METR revisited its own result and called the 2025 slowdown finding outdated. It said speedups now seem likely, while noting that changes in developer behavior made its new data unreliable. A rigorous study of last year’s tools does not describe this year’s, and a headline built on one result, favorable or not, may already be stale.
The date of the evidence should be checked against the date of the model or product being announced.
3. Does It Apply to Our Work?
Results rarely transfer evenly across tasks. A Harvard Business School study with Boston Consulting Group tested 758 consultants using GPT-4 in 2023. On 18 realistic tasks within the model’s capability, consultants using it completed 12.2% more tasks and finished 25.1% faster, with better quality. On a complex managerial task chosen to fall outside that capability, consultants using AI were 19% less likely to produce correct solutions than those without it.
The authors call this uneven boundary a jagged frontier, where tasks of similar apparent difficulty can land on opposite sides. METR makes a related point about its own study, which covers only software development. The test for an announcement is whether the demo tasks resemble the company’s own tasks, data, and tolerance for error.
4. What Would Change Inside, and Where Would It Show?
An announcement matters only if something in the company changes because of it, and only if the effect would appear in a number the company already tracks. The NBER survey shows what happens when that link is missing: adoption is widespread, measured effects are small, and expectations run ahead of both.
Before acting, an executive should be able to name the workflow that would change and who owns it, the metric that would move, by how much, and by when. A baseline should be set before any pilot starts. If no one can name the metric, the announcement has not yet become a decision.
The questions work best in order. An announcement that cannot answer the first two can wait. One that passes all four earns a bounded test, with a baseline and a date for judging the result.





