A benchmark map of the US economy | Mercor

The Benchmark MapEach tile represents one of the 923 O*NET occupations making up the US labor force, grouped into 22 job families. Occupations are shaded according to the share of their tasks which are represented in our initial list of AI benchmarks. A single benchmark prompt can be mapped to multiple O*NET tasks.All 22 job familiesclick a family to open itCoverage byCovered with at leastBenchmark IndexNo coverageAll 22 O*NET job families, with the share of their occupations covered by an AI benchmark.Job familyOccupations covered29 / 3633 / 4810 / 145 / 721 / 5121 / 5619 / 566 / 2213 / 4015 / 6012 / 899 / 621–12 of 22In our data set: only 208 of 923 occupations have a task with at least 5 benchmark prompts, and 6 of 22 job families have no such occupations.Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.The coverage of existing benchmarks is not random, either. Our map suggests that the AI community is focusing on the largest labor markets first:Benchmark coverage of O*NET tasks by occupation wage billAdjusted probability of task coverageAnnualized occupation wage bill (log scale)Each point shows the estimated probability that an O*NET task with a given AI-exposure tag belonging to an occupation with a given wage-bill will be covered by at least 5 prompts in our benchmark index. AI-exposure tags are binary and come from Eloundou et al. (2023). Occupation wage-bills are calculated from BLS estimates of the mean wage and number of Americans working in each occupation, assuming a 40 hour work week. Estimated coverage probabilities come from a task-level logistic regression of coverage probability on AI-exposure and log occupational wage bill. Error bars are pointwise 95% delta-method confidence intervals based on standard errors clustered at the occupation group level.For example, benchmarks in our initial list heavily cover AI-exposed lawyers’ tasks, but barely cover any of the O*NET tasks of similarly AI-exposed legal secretaries or court clerks. The long tail of smaller, middle-income knowledge work jobs appears undercovered.As we build out our coverage of model capabilities across the economy, we hope to both surface and help fill more gaps in our understanding of what models can do.Methodology in briefTo break down the economy into sectors, occupations, and tasks, we used the US Department of Labor’s O*NET taxonomy. O*NET divides the US workforce into 923 occupations; for each occupation, O*NET attempts to provide a comprehensive list of “tasks” which make up the occupation, totaling 18,797 tasks across all occupations.Example — 6 of the 22 tasks O*NET associates with Lawyers1Prepare, draft, and review legal documents, such as wills, deeds, patent applications, mortgages, leases, and contracts.2Advise clients concerning business transactions, claim liability, advisability of prosecuting or defending lawsuits, or legal rights and obligations.3Interpret laws, rulings and regulations for individuals and businesses.4Analyze the probable outcomes of cases, using knowledge of legal precedents.5Present evidence to defend clients or prosecute defendants in criminal or civil litigation.6Supervise legal assistants.To measure benchmark coverage, we start with a list of 29 benchmarks selected for their quality and economic relevance. (The full list and selection criteria are in the Benchmark Index tab). In selecting benchmarks for this list, we prioritized:Public Availability. Included benchmarks have at least 25 publicly available tasks. We also include Mercor’s APEX benchmarks, where we have access to held-out benchmark prompts and are therefore able to map them to O*NET tasks.Agentic Tasks. Included benchmarks ask models to perform agentic tasks, rather than answering questions or advising the user on how to complete a task.Economic Realism. Benchmark tasks resemble tasks which humans (or agents) are frequently asked to do in real-world enterprises. Economically valuable tasks are prioritized over academically complex ones.Economic Diversity. We capped the number of included benchmarks per job family, in significant part to account for the massive overrepresentation of software engineering and research tasks in popular benchmarks.Balanced Difficulty. Benchmarks are not yet saturated by frontier models.Following work by Wang et al., we then ask an LLM judge to tag each benchmark prompt with as many O*NET tasks as it thinks correspond to that prompt. We validate our LLM tagging with human expert tags on Mercor’s APEX benchmarks, a subset of this list of 29. On the Benchmark Map, an O*NET task counts as “covered” if our LLM judge has mapped it to at least 5 prompts from our 29 benchmarks.To estimate the total share of working hours Americans spend on tasks covered by benchmarks on this list, we combine our coverage estimates with Hatgis-Kessell et al.’s LLM-derived estimates of the working time share of each O*NET task.A more detailed account of our methodology can be found in the Methodology tab.LimitationsOur estimates of total economic coverage, and of patterns in coverage, all come from the initial list of 29 benchmarks we chose. This is not a complete or perfectly representative sample of the broader universe of unsaturated, public, agentic economic benchmarks. Any incomplete list will necessarily bias down estimates of economic coverage, and it is difficult to bound that bias. We want to stress, therefore, that these results are provisional. We will update them as we expand the initial benchmark list.We also capped the number of included benchmarks in each job family in order to maximize breadth of coverage. We think this makes our estimate of economy-wide benchmark coverage more accurate, at the cost of biasing down our estimates of coverage in the most-benchmarked job families (e.g. software engineering).By selecting for unsaturated benchmarks, we risk understating benchmark coverage on O*NET tasks which are easier for LLMs to complete. On the other hand, including saturated benchmarks would risk overstating benchmark coverage if saturation is downstream of lower task realism. The latter claim seems likely, given that benchmarks often overstate LLM performance in real-world deployments. We think that the bias from including saturated benchmarks would be greater than the bias from excluding them, but neither option is ideal.Finally, while we outline our benchmark selection criteria in the Benchmark Index tab, several criteria (like “Economic Realism”) are difficult to formalize and introduce some subjectivity in our choices.Patterns in coverageThe benchmarks in our index have relatively deep and narrow coverage of the overall economy. They disproportionately focus on the economic tasks most exposed to AI automation (as measured by Eloundou et al.), with O*NET tasks rated as “exposed” about 6x more likely to be represented on benchmarks than tasks rated “unexposed”. This makes sense: benchmarks are designed to measure AI progress, and AI progress is most plausible on the most AI-exposed tasks. This explains why benchmarks are full of tasks in Software Engineering and Financial Analysis while sparse on tasks in Truck Driving and Gas Plant Operation.But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:Benchmark coverage of O*NET tasks by occupation wage billAdjusted probability of task coverageAnnualized occupation wage bill (log scale)Each point shows the estimated probability that an O*NET task with a given AI-exposure tag belonging to an occupation with a given wage-bill will be covered by at least 5 prompts in our benchmark index. AI-exposure tags are binary and come from Eloundou et al. (2023). Occupation wage-bills are calculated from BLS estimates of the mean wage and number of Americans working in each occupation, assuming a 40 hour work week. Estimated coverage probabilities come from a task-level logistic regression of coverage probability on AI-exposure and log occupational wage bill. Error bars are pointwise 95% delta-method confidence intervals based on standard errors clustered at the occupation group level.After controlling for occupation and AI-exposure score, O*NET “task importance” is another predictor of benchmark coverage. In particular, O*NET tasks come with domain expert estimates of the importance of the task for its occupation on a scale of 1-5. We find that the tasks scored most important are about 6–10x more likely to be represented on benchmarks than those scored least important:Benchmark coverage by O*NET task importance within occupationsAdjusted probability of task coverageO*NET task importance (1–5 rating scale)Each point shows the estimated probability that, controlling for occupation, an O*NET task with a given AI-exposure tag and a given quintile O*NET importance score is covered by at least 5 prompts in our benchmark index. AI-exposure tags are binary and come from Eloundou et al. (2023). Importance quintiles are calculated from O*NET importance scores. Estimated coverage probabilities come from a task-level logistic regression of coverage probability on AI-exposure and importance quintiles, with occupation fixed effects. Error bars are pointwise 95% delta-method confidence intervals based on standard errors clustered at the occupation group level.The plots above suggest a reasonable allocation of the relatively limited resources devoted to high-quality, economically relevant benchmarks so far; it’s natural to prioritize making benchmarks which cover the most important and AI-exposed tasks in the occupations with the largest wage bills. But barring a significant increase in the number and diversity of such benchmarks, these patterns will result in significant blind spots in our view of economically valuable model