Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning

Today, we are releasing OfficeQA Pro V2, a new benchmark designed to evaluate whether AI agents can generalize to unfamiliar, enterprise-style grounded-reasoning tasks.

Seven months ago, we introduced the OfficeQA benchmark to measure how well AI systems answer analytical questions using evidence from large document collections, an extremely common and important enterprise task that we found agents struggled with. Since its introduction, OfficeQA, and its frontier subset, OfficeQA Pro, has become an important measure for frontier model and agent capabilities, driving progress in document retrieval, parsing, and analytical reasoning. But this progress raises a fundamental question: do these improvements reflect broader advances in grounded reasoning, or progress specific to one corpus and task distribution? This distinction matters in enterprise settings, where agents rarely operate on a single, stable document collection.

Our new benchmark, OfficeQA Pro V2, is designed to test that generalization directly. We initially developed OfficeQA Pro V2 as the benchmark for the inaugural Databricks Grounded Reasoning Cup, a live competition in which 11 academic teams, supported by OpenAI, Anthropic, and Google DeepMind, built agents that were evaluated on a previously unseen corpus and set of grounded-reasoning tasks. The benchmark contains 90 questions grounded in approximately 120,000 pages from the U.S. Treasury’s Accounts of Receipts and Expenditures, which was released as a cohesive dataset by the U.S. Treasury for the first time in conjunction with the 250th anniversary of the United States. Now, we are excited to release the benchmark to AI practitioners more broadly to use for their own development.

OfficeQA Pro V2 remains challenging for today’s AI systems. Out-of-the-box frontier agents using Claude Code with Claude Opus 4.8 and Claude Fable 5, or Codex with GPT-5.5 and GPT-5.6 Sol, achieved an average accuracy of 37.5%. Agents developed specifically for the Grounded Reasoning Cup performed better, averaging 41.1%, with the winning team reaching 63.3%. We also evaluated Databricks Genie, Databricks’ AI agent designed to answer complex questions over enterprise data, using the same underlying models. Genie, which also uses Databricks’ ai_parse to pre-parse the document corpus, improved accuracy by an average of 24.0 percentage points over the default harnesses, with the strongest configuration reaching 60%. Together, these results show that grounded reasoning remains far from solved, but that the right agent harness can unlock significant gains from existing frontier models.

Below, we describe in more detail how agents perform on OfficeQA Pro V2, how it was built, how it differs from the original OfficeQA benchmark.

Agent Performance on OfficeQA Pro V2

We first evaluated frontier models using each model provider’s associated harness, with Codex for GPT models and Claude Code for Claude models, using the same deterministic exact-match metric with 0.0% tolerance metric as used for OfficeQA. Across the five models evaluated under both harnesses, these baseline configurations achieved 26.0% accuracy on average. Performance varied substantially by model, and higher cost did not consistently translate into greater accuracy. For example, Sonnet 5 on Claude Code scored 15.6% at $5.01 per rollout, while GPT-5.6 Sol on Codex reached 33.3% at a comparable $4.70.

Using these models within Genie produced substantial gains. Across matched model comparisons, Genie improved accuracy by 24.0 percentage points on average (a 92% relative improvement). Genie configurations using GPT-5.6 Luna, GPT-5.6 Terra, and Claude Fable 5 also dominate the cost-quality Pareto frontier, demonstrating that these improvements do not require trading efficiency for accuracy. In the case of Claude Fable 5, switching to Genie improves performance by 14.4 percentage points (+32% relative) while reducing cost by ~9x. From inspecting traces, we find that Fable 5 is prone to looping attempts to parse many documents, which leads to long rollouts costing $37.36 on average. Genie parses documents using ai_parse document, which enables the model to efficiently identify the correct information from each page, thereby saving cost and improving performance.

While these harness improvements lead to substantial gains, significant headroom remains in OfficeQA Pro V2. Systems continue to exhibit similar failure modes to those we observed in the original OfficeQA – parsing fidelity, incorrect temporal reconciliation as accounting conventions change over time, and misinterpretation of entity scope or category granularity.

Building a New Grounded Reasoning Benchmark

To truly test generalization, we needed a new corpus that still reflected the same enterprise-relevant skills OfficeQA was designed to measure: parsing complex documents, retrieving the right evidence, and performing analytical reasoning grounded in that information.

We worked with the U.S. Treasury to build the benchmark around a new corpus: the U.S. Federal Accounts of Receipts and Expenditures, which the Treasury was preparing to release as a unified dataset for the first time in conjunction with the 250th anniversary of the United States. The corpus consists of roughly 1,400 PDFs and 120,000 pages containing detailed U.S. accounting records, spanning 1793 through 2024.

Like the U.S. Treasury Bulletins used in the original OfficeQA benchmark, this new corpus reflects many of the challenges common across enterprise document collections: dense tables and charts, values revised over time, evolving reporting conventions, and deep institutional knowledge. These complexities also translate into a challenging evaluation: as with the original OfficeQA benchmark, we find that baseline frontier agents struggle to achieve consistently high accuracy on OfficeQA Pro V2.

Scaling OfficeQA Pro V2 with Synthetic Data

When we created the original OfficeQA benchmark in late 2025, the process was highly manual. Human annotators constructed questions and answers by hand while combing through a corpus comprising 89,000 U.S. Treasury Bulletin pages, often requiring numerous rounds of human review in order to ensure question quality. Although this approach eventually produced a high-quality benchmark, it was time-consuming, costly, and difficult to scale. Since then, we have developed techniques to automate the creation of rigorous benchmarks much more efficiently and reliably.

To build OfficeQA Pro V2, we leveraged asynth, our internal library for building synthetic data-generation pipelines. This enabled us to quickly and scalably create a diverse, verifiable benchmark that accurately reflected the challenges of grounded reasoning using a systematic process for generating and vetting questions:

1. Inputs. To promote sample diversity, the synthesizer was seeded with a randomly selected time period, analytical method, target number of source documents, and salient topic drawn from a list developed with our partner, USAFacts, to reflect questions that analysts might naturally ask of the corpus.
2. Synthesis. For each candidate question, a synthesis agent searched the corpus for evidence matching these criteria and generated a sample supported by a traceable chain of grounded reasoning.
3. Verification. Next, samples went through several quality gates to audit for source fidelity, analytical relevance, diversity, and answerability. Independent solver agents attempted the question from scratch, producing alternate solution trajectories, which were reviewed by a verification agent. Only candidates with verifiably correct answers advanced to the next stage.
4. Final Review. As a final quality-control step, samples were manually reviewed alongside solutions produced by additional frontier models. Reviewers examined the question, answer, reasoning, and cited evidence together, retaining only samples whose interpretation and ground truth were unambiguously correct.

The resulting questions require the same core grounded-reasoning capabilities measured by the original OfficeQA benchmark: retrieval across multiple source documents, analytical reasoning, and specialized capabilities such as supplemental web search and multimodal interpretation of figures. Example questions from the benchmark, as well as details about its composition, can be seen below.

Example Questions

1. Low Difficulty: Compute Chief Justice Earnings: “What is the total nominal dollar value for John Jay's earnings as Chief Justice of the Supreme Court, according to the Account of Receipts and Expenditures of the United States during the back half of CY 1793, and which warrant numbers corresponded to the payments? Return your answer as comma-separated values in enclosed brackets in the order of his earnings value, and the warrant numbers in calendar year order.”

This question requires locating a single 1793 Account of Receipts and Expenditures page and reading the four warrant payments recorded to Chief Justice John Jay (highlighted in red) while correctly interpreting that "do." is shorthand for "ditto," indicating the same data category as the preceding row. The question then requires understanding that the “back half of CY 1793” scopes the answer to only the July and November payments. Finally, retrieval of corresponding warrant numbers and a simple sum is required to compute the final answer.

2. Medium Difficulty: Forecast Increase in Medicare Outlays: “Between FY1990 and FY1994 inclusive, Medicare outlays grew rapidly as the program expanded. Fitting an OLS linear regression to the reported annual Medicare outlay figures for those fiscal years, what is the estimated average annual dollar increase in Medicare outlays reported in millions of dollars, rounded to two decimal places?”

This question requires retrieving the Medicare outlay figure from the "Outlays by Function" table in five separate annual Combined Statements (FY1990–FY1994, highlighted in red). Importantly, the agent must read each year’s value from the latest revised prior-year column, rather than the first reported figure. Finally, the agent must fit the series with a linear regression model to estimate the average annual increase.

3. High Difficulty: Forecast Increase of Outlays Across Agencies: “For the combined net outlays of the Departments of Commerce, Interior, and State, the Environmental Protection Agency, and the National Aeronautics and Space Administration rolled up from their constituent Treasury Account Symbol (TAS) families as reported by the U.S. Treasury, fit the standard nonseasonal Theta method on the reported outlays: estimate the θ=0 component as an OLS linear trend, construct the θ=2 transformed series, forecast that component with simple exponential smoothing initialized at its first value, and choose α over [0, 1], allowing α = 1.0. Using this forecasting and averaging the θ=0 and θ=2 one-step-ahead forecasts, what is the FY1989 forecast in millions of dollars rounded to the nearest hundredth?”

This question requires assembling an eight-year series (FY1981–FY1988) of net outlays for five different agencies (Commerce, Interior, State, EPA, and NASA) from eight separate Combined Statements. Again, the agent must use only the latest reported figures for each year. The resulting series then must be used to forecast net outlays for FY1989, using a specific forecasting method.

Benchmark Details

OfficeQA Pro V2 consists of 90 questions, all of which require evidence from the benchmark’s corpus. As part of our synthetic-data verification pipeline, we filtered out questions that could be answered using parametric knowledge or web search alone.

Although built on a new corpus, OfficeQA Pro V2 preserves the core enterprise grounded-reasoning capabilities measured by the original OfficeQA benchmark, both across the full question set, OfficeQA Full, and within its frontier-difficulty subset, OfficeQA Pro. 7% of questions require visual understanding of charts, graphs, or figures, compared with approximately 3% in OfficeQA Pro. Another 10% require supplemental information obtained through web search (such as inflation indices, GDP series, or population figures), compared with 21.8% in OfficeQA Pro and 15.9% in OfficeQA Full. Together, roughly one in six OfficeQA Pro V2 questions requires at least one of these specialized capabilities beyond text retrieval and analysis.

Compared to OfficeQA Pro, OfficeQA Pro V2 requires evidence from substantially more source documents per question. OfficeQA Pro questions draw on approximately 2 Treasury Bulletin documents on average, whereas OfficeQA Pro V2 questions require evidence from 6.7 source documents on average, with a median of 5.5 and a maximum of 24. In total, 74.4% of OfficeQA Pro V2 questions require four or more sources, compared with 62.4% in OfficeQA Pro and 56.1% in OfficeQA Full.

Taken together, these characteristics make OfficeQA Pro V2 a more demanding test of end-to-end grounded reasoning than the original benchmark, while preserving the same realistic mix of analytical, multimodal, and external-knowledge requirements found in enterprise workflows. To understand why the benchmark remains challenging, it is useful to examine how dramatically the underlying documents and reporting conventions have evolved over time.

A Corpus Spanning over Two Centuries

Spanning 232 years of U.S. federal financial reporting, the corpus also captures the evolution of document structures, accounting conventions, terminology, and institutions over more than two centuries. For instance, records in the late 1700s and early 1800s often record payments to individuals. These documents often include wide landscape foldouts, archaic typography like the use of long-s in words such as “Treaſury” or “Preſident”, and historical writing conventions like recording “do./ditto”, which pose novel challenges for modern parsing solutions.

By the mid-nineteenth century, the data was presented in two-page T-account formats and dense, ruled tables, while the reporting period changed from calendar to fiscal years. The early 1900s then introduced new financial terms like opening and closing balances, multi-year appropriations, and surplus and deficit conventions.

By the 1980s, the Combined Statement had evolved into an annual report with a detailed appendix. The reports became digital-native in the early 2000s and began incorporating visual charts alongside a standardized table of contents.

OfficeQA Pro V2 builds on the complexities of the Accounts of Receipts and Expenditures, creating an especially demanding stress test for grounded reasoning. Across the corpus, the same financial concept may change in name, location, table schema, unit, time basis, and level of aggregation. In addition to parsing documents and retrieving values, this means agents must reconcile information across changing reporting conventions – a complexity that is typical in enterprise settings.

Conclusion & Acknowledgements

The benchmark and parsed corpus are now publicly available on Hugging Face, with evaluation code available on GitHub. We encourage researchers and practitioners to evaluate their own agentic systems on OfficeQA Pro V2, and recommend its use as a test set for OfficeQA Pro. For enterprise developers, OfficeQA Pro V2 can serve as a testbed for building agents that must answer complex questions across large, heterogeneous document collections. Teams can use it to compare models and agent architectures, identify whether failures originate in parsing, retrieval, reasoning, or verification, and measure how changes to their systems affect accuracy, latency, and cost before applying those approaches to their own enterprise data. Finally, we leave readers with three takeaways:

1. Representative evaluations are essential, and synthetic data can help scale them. Frontier agents still struggle with enterprise workflows out of the box, making it critical to evaluate systems on tasks that reflect real-world complexity and test whether improvements generalize to unfamiliar settings. At Databricks, we combine synthetic data techniques with our understanding of customer workflows to build representative evaluations like OfficeQA Pro V2 more efficiently and at greater scale.
2. An optimized agent harness is critical for achieving the best performance. Moving from model providers’ baseline harnesses to Databricks Genie improved performance by 24.0 percentage points on average (+92% relative) across evaluated models. These results demonstrate that document processing, retrieval, tool use, and orchestration choices can substantially affect both the accuracy and efficiency of grounded-reasoning systems.
3. Grounded reasoning still has significant room for improvement. Although harness optimizations produced substantial gains over the out-of-the-box baselines, even the strongest systems continued to struggle with challenges such as faithful document parsing, reconciling values across time, and retrieving the latest revised figures. Continued progress will require innovations across the full agent stack, from parsing and retrieval to analysis and verification.

We thank USAFacts for their continued partnership on the OfficeQA benchmark suite, including helping us identify the new corpus, develop representative analytical topics and questions, and organize the Grounded Reasoning Cup. We also thank the U.S. Treasury for helping identify the U.S. Accounts of Receipts and Expenditures as the foundation for the competition benchmark, and for releasing the dataset as a cohesive collection for the first time.

Authors: Krista Opsahl-Ong, Arnav Singhvi, Josh Joseph, Jasmine Collins, Ivan Zhou, Shubham Toshniwal, Michael Bendersky, Erich Elsen,  Xing Chen, Matei Zaharia

If you want to learn more and get started with OfficeQA Pro V2, check out the benchmark on Hugging Face.