Evaluating a new kind of Search API

High-quality research and analysis depend on access to fresh, trustworthy information. A lot of mixed-quality information lives on the open web, and a new generation of AI web search APIs (e.g., Exa, Parallel) has emerged to help agents find, extract, and reason over web pages. Agents also need access to fresh, structured data to tackle a broader range of real-world research use cases: analyzing trends, building complete lists, constructing charts.

To an agent, web text and structured data solve the same problem: getting good information. But sourcing, ingesting, indexing, retrieving, and operating on trillions of structured data points demands a fundamentally different architecture than traditional web search does. Tako’s APIs combine web search and premium data search in a unified response payload to efficiently and accurately serve agents’ comprehensive grounding needs. Tako’s endpoints deliver unparalleled accuracy and efficiency compared with other web search APIs. In this post, we demonstrate those benefits across three categories of benchmarks:

• Established web-search benchmarks (SimpleQA, FRAMES)
• Deep-research benchmarks that require synthesis and reasoning across sources (FinSearchComp)
• VerticalRTK, our own benchmark focused on real-time, domain-specific knowledge

Tako performs at or above parity on established benchmarks and pulls significantly ahead on the harder questions that require reasoning and numerical precision to answer accurately.

Benchmarking the Status Quo

Standard benchmarks for evaluating web-search quality, like SimpleQA and FRAMES, consist of factual questions with a single verifiable answer.

“Who received the IEEE Frank Rosenblatt Award in 2010?” (SimpleQA)

These answers are widely reported across the web, and they do not change over time. We wanted to ensure that Tako’s combined web + data search endpoints perform at parity with web search APIs on both accuracy and latency.

We fetched search results from each provider’s API, synthesized an answer from the results with an LLM, and then used a separate LLM to judge the answer against the ground truth.

For each provider, default configurations were used. For more details, take a look at the harness used for this evaluation.

On these conventional benchmarks, Tako performs at parity with the best web search APIs in both accuracy and latency.

In other words, combining web and data search incurs no accuracy or speed penalty on basic web-search benchmarks.

VerticalRTK: Benchmarking More Serious Research Questions

Static Wikipedia-style trivia about the past ignores the kinds of queries agents actually need answered to perform economically valuable research tasks. Agents need quality, domain-specific information to handle these questions. To provide a more realistic measurement of search capabilities, we created our own benchmark — VerticalRTK.

This benchmark includes queries spanning multiple domains: finance, macroeconomics, sports, weather, and more. The queryset covers questions not commonly written about on the web, as well as questions with constantly changing ground-truth values like commodity prices and sports scores.

Each row in the set records when its answer was true and names the venue it came from, so the time-sensitive questions get re-sourced before a run rather than quietly going stale. The benchmark's own page covers how VerticalRTK is built and how its ground truth stays current, and the data ships under CC BY 4.0.

We used the same LLM-synthesis and LLM-judge methodology as above to assess quality. When we evaluate search quality through this more realistic lens, Tako outperforms the next-best web search API by 21%.

Tako Makes Your Agents Smarter, Faster, and Cheaper

Web search excels at single-fact retrieval, but single facts are rarely sufficient for real analysis. When tasked with problems such as identifying outliers, projecting trends, or correlating findings, an agent needs a larger series of data. Tako’s APIs give access to the raw, structured data returned by search results, enabling agents to efficiently perform this novel type of analysis. This ultimately means fewer tool calls, lower token cost, and significantly lower latencies.

To concretely demonstrate this, we built a simple Research Agent harness and gave it the corresponding Search and Contents APIs of each web search provider as well as basic code execution functionality. The agent was then tasked with a research question using the various combinations of tools.

Agents equipped with Tako’s toolset consistently perform the research task more accurately while using up to 75% fewer tool calls, at as little as one-tenth the cost.

EnergyMacroHousingFinance

Research objective

Compare Spain's and Vietnam's power-sector CO2 emissions since 2000. When did each country's emissions peak, and how far below its peak is each today? How many times larger are Vietnam's emissions now than in 2000? Has Vietnam overtaken Spain — and if so, in which year?

Run timeline

Replay

Tako

OpenAI web search

Exa

Parallel

search contents other

0s

Fact-by-fact rubric

Fact	Tako	OpenAI web search	Exa	Parallel
Retrieved
spain latest mt44.2 MtCO2 ±3.5	✓	✗	✗	✗
vietnam latest mt143.4 MtCO2 ±5	✓	✗	✗	✗
Computed
spain peak year2005 year	✓	✗	✗	✗
spain pct below peak67.8 % ±3	✓	✗	✗	✓
vietnam peak year2024 year ±1	✓	✓	✓	✓
vietnam multiple of 200016.8 x ±1.5	✓	✓	✗	✗
vietnam overtook spain year2016 year ±2	✓	✓	✓	✓

Run breakdown

	Tako	OpenAI web search	Exa	Parallel
Time	15.5s	2m 4s	1m 41s	1m 31s
Cost	$0.02	$0.21	$0.21	$0.10
Tokens in/out	13.5k / 2.1k	188k / 5.8k	926k / 5.6k	293k / 5.5k
0s2m 4s	✓	✓	✓	✓

Tako’s Agents Push the State of the Art Forward

Tako’s API primitives unlock a new class of capabilities for agents. Thanks to this, we’ve also been able to build our own proprietary agents on top of these primitives.

The Tako Answer Agent is able to plan a route through our knowledge graph and reason directly over the structured data it needs, performing analysis, synthesizing its research, and maintaining source provenance through citations as it works. This enables it to exceed state-of-the-art research capabilities at less than half the latency of competing systems.

What’s Next

We’ve released our Research Agent harness and published our set of realistic evaluations. There are even more facets of search quality that we’re constantly working on improving internally. In the future, we hope to share more on how we think about relevance alongside accuracy and go deeper on a wider range of data verticals.