VerticalRTK: a real-time search API benchmark

An open benchmark of 286 research questions whose answers change over time. In our July 2026 run, Tako’s Answer API scored 66.4%, 11.4 points higher than the next-best API, Exa (55.0%).

RESULTS

Which search API is most accurate on real-time questions?

Tako’s Answer API was the most accurate, at 66.4%, 11.4 points higher than the next-best API, Exa (55.0%). Parallel scored 51.1%, Nimble and Tavily (advanced) 45.8%, Brave Search 43.5%, and Ceramic 7.6%. We ran the 131-question set against seven search and answer APIs in July 2026, with one harness: the same synthesis step and the same judge for every provider.

Answer accuracy

ProviderAccuracy
Tako Answer66.4%
Exa55.0%
Parallel51.1%
Nimble45.8%
Tavily (advanced)45.8%
Brave Search43.5%
Ceramic7.6%
131-question set, July 2026. Higher accuracy is better.

Scores move as providers ship, so read this as a dated run rather than a standing leaderboard. The fuller picture — SimpleQA, FRAMES, and FinSearchComp alongside this set — is on the Tako benchmarks page, and the write-up behind the run is Evaluating a new kind of Search API.

THE SET

What is VerticalRTK?

VerticalRTK is an open benchmark of 286 research questions with reference answers, published by Tako under CC BY 4.0. You send each question to a search or answer API and grade its answer against the reference, so it measures whether a system gets the answer right, not whether it retrieved a page that contains it. The data is on GitHub: data only, with no runner, no graders, and no results baked in.

The questions span seven verticals: companies, government and defense, macro and markets, society and health, energy and climate, sports, and digital usage. Many take several pieces of data to answer, so a system has to find each figure and combine them.

research questions across two sets
286
verticals, labeled per question
7
kinds of answer validity
3
free to use, including commercially
CC BY 4.0
Companies Example Question

Using Apple's and Microsoft's fiscal 2025 R&D expense and total revenue, which company spent a higher share of revenue on R&D, and by how much?

Ground truthMicrosoft, at about 11.5% of revenue ($32.488B R&D / $281.72B revenue) versus Apple's 8.3% ($34.55B / $416.161B) — a gap of roughly 3.2 percentage points.
1 of 3

THE GAP

Why don't SimpleQA and FRAMES measure real-time search?

Standard web retrieval benchmarks keep the answer on a page you can fetch. Score well and you've shown your index is broad and your retriever works. That's a real capability. It isn't the one an agent needs when someone asks how a company's revenue growth compared with its membership fee growth, five years running.

VerticalRTK closes that gap. Some vRTK questions have an answer no single page states, so a system has to pull together all the data needed to satisfy the question. Others have a ground truth the open web documents poorly, not at all, or a ground truth that moves: commodity and crypto prices, FX rates, sports results, and prediction markets.

The point isn't to make retrieval look bad. It's to highlight the gap between what web retrieval measures and what people need — an agent is judged on its answer, so a benchmark for agents should be too.

COVERAGE

What's in the VerticalRTK dataset?

VerticalRTK ships two files: a 131-question original set and a 155-question fast set. Score them separately: the fast set revises and expands the original, and 27 questions appear in both, so pooling them double-counts.

The original set holds 131 questions, each with an id, a query, and an expected answer. The fast set holds 155, re-sourced in August 2026, and adds two fields the first set doesn't carry — a vertical label, so you can score coverage per domain instead of only in aggregate, and a per-row as_of that says when the answer was true.

Fast set, questions per vertical

VerticalQuestions
Companies61
Government and defense24
Macro and markets23
Society and health21
Energy and climate15
Sports8
Digital usage3
One row of verticalrtk_fast.jsonl
{"id": "vrtk_fast_0001",
 "query": "What was Apple's iPhone revenue in fiscal year 2025?",
 "vertical": "companies",
 "answer": "Apple's iPhone revenue was $209.59 billion in fiscal year 2025.",
 "as_of": "FY2025"}

The distribution is deliberately uneven — it follows what people research, not a quota. Read the small verticals as color, not as a score.

METHOD

How do you evaluate a web search API for AI agents?

To evaluate a web search API for AI agents, ask questions whose answers change, record when and where each answer was true, re-source the volatile answers right before the run, and grade the API's answer rather than an exact string. The hard part isn't the questions. It's that the answers move while you're measuring. Any benchmark of live knowledge needs these six, whether you use our set or build your own:

  1. Ask questions whose answer moves

    Not questions whose page is merely hard to find. If the answer was equally true last year, you're measuring retrieval, not currency.

  2. Record when each answer was true

    One field per row. Without it you cannot tell a wrong answer from a stale reference, and every later decision rests on that distinction.

  3. Name the venue each answer came from

    A price is only a fact relative to where it was quoted. Naming the venue is what makes the row reproducible, and what lets you refresh it.

  4. Separate the rows that expire from the rows that don't

    A fiscal year and a settled match never change. A capture timestamp drifts within hours. Treating them alike manufactures failures.

  5. Re-source the volatile rows immediately before the run

    From the venue the row names. This is the step that turns rot into a maintenance task instead of an expiry date.

  6. Grade the answer, not the string

    Synthesize an answer from what the API returned, then judge it against the reference with a separate model. Exact-match scoring fails correct answers that round differently.

Steps two through five are why this set is still usable months after we wrote it, and they're the ones most benchmarks skip.

METHOD

How do you keep benchmark answers from going stale?

A benchmark of live questions starts decaying the day you publish it. USD/NOK moved from 9.25 to 9.35 across a single afternoon of our own runs. Grade a correct answer against a stale capture and you've recorded a failure that tells you nothing about the system.

So every row in the fast set says when its answer was true, and each answer names the venue it came from. Those two fields are what make the set refreshable: before a run, we re-source the rows whose truth has moved, and the rest stand as published. Rot becomes a maintenance step instead of an expiry date.

The as_of field carries three different kinds of validity. Treat them alike and you'll manufacture failures:

A reporting period
FY2025, 2026-Q2, 2026-07. The answer doesn't change. Score these any time.
A settled event
A past market close, a completed match. The answer doesn't change either.
A capture timestamp
True at that instant, and drifting within hours. Re-source these from the venue named in the answer immediately before you run, or drop them from the run.

Where a fact genuinely has no single value — a traffic share that providers measure differently, a forecast, an obligation still being revised — the reference answer states the accepted range. Where the fact has one true value, the range is only the spread between named venues at the capture moment. It's never a courtesy margin.

SCORING

How do you grade a search API's answers?

Grade each answer against its reference answer, not as an exact string match, with whatever harness suits you. Ours uses an LLM judge: fetch results from the API under test, synthesize an answer from those results, then score that answer against the reference with a separate model.

The repository ships data only, on purpose — a runner would bake our synthesis and judging choices into everyone's numbers. If you'd rather not build one, web-search-api-evals does the fetch, synthesize, and judge loop out of the box.

LICENSE

How do I download and run VerticalRTK?

Download the JSONL files from the GitHub repository, pin a commit so your scores stay comparable, and send each question to the API you're testing.

VerticalRTK is licensed under Creative Commons Attribution 4.0. Share it, adapt it, and use it commercially — give credit and you're covered.

Citation
Tako (TakoData). VerticalRTK: a benchmark of research questions
with reference answers. 2026. CC BY 4.0.
https://github.com/TakoData/VerticalRTK

Found a reference answer you can't reproduce? Open an issue — a wrong answer in the set is worth more to us as a report than as a quiet failure in your run.

FAQ

Questions about VerticalRTK

What topics does VerticalRTK cover?

Seven verticals. In the 155-question fast set: companies (61 questions), government and defense (24), macro and markets (23), society and health (21), energy and climate (15), sports (8), and digital usage (3). The mix follows what people research rather than a quota, so read the small verticals as color, not as a score.

What format is the VerticalRTK data in?

JSONL, one question per line. Every row has an id, a query, and a reference answer. Rows in the fast set also carry a vertical label, so you can score per domain, and an as_of that says when the answer was true.

How many questions does VerticalRTK have?

Two sets ship in the repository: 131 questions in the original set and 155 in the fast set, which was re-sourced in August 2026 and adds a vertical label and an as_of per row. Score them separately — 27 questions appear in both.

Does VerticalRTK ship a runner or a grader?

No. The repository is data only, so that our synthesis and judging choices don't get baked into everyone's numbers. We recommend a harness such as web-search-api-evals, which fetches results, synthesizes an answer, and scores it against the reference with an LLM judge.

Which search APIs has Tako scored on VerticalRTK?

Our July 2026 run of the 131-question set covered Tako’s Answer API, Exa, Parallel, Nimble, Tavily (advanced), Brave Search, and Ceramic, with the same synthesis and judging step for each. Tako’s Answer API scored 66.4%, 11.4 points ahead of the next-best API, Exa (55.0%).

Can I use VerticalRTK commercially?

Yes. VerticalRTK is licensed under Creative Commons Attribution 4.0 International, so you may share and adapt it for any purpose, including commercially, as long as you give appropriate credit.

How do I cite VerticalRTK?

Cite it as: Tako (TakoData). VerticalRTK: a benchmark of research questions with reference answers. 2026. CC BY 4.0. https://github.com/TakoData/VerticalRTK

What if a reference answer in VerticalRTK is wrong?

Open an issue on the GitHub repository with the question id and the answer you found. A wrong reference answer is worth more to us as a report than as a quiet failure in your run.

Put real-time answers in your agent

Tako’s Answer API led VerticalRTK at 66.4%. Start free with 2,000 requests, or run the benchmark on your own stack.