Aethos: 94% on EnterpriseRAG-Bench (42k)

TL;DR

Today we are releasing preliminary benchmark results: Aethos X scores 94% on EnterpriseRAG-Bench at 42,587 files, approaching what we estimate to be the benchmark’s effective ceiling at this scale. Surprisingly, agentic search with 5–10 parallel Astra agents also performs well: 86.03% on the same corpus size.

We are confident that enterprise-level memory will be a fully solved problem in H1 2027 and will serve as a foundation for self-improving agents. This is our forecast. We will release the full benchmark report later.

A note on the trade-off: in this research, we pushed retrieval quality as far as we could, at the expense of latency and cost. Production use cases need different combinations of answer quality, response time, and price.

A benchmark that looks like a company

EnterpriseRAG-Bench, released by Onyx, contains more than 500,000 synthetic documents and 500 questions. The documents resemble the material a company actually produces: Slack discussions, emails, tickets, shared documents, CRM records, meeting transcripts, code reviews, and wiki pages.

That makes it a particularly useful benchmark for company brains and enterprise retrieval systems. The challenge includes finding information across documents, resolving conflicting evidence, gathering every part of an answer, and recognizing when the information is missing. Its combined score measures answer correctness and completeness together. A score of 94% is therefore not the same as retrieving 94% of relevant documents.

To study scaling, we used selected sizes from the corpus ladder developed in a separate scaling study: 1.1k, 2.3k, 7k, 21.6k, 42.6k, 132k, and 512k files. The ladder keeps the questions and their supporting evidence fixed while adding more surrounding material. It asks a practical question: how well can a system still find the answer as the company’s knowledge grows?

The chart covers that full range. Our Aethos results in this preliminary report stop at 42.6k files; the chart also includes projections, identified below.

What we compare against

The official leaderboard includes specialized retrieval systems, cloud search products, file-search agents, and common RAG frameworks. Here is a selection of its published combined scores on the full corpus, checked on September 19, 2026:

Our comparison chart includes Prism, metor.com, and Troml, alongside file-system agents using GPT-6 Astra, GPT-5.6 Sol, GPT-5.6 Terra, and Qwen3.6-27B. The Qwen results come from the scaling study. The OpenAI-model results and their projections come from our evaluation data.

For the three leaderboard systems, only the full-corpus endpoint is a published measurement; their smaller-corpus curves are extrapolations. Those curves are context for the experiment, not evidence of a measured head-to-head win. The leaderboard has also moved on since the chart’s comparison set was chosen, as the Mixedbread result shows.

What is Aethos?

Aethos is the graph-based memory system behind the Codos Company Brain. It turns company documents and conversations into connected, maintained knowledge that people and agents can use.

The architecture starts with source material. Aethos extracts small facts and relationships, retains their evidence and dates, resolves references to the same people and projects, and uses them to maintain connected records. As new information arrives, those records preserve history and explain what changed. Agents can follow the relationships and return to the underlying sources when they need more detail. We describe the process in How the Codos Company Brain works.

This shifts part of the work to ingestion: the system organizes knowledge before an agent asks a question. The experiment measures whether that preparation helps the agent answer accurately, and what it costs.

Benchmark results

We evaluated three Aethos configurations:

• Aethos Balanced: a practical balance between answer quality, speed, and cost.
• Aethos Performance: a stronger emphasis on retrieval and answer quality.
• Aethos X: the same focus on quality, plus an additional technology we are not disclosing yet. It is an experimental configuration and is not yet available to customers.

At 42,587 files, the results and estimates are shown below. Balanced’s figures are projected; Aethos X is a preliminary result.

The table distinguishes measurements from projections in the evaluation data. Exact Aethos X latency and cost are not included in this preliminary table. A dash means not reported.

The strongest file-system baseline deserves attention. Parallel Astra agents reached 86.03% without the Aethos memory architecture. At the same corpus size, Aethos Performance reached 90.55%, a 4.52-point improvement, with lower mean query time and cost in these runs.

That result gives us two useful observations. First, a strong model with parallel search can already recover a great deal of company knowledge. Second, there is still room for a memory architecture that makes the right evidence easier to find.

The price of another point

Better answers do not come free. Increasing the search budget gives an agent more opportunities to find missing evidence, but it also increases the work done for each question. The right budget depends on the task: an interactive assistant and a background research agent can tolerate very different response times.

What we learned about memory architecture

1. Parallel agents need a stopping rule

Subagents that solve separate parts of a question can improve coverage substantially. They can also keep searching long after the useful work is done. A timeout bounds that cost, but a single timeout for every question is a blunt instrument: the appropriate budget depends on the corpus size and the complexity of the request.

The architectural question is how to allocate that budget. Which parts of the question can run independently? When has a search found enough evidence? When is another pass worth the delay?

2. Search tools determine how far an agent can go

On a small corpus, repeated text searches and file reads can work well. As the corpus grows, the agent spends more effort separating relevant evidence from plausible matches. A stronger model helps, but it still needs tools that narrow the search effectively.

Our lesson is to design the search interface alongside the memory architecture. If the system has already organized people, projects, and relationships, the agent needs a useful way to navigate that structure.

3. A graph helps discovery; noise still matters

A knowledge graph can make evidence easier to discover by connecting information that is scattered across sources. It does not make irrelevant or conflicting information disappear. More documents can introduce more aliases, competing accounts, and relationships to evaluate.

Scaling this approach to millions of documents remains an open research problem for us. These preliminary results reach 42.6k files; they do not yet establish Aethos performance at the largest tiers.

4. What you choose to model shapes every answer

One of the biggest decisions in building a knowledge graph is which things deserve their own records and which relationships matter. The answer depends on the domain.

For a company brain, a person, a commitment, and a project can each be useful units of memory. The vocabulary determines what the system extracts, what it connects, and what an agent can discover later. If it misses a distinction that matters to the business, later retrieval inherits that blind spot.

Why the benchmark ceiling matters

Even a very strong system on a small corpus does not necessarily score 100%. Some remaining errors come from the benchmark itself.

In our preliminary review, we identified problems with roughly 2% of questions and a further 11% whose evaluation was ambiguous or unstable. We currently estimate the effective ceiling at around 94%. We will publish the detailed review with the full benchmark report.

One example is question qst_0433, which asks about Redwood Inference’s process for rotating production secrets and credentials, including emergency rotation.

The source documents: routine rotations go into a change record; emergency rotations can go into an incident record, with follow-up actions after the incident.

The expected answer: all secret rotations are recorded in change records.

The expected answer loses the exception. A system that preserves the distinction can disagree with the answer key while remaining faithful to the documents.

This is why we describe Aethos X as approaching our estimated ceiling at this scale. The estimate remains preliminary.

Memory as a foundation for self-improving agents

Our confidence about H1 2027 comes from the direction of progress: stronger agents can already do much of the search themselves, and structured memory can improve how they find and connect evidence.

Reliable memory gives an agent somewhere to retain the context of its work: what it tried, which decision changed, what feedback it received, and what happened afterward. That is a foundation for improving the next attempt. The benchmark tests part of that foundation; the learning loop also needs feedback, evaluation, and a way to retain useful changes.

We will release the full benchmark report later. For now, the preliminary results show how close enterprise retrieval is getting, and why the remaining work involves cost, latency, scale, and the quality of the evaluation itself.

Thanks to Dima Khanarin, Rahul Sethuram, Ojas Mor, and Kevin Simback for their work on this research.