Performance

This is the only run on this page that answered through the system a customer would use. The questions are a visible development set, not a sealed test. Graded by an automated judge that has not been validated, under two versions of the rubric.

No request failed in 150. The right document was cited on 122 of 150 questions. Service time was 9.7s median and 13.8s at the 95th percentile. The full round trip was 19.5s median and 27.7s at the 95th percentile, with a slowest request of 29.9s. Source: docs/verification/2026-08-04-eval150-deployed-system.md.

Why not just let an agent grep the folder?

It is a fair question and we measured it rather than arguing about it. Grep is genuinely faster than our retrieval call — about 18× faster on the search step alone. But a shell-search agent has to think three times to get one answer: pick search terms, pick files, then answer. Generating text is the slow part of a language model, roughly 485× slower than reading it. So the extra thinking costs far more than the extra reading, and the two approaches finish within a fifth of a second of each other today.

That near-tie comes from the August 2 development run. The deployed service measured later is much slower than that run, so the speed comparison has not been shown to hold on the production path. The arithmetic for the August 2 run is in bench notes.

A user waits for two things: finding the evidence, then writing the answer. Both arms below run the same model on the same deployment.

Retrieval costs 4.2s at 4B and 7.7s at 8B in this smoke. The 4B retrieval arm matched the 8B retrieval arms on both behaviour checks while running about 30% faster. That makes the 4B base model with search the leading production candidate — as a hypothesis for the next run, not a launch claim.

Speed only matters if the answer is right. On the 8B development matrix, retrieval is the only component with a large within-model quality signal.

Strict correctness · 100 development questions per arm · Wilson 95%

Retrieval moved the base 8B model from 15% to 86% — a 71-point difference on this set. The tuned rows are marked overlap because the development set overlaps tuning exposure, so they cannot support a generalisation claim. They are published to guide experiment design, not to sell fine-tuning. Tuning did not beat the base model in either mode.

A separate leakage-free rescore on a fresh 49-question draw is published with its confidence intervals and a paired significance note on the evaluation screen.

The comparison people ask for is grep against vector search. That is the wrong unit. The right unit is the whole path to an answer, because a shell-search agent pays for its search again on every extra model turn.

Below, each line is one cost term, measured as shell search minus retrieval. Bars to the right are where retrieval is ahead.

Reading is nearly free; writing is not. That is why handing the model 4.1× more text costs only 0.6s, while making it generate two more times costs 3.7s. Any estimate that prices context at a decode rate overstates it by two orders of magnitude.

Everything above this line is a model built from measured rates. On August 3 the comparison itself was executed: 30 paired questions through a real agent that chooses its own rg searches and file reads, against the same retrieval pipeline, all three arms answering with one equalized hosted model.

The paired result is one-sided: the agent was slower than non-reranked retrieval on 28 of 28 usable pairs, a median 1.418s behind. Its retrieval-recall deficit is statistically significant (exact McNemar p = 0.0117); final correctness is not — and the mechanism matters. The file-tool agent's failure mode was silence, not wrong answers: it declined to answer a third of answerable questions, almost always after failing to find the gold document. The reranker was slower on every question, a paired median 0.552s, and bought no measured recall or quality — which is the modelled conclusion above, confirmed by a second, independent method.

Two honest limits. The agent arm is a floor, not a ceiling: a small model under tight turn caps; a stronger agent would buy recall back by paying more per turn. And this ran on local hardware with a hosted answer model — it has not been repeated on the production serving path.

Everything above rests on an arithmetic model with two unmeasured inputs. This section states the model, shows how far the conclusion moves when those inputs move, and names what would settle it. It is deliberately text-first: the charts here are for people who intend to check the reasoning.

How the rates were obtained

A calibration sweep asks the live deployment the same question at six increasing context sizes plus six closed-book probes, reads generation milliseconds and token counts off the service meter, and fits three parameters jointly by least squares: fixed per-call overhead, milliseconds per prompt token, and milliseconds per completion token.

An earlier version fitted these in sequence — decode from the closed-book probes, prefill from the residual. It was unstable: the same deployment returned 3,743, then 9,567, then 9,090, then 15,748 tokens per second of prefill on consecutive runs, while decode never moved. Overhead and prefill were absorbing each other. The joint fit that replaced it repeats to within 1.6%. Its solver is covered by unit tests that check it recovers known coefficients exactly, keeps prefill the faster direction under noise, and returns nothing rather than a number when the probes did not vary enough to identify a rate.

The model

Writing shell search minus retrieval, in seconds:

net = search delta + context + 2 × overhead + T / d
    = −4.16 + 0.60 + 0.16 + T / d

T is the number of tokens the agent generates across its two extra turns. d is the decode rate. Everything else is measured. The ledger in the section above is this function at the declared T = 100 and the measured d = 28.

Neither T nor d is a constant of nature. T has never been counted on a real agent loop — the harness does not run one. d is our L4 deployment. Both charts below exist because a single number here would be a guess wearing a decimal point.

The finding that survives both charts

Remove the reranker and retrieval falls to roughly 0.40s. The search-leg term goes from −4.16s to −0.16s and the model reduces to:

net = context + T / d

Both terms are non-negative by construction. Context can only be zero or positive; T / d is positive for any agent that emits anything at all. So with the reranker removed, shell search loses end to end for every value of T, every decode rate, and every prefill rate. No unmeasured parameter can flip it.

In the August 2 run, the near-tie was not evidence that grep competes with retrieval. It was evidence that one stage of our own pipeline cost about as much as an entire agent loop. The August 4 deployed run measured that stage at a much smaller 1.6s, so this argument is unproven for the deployed system.

Measured, declared, unmeasured

Measured. Both token rates. Fixed per-call cost. Context sizes. Question lengths. Grep output size. Search times. Recall.

Declared, and chosen to favour shell search. Three turns. A 400-token system prompt. 400 tokens of tool schemas. 40 tool-call tokens on the first turn, 12 per file read on the second. A real harness carries more of each and usually takes more than three turns, so the shell total is a floor.

Extrapolated. The shell arm’s largest turn prompt is 8,819 tokens. The calibration sweep reaches 3,552. That one line is an extrapolation, and the harness prints a warning where it happens. The conclusion above is deliberately stated in a form that does not use the prefill rate at all.

Prefix caching. The harness computes a cached and an uncached shell total, because caching decides which is real. vLLM caches by default, so the cached — smaller — number is the one quoted. The compounding survives the assumption favourable to shell search. Cache-hit rate was not recorded.

A paired 150-question experiment compared a plain answer prompt against a source-restricted prompt on gemini-2.5-flash-lite (laboratory, not the shipped service). Strict correctness and the absolute number of fully supported answers were identical. It is published because a null result is still evidence, and because the supported-answer “improvement” it appeared to show was an artefact.

An omission a reader can see is not a claim. These are the fields a complete proof artifact needs, and how far the runs so far fill them.