pith. machine review for the scientific record. sign in

arxiv: 2605.11374 · v2 · submitted 2026-05-12 · 💻 cs.LG · cs.CL· cs.IR

Recognition: no theorem link

Test-Time Compute for Dense Retrieval: Agentic Program Generation with Frozen Embedding Models

Han Xiao

Pith reviewed 2026-05-13 02:24 UTC · model grok-4.3

classification 💻 cs.LG cs.CLcs.IR
keywords dense retrievaltest-time computefrozen embedding modelsagentic program generationinformation retrievalnDCGBEIR benchmark
0
0 comments X

The pith

A parameter-free test-time algebra lifts retrieval accuracy for any frozen embedding model.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that test-time compute improves small embedding models for dense retrieval even when the model remains completely frozen. An agentic search over hundreds of possible inference programs reveals that every high-performing option reduces to one simple operation: take the initial top-K documents, form a softmax-weighted centroid of their embeddings, and linearly interpolate that centroid with the original query vector. This default algebra raises nDCG@10 with statistical significance on seven different embedding families that span a tenfold size range, and the same lift appears on fully held-out BEIR data for every model tested.

Core claim

The entire Pareto frontier of candidate test-time programs collapses onto a single algebra: a softmax-weighted centroid of the local top-K documents interpolated with the query. This parameter-free default produces statistically significant nDCG@10 gains across seven embedding-model families spanning a tenfold parameter range, with held-out full-BEIR validation confirming the improvement on every model.

What carries the argument

An agentic program-search loop that evaluates 259 candidate inference programs over ninety generations on a frozen embedding API and identifies the single algebra that dominates the frontier.

If this is right

  • Any frozen embedding checkpoint can receive the same algebra at inference time to improve retrieval without retraining or parameter changes.
  • The performance lift holds across model families and sizes from small to large, indicating the algebra is not tied to a particular scale.
  • The same held-out validation success on full BEIR suggests the algebra is not overfit to the search-time validation split.
  • No additional training data or fine-tuning is required; the improvement is obtained purely through extra inference compute.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The result implies that embedding spaces distilled from LLMs retain latent geometric structure that can be exploited by simple post-retrieval operations.
  • Similar agentic program search could be applied to other frozen models in tasks such as reranking or clustering to discover additional parameter-free improvements.
  • The finding challenges the assumption that test-time compute benefits only large reasoning models and suggests a broader class of frozen encoders can profit from it.

Load-bearing premise

The representation space inherited from LLM backbones permits beneficial test-time programs that generalize beyond the data used to guide the program search.

What would settle it

Applying the softmax-weighted centroid interpolation to a new embedding model on a fresh held-out dataset and observing no nDCG@10 improvement or a statistically significant drop would falsify the central claim.

Figures

Figures reproduced from arXiv: 2605.11374 by Han Xiao.

Figure 1
Figure 1. Figure 1: Agentic generation loop. At each round g, the proposer reads the Pareto frontier and the lesson ledger, writes new Python programs over the frozen embedding-model API, and queues them on the harness. The harness evaluates each program under a GPU lock. Surviving programs update the frontier, and all results, both positive and ruled-out, become lessons in the ledger. The loop runs unsupervised for ninety ro… view at source ↗
Figure 2
Figure 2. Figure 2: Four representative programs discovered by the agentic loop. (a) P1: top-1 self-amp, founder of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cumulative best universal ∆nDCG@10 averaged over six cells, as a function of generation. The curve flattens after generation 30 and saturates after generation 78, suggesting the 2× universal frontier is exhausted under the current encoder API. phaEvolve (Novikov et al., 2025). What is new here is the application surface, training-free dense re￾trieval, and the lesson-ledger discipline that drives the searc… view at source ↗
Figure 4
Figure 4. Figure 4: ArguAna full-BEIR lift on each of the seven [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Default SOFTCENTROID (K=3, α=0.5, τ=0.05) on the four headline tasks at full BEIR, showing the three regimes: ArguAna (symmetric) breaks decisively; NFCorpus (medical-IR) bends with moderate gain; SciFact and FiQA (asym￾metric) hold near baseline. 5.8 Regimes of test-time compute The experimental evidence across seven embedding-model families, thirteen retrieval tasks, and two held-out benchmarks converges… view at source ↗
Figure 6
Figure 6. Figure 6: Cost-accuracy Pareto frontier over all 259 registered programs. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Test-time scaling of dense retrieval. Each blue point is one of 240 programs with at least four cells [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Iterative SOFTCENTROID on e5-base-v2 full-BEIR ArguAna across twenty iterations. The lift saturates at iter 5 at +15.73 nDCG@10 and decays gently afterwards, remaining statistically significant at p < 10−4 throughout the range. C Held-Out Cross-Model Validation [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Held-out full-BEIR validation across the eight model-task cells in Table [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Parameter sensitivity of SOFTCENTROID on e5-base-v2 full-BEIR, showing ∆ nDCG@10 (×100) over the cosine baseline for each parameter value across four tasks (NFCorpus, ArguAna, SciFact, FiQA-2018). The paper default is indicated by a navy border. Green cells are positive lifts; red cells are regressions. (a) α sweep (K=3, τ=0.05): NFCorpus is stable for α ∈ [0.2, 0.5]; ArguAna scales monotonically. (b) K s… view at source ↗
Figure 11
Figure 11. Figure 11: Program P1 (top-1 self-amplification). Encodes query and document under the retrieval LoRA, finds the [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Program P19 (first-sentence anchor). Same skeleton as P1, but the top-1 document is truncated to its first [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Program P22 (four-LoRA majority gate). Re-encodes the query under all four task-specialized adapters [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Program P24 (NFCorpus first-sentence variant). An early NFCorpus-tuned member of the AMP family [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Program P32 (tail-gap weighted amp). Replaces the cosine weight in P1 with a similarity-tail gap [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Program P33 (similarity-floor first-sent amp). Adds a lower bound on cosine before the amp fires. [PITH_FULL_IMAGE:figures/full_fig_p027_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Program P35, the DROPOUTENS stochastic dropout ensemble. Encodes the query M times under independent dropout masks, mean-pools the resulting embeddings, and reranks. Direct port of the LLM best-of-N recipe. Ruled out because the median of unit-norm embeddings leaves the unit sphere and there is no verifier signal to select among samples. Query q Embed q Retrieve top-K docs Compute centroid d¯ q ← L2(q + w… view at source ↗
Figure 18
Figure 18. Figure 18: Program P40 (multi-round Rocchio). Iterates the centroid update for [PITH_FULL_IMAGE:figures/full_fig_p028_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Program P47 (cross-LoRA top-3 overlap gate). Fires the amp only when the retrieval-LoRA and the [PITH_FULL_IMAGE:figures/full_fig_p029_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Program P76 (universal-min champion at 2×). Combines first-sentence anchor with a similarity-floor gate. The longest-defended 2×-cost universal-positive in the registry, with +0.29 nDCG@10 ×100 minimum across all six universal cells. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Program P97 (top-2 chunked anchor). Uses both the top-1 and top-2 documents’ first sentences as [PITH_FULL_IMAGE:figures/full_fig_p030_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Program P102 (gap-concentration amp). Replaces the gap-confidence gate with a top- [PITH_FULL_IMAGE:figures/full_fig_p030_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Program P103 (two-round first-sent amp). A multi-round member of the AMP family that re-retrieves [PITH_FULL_IMAGE:figures/full_fig_p031_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Program P164 (query-projection rotation). Projects the query onto the top- [PITH_FULL_IMAGE:figures/full_fig_p031_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Program P171 (universal-avg champion at 4×). A composite anchor program combining the first￾sentence body of P19 with adaptive-rank query projection. The longest-defended 4×-cost universal champion, with +2.67 nDCG@10 ×100 average across all six universal cells. Query q Embed q Retrieve top-K docs Form K-dim subspace V from top-K doc embeddings Amp vector δ = w d (1) Project: δ ′ = projV (δ) q ′ = L2(q + … view at source ↗
Figure 26
Figure 26. Figure 26: Program P174 (subspace-bounded amp). Restricts the amp update to lie within a [PITH_FULL_IMAGE:figures/full_fig_p032_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Program P28 (norm-convergence amp). Iterates the top-1 amp and tracks the update norm [PITH_FULL_IMAGE:figures/full_fig_p033_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Program P77 (doc-chunk K=2 redundancy gate). Splits the top-1 document into four disjoint sentence￾window chunks and encodes each separately. The amp fires only if at least two chunks have cos(q, chunki) above the per-query median similarity over the top-100 pool. Representative of the DOCCHUNK substrate family. Cost ≥ 4×. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Program P37 (top-3 anchor Borda rank vote). Builds three separate re-rankings, each from amplifying [PITH_FULL_IMAGE:figures/full_fig_p035_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Program P203 (per-query lambda). Uses two LoRA adapters: the retrieval LoRA for the initial retrieval [PITH_FULL_IMAGE:figures/full_fig_p035_30.png] view at source ↗
read the original abstract

Test-time compute is widely believed to benefit only large reasoning models. We show it also helps small embedding models. Since modern embedding models are distilled from LLM backbones, a frozen encoder should benefit from extra inference compute without retraining. Using an agentic program-search loop, we explore 259 candidate inference programs over a frozen embedding API across ninety generations. The entire Pareto frontier collapses onto a single algebra: a softmax-weighted centroid of the local top-K documents interpolated with the query. This default, which introduces no trainable parameters, lifts nDCG@10 statistically significantly across seven embedding-model families spanning a tenfold parameter range, with held-out full-BEIR validation confirming the lift on every model tested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper claims that test-time compute via an agentic program-search loop over a frozen embedding API can improve dense retrieval. Exploring 259 candidate programs across 90 generations causes the entire Pareto frontier to collapse onto one parameter-free algebra: a softmax-weighted centroid of the local top-K documents interpolated with the query. This algebra produces statistically significant nDCG@10 lifts across seven embedding-model families (tenfold parameter range) with confirmation on held-out full-BEIR data.

Significance. If the result holds, the finding is significant: it shows that test-time compute can benefit small embedding models by exploiting representation spaces inherited from LLM backbones, without retraining. The work supplies a simple, reproducible default that improves retrieval across model scales and supplies held-out validation on a standard benchmark suite.

major comments (3)
  1. [Abstract] Abstract and experimental validation sections: the manuscript states that held-out full-BEIR validation confirms the lift on every model, but provides no information on whether the search-time validation split used inside the 259-candidate loop is disjoint from the final held-out sets. Overlap would make the reported statistical significance on nDCG@10 vulnerable to selection bias.
  2. [Abstract] Abstract: the claim that 'the entire Pareto frontier collapses onto a single algebra' is presented without quantification (e.g., fraction of programs within 1% of the best, or stability across random seeds and BEIR partitions). This metric is load-bearing for the universality argument.
  3. [Methods] Search-procedure description (Methods/Experiments): no details are given on how the 259 candidates were generated, what statistical tests or multiple-testing corrections were applied during the 90-generation search, or how overfitting was controlled. These omissions directly affect whether the discovered algebra is intrinsic to the embedding space or an artifact of the search-time data.
minor comments (1)
  1. [Abstract] Abstract: the precise interpolation formula between the softmax-weighted centroid and the query vector should be written explicitly (e.g., as an equation) rather than described in prose.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the thoughtful and constructive report. The comments highlight important aspects of experimental rigor that we address point by point below. We commit to revisions that clarify the data handling, add quantitative support for key claims, and expand the Methods description without altering the core findings.

read point-by-point responses

  1. Referee: [Abstract] Abstract and experimental validation sections: the manuscript states that held-out full-BEIR validation confirms the lift on every model, but provides no information on whether the search-time validation split used inside the 259-candidate loop is disjoint from the final held-out sets. Overlap would make the reported statistical significance on nDCG@10 vulnerable to selection bias.

    Authors: The search-time validation split employed inside the 259-candidate program-search loop is disjoint from the final held-out full-BEIR evaluation sets. Program discovery used a designated internal validation subset drawn from a subset of BEIR tasks, while the reported nDCG@10 results and statistical tests are computed on the complete, non-overlapping BEIR suite. We will insert an explicit description of this partitioning into the revised abstract and experimental sections to eliminate any ambiguity regarding selection bias. revision: yes

  2. Referee: [Abstract] Abstract: the claim that 'the entire Pareto frontier collapses onto a single algebra' is presented without quantification (e.g., fraction of programs within 1% of the best, or stability across random seeds and BEIR partitions). This metric is load-bearing for the universality argument.

    Authors: The collapse of the Pareto frontier onto the softmax-weighted centroid algebra is observed across the full set of 259 evaluated programs and 90 generations. To strengthen the universality argument, we will add quantitative metrics in the revision, including the fraction of programs that lie within 1% of the best nDCG@10 and stability results across random seeds and alternative BEIR partitions. revision: yes

  3. Referee: [Methods] Search-procedure description (Methods/Experiments): no details are given on how the 259 candidates were generated, what statistical tests or multiple-testing corrections were applied during the 90-generation search, or how overfitting was controlled. These omissions directly affect whether the discovered algebra is intrinsic to the embedding space or an artifact of the search-time data.

    Authors: We agree that the current Methods section is insufficiently detailed on these points. We will expand it to describe the agentic procedure used to generate the 259 candidate programs, the statistical tests and any multiple-testing corrections applied across the 90 generations, and the specific controls (including validation-set separation) used to mitigate overfitting. These additions will enable readers to evaluate whether the discovered algebra reflects properties of the embedding spaces. revision: yes

Circularity Check

0 steps flagged

No circularity: algebra discovered via search and validated on held-out data

full rationale

The paper's central result is obtained by running an agentic search over 259 candidate programs for 90 generations on a frozen embedding API, observing that the Pareto frontier collapses to one algebra (softmax-weighted centroid of local top-K interpolated with query), and then confirming statistically significant nDCG@10 lifts on held-out full-BEIR splits across seven model families. No equation or claim reduces to its own inputs by construction: the algebra is not defined in terms of the performance metric it is later tested on, no parameter is fitted on the final test sets and then renamed a prediction, and no self-citation or uniqueness theorem is invoked to force the outcome. The held-out validation step is independent of the search loop, satisfying the requirement that the derivation remain self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that embedding models distilled from LLMs inherit a representation space amenable to test-time program improvement, plus standard mathematical operations such as softmax and vector interpolation. No free parameters or invented entities are introduced in the final reported method.

axioms (1)
  • domain assumption Embedding models inherit representation space from LLM backbones and therefore benefit from extra inference compute without retraining
    Explicitly stated in the abstract as the basis for applying test-time compute to frozen models.

pith-pipeline@v0.9.0 · 5412 in / 1402 out tokens · 70265 ms · 2026-05-13T02:24:19.798009+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.