Pith. sign in

REVIEW 4 major objections 4 minor 18 references

Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VDS-TTT claims that an LLM can continuously self-improve at test time by having a verifier select high-confidence pseudo-labels and fine-tuning only a LoRA adapter, yielding up to 32.29% relative accuracy gains over the base model.

desk verdict A plausible combination of verifier-based selection and LoRA test-time training, but the experimental evidence is unreliable: base accuracy moves with N, there is no TTT-level held-out evaluation, and Table 2 has arithmetic errors. read the letter →

arxiv 2505.19475 v2 pith:DZOW5YAZ submitted 2025-05-26 cs.CL

Mohammad Mahdi Moradi , Hossam Amer , Sudhir Mudur , Weiwei Zhang , Yang Liu , Walid Ahmed This is my paper
classification cs.CL
keywords test-timetrainingverifier-drivensampleselectionpseudo-labelingLoRAadaptationmathematicalreasoninglargelanguagemodelsself-improvementtransductivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes VDS-TTT, a method for letting a large language model improve itself at test time without ground-truth labels. For each test question the model generates several candidate answers; a learned verifier scores them, and the highest-scoring answer above a confidence threshold becomes a pseudo-label. The model then fine-tunes only lightweight LoRA adapter parameters on that pseudo-labeled example, leaving the base weights frozen. Across three math benchmarks and several LLMs, the authors report up to a 32.29% relative improvement over the base model and a 6.66% gain over verifier-based selection without test-time training, with the largest jumps on hard competition problems. The motivation is that models can adapt to novel reasoning distributions on the fly instead of remaining frozen after pretraining.

What carries the argument

The load-bearing object is the verifier-scored pseudo-label: for each query, N temperature-sampled candidate responses are scored by a learned verifier, and the highest-scoring response above threshold τ is treated as the target for a standard language-modeling loss. The update is confined to LoRA low-rank adapters attached to attention and MLP projections, so the base model stays frozen and the adaptation is parameter-efficient. This pair—confidence-filtered pseudo-labels and low-rank updates—is what carries the claimed self-improvement, because the verifier decides which samples are worth learning from and the adapter limits the damage a wrong pseudo-label can do.

What would settle it

Split a benchmark like GSM8K into an adaptation set and a disjoint evaluation set; run VDS-TTT on the adaptation set and check whether accuracy on the disjoint evaluation set rises above the base model. If it does not, the reported gains are memorization of verifier-selected responses on the adapted queries rather than self-improvement.

Watch

Extended reading notes

Core claim

The central claim is that verifier-guided pseudo-labeling can replace ground-truth labels during test-time training. Algorithm 1 does Best-of-N selection: for each query, N responses are sampled with temperature, a verifier assigns a reliability score, the argmax response is kept only if its score exceeds threshold τ, and the model minimizes the negative log-likelihood of that response on the query by updating only LoRA adapter parameters. The authors find that this one-sample supervised update improves exact-match accuracy on Math-500, GSM8K, and AIME1983–2024 across Llama-3.2-1B, Llama-3.2-3B, and DeepSeek-R1-Distill-Qwen-1.5B, and that repeated iterations continue to improve accuracy, eventually matching or exceeding an oracle verifier. They conclude that VDS-TTT addresses the absence of ground-truth labels well even under structurally novel reasoning tasks.

Load-bearing premise

The paper assumes that accuracy on the exact test questions used to update the model reflects real improvement, not memorization of the answers the verifier selected, because no separate set of unseen questions is reported.

Editorial extensions

If this is right

  • If VDS-TTT is right, frozen base models can be continuously specialized at test time using only unlabeled queries and a verifier.
  • Accuracy on hard reasoning benchmarks rises with more candidate samples N and with more training iterations, until diminishing returns.
  • Test-time training with verifier-filtered pseudo-labels beats both no-adaptation verifier selection and RL-based test-time training on the tested models, especially when the base model is weak on the domain.
  • Only a low-rank adapter needs to be updated, so the base model's original capabilities are preserved and overfitting is limited.
  • Iterative application can surpass an oracle verifier's best-of-N accuracy, meaning the model learns to generate better candidates than static verification would select.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to measure accuracy on held-out queries after adaptation, since the paper's protocol reports accuracy on the same queries used for adapter fine-tuning.
  • The threshold τ controls a precision-recall trade-off: a high threshold protects against bad pseudo-labels but discards hard queries, so an automatic threshold-adaptation mechanism could broaden applicability.
  • Because the verifier is math-specific, extending VDS-TTT to code or commonsense would require a general-purpose verifier or a mixture-of-experts verifier, a direction the paper mentions as future work.
  • The result that iterative adaptation surpasses the oracle verifier suggests that verification and learning compound, which could be tested by ablating the verifier's role after the first iteration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces VDS-TTT, a test-time training framework that, for each test query, generates multiple candidate responses, uses a trained verifier to select the highest-scoring candidate above a confidence threshold, and fine-tunes low-rank LoRA adapters on the resulting pseudo-labeled examples. The authors claim that this procedure enables continuous self-improvement of LLMs at test time, reporting up to 32.29% relative improvement over the base model and 6.66% gains over verifier-based selection without TTT, across three math benchmarks (GSM8K, Math-500, AIME1983-2024) and three base LLMs.

Significance. The core idea—using a verifier's confidence to select pseudo-labels for test-time adaptation—is timely and relevant, and the LoRA-based parameter efficiency is a sensible design choice for on-the-fly adaptation. If the empirical claims were valid, the paper would make a meaningful contribution to test-time training for reasoning tasks. However, the experimental evaluation has a fundamental flaw: the model is adapted on the exact test queries used for evaluation, so the reported gains may reflect memorization of verifier-selected pseudo-labels rather than genuine self-improvement on unseen problems. In addition, several internal inconsistencies (e.g., a varying Base accuracy for a frozen model, and a clear arithmetic error in Table 2) cast doubt on the reliability of the reported numbers. Because the central claim of continuous self-improvement rests on the validity of these experiments, the significance of the paper is currently not established.

major comments (4)
  1. [Algorithm 1 and Section 4.3] The evaluation is circular with respect to test-time adaptation: Algorithm 1 fine-tunes a LoRA adapter on each test query q_i (lines 16-19), and Section 4.3 then reports exact-match accuracy on the same benchmark test sets. This measures performance on the training data of the adaptation procedure, not generalization to unseen queries. The phrase 'held-out test splits' in Section 4.1 refers only to the standard train/test split of the benchmark datasets, not to a held-out split at the TTT level. To support the claim of continuous self-improvement, the authors must evaluate on a separate set of queries that the model has not seen during test-time adaptation, or use a leave-one-out protocol. Without such an evaluation, the reported gains could simply reflect memorization of verifier-selected pseudo-labels.
  2. [Table 1] The Base column, defined as the frozen model without any test-time training, should be constant for a given model and benchmark regardless of N. However, Table 1 shows Base accuracy varying with N (e.g., Llama3-1B on Math-500: 20.8, 26.80, 33.60, 36.80 for N=2,4,8,16). This is impossible for a fixed base model and indicates either a mislabeling of the rows/columns, a hidden dependence on N in the base evaluation, or a serious reporting error. This inconsistency undermines the reliability of all comparisons against the Base baseline.
  3. [Table 2] In Table 2, the Average for VDS-TTT is reported as 38.3, but the row values (10.0, 38.5, 54.2) average to 34.23, not 38.3. Similarly, the average of the ΔVDS-TTT row (6.7, 19.2, 6.4) is 10.77, which is reported as 10.8; this part is consistent, but the main average is arithmetically wrong. This error directly affects the paper's comparative claim against TTRL and must be corrected.
  4. [Section 4.3 and Eq. (2)] The early-stopping criterion in Eq. (2) uses A(t), the test-set accuracy at iteration t, to select the optimal stopping point t*. This is a form of test-set leakage: in a genuine test-time setting, ground-truth labels on the test queries are not available to decide when to stop. Moreover, the claim that iterative VDS-TTT surpasses the Oracle Verifier (Figure 3) is suspect because repeated adaptation on the same test queries naturally leads to overfitting, so exceeding an oracle that only selects best-of-N from the original candidate pool is not evidence of general self-improvement. The authors need to clarify how early stopping would be determined without access to test labels, and provide held-out accuracy to support the iterative improvement claim.
minor comments (4)
  1. [Section 3.1] The notation in Section 3.1 introduces D={q_i, r_ij, a_ij}_{j=1}^N, but N later denotes the number of generated responses per query, not the dataset size. Please use distinct symbols for the number of candidates and the number of test queries to avoid confusion.
  2. [Eq. (1)] In Eq. (1), the pseudo-label Žy is used in the loss L(LM_Δ(q, r), Žy) but is never defined. If the loss is simply language-modeling loss on the selected pseudo-label r, please state this explicitly and avoid introducing an undefined variable.
  3. [Abstract and Section 1] The claim of being 'the first to synthesize verifier driven test-time training data' is too strong given that TTRL (Zuo et al., 2025) also uses reward-based pseudo-labeling at test time, albeit with RL. Please temper the novelty claim or clearly differentiate the contributions.
  4. [Throughout] There are several typos and grammatical errors, e.g., 'wihout' (Section 2), 'dramtically' (Section 2), and 'aligning better with the query task,' (Section 3.2 with a stray comma). The manuscript would benefit from a careful proofreading pass.

Circularity Check

1 steps flagged · score 3.0 of 10

Evaluation circularity: VDS-TTT adapts the LoRA adapter on each test query and then reports accuracy on those same queries, so the claimed self-improvement and oracle-surpassing iterative curves are partly training-set fits, not independent predictions.

  1. other [Algorithm 1 (Stage 3, lines 15-19); Section 4.1; Section 4.3 (Table 1 and Figure 3)]
    "for all test query qi do ... Compute loss L(∆) = - |ri| X t=1 log f_{θ0+∆}(ri,t | qi, ri,<t) ... ∆ ← ∆ - η ∇∆ L(∆) (Algorithm 1). ... We report exact-match accuracy on held-out test splits (Sec. 4.1). ... Table 1 reports exact-match accuracy for our three baselines ... using N=2,4,8,16 candidate responses (Sec. 4.3)."

    Algorithm 1 minimizes the language-modeling loss on each test query qi with its verifier-selected pseudo-label ri, and Section 4.3 measures exact-match accuracy on those same queries. The improvement over Base/VB is therefore computed on the very inputs used for adaptation: the model has directly optimized to reproduce ri for qi, and ri was selected as the highest-confidence candidate (usually containing the correct answer). The iterative results in Figure 3 re-train on the same benchmark test set many times, so exceeding the iteration-zero Oracle Verifier is a training-fit artifact rather than a prediction for unseen queries.

full rationale

The paper has no load-bearing self-citations and no mathematical derivation that presupposes its conclusion; the verifier is an external pretrained Skywork model, and the algorithm is a straightforward LoRA fine-tuning loop on verifier-selected pseudo-labels. The only circularity is in the evaluation design: Algorithm 1 adapts on every test query, and Sections 4.1/4.3 report exact-match accuracy on those same queries with no held-out split at the TTT level. For a purely transductive TTT method this protocol is partly intended, but the title and conclusion claim 'continuous self-improvement' and 'structurally novel reasoning tasks' without showing transfer to inputs not used for adaptation, so the headline gains (and especially the oracle-surpassing iterative curves in Figure 3) are in part a training-fit effect. A separate, non-circular reliability concern: Table 1 lists different Base accuracies for the same supposedly frozen model as N changes (e.g., Llama3-1B on Math-500: 20.8, 26.80, 33.60, 36.80), which is impossible for a fixed model and undermines the baseline comparisons; this is a correctness issue rather than a circularity. Overall, the central empirical claim is partially circular in evaluation but not derivatively circular.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method relies on several unstated or hand-chosen hyperparameters and two untested assumptions about verifier reliability and the meaning of test-time accuracy. No new entities are introduced.

free parameters (5)
  • verifier threshold tau = 0.99 for GSM8K/Math-500, 0.9 for AIME
    Chosen by hand per dataset to filter pseudo-labels; no sensitivity analysis reported (Section 4.2).
  • LoRA rank = 128 (most cases), 8 (AIME 2024 low-resource)
    Chosen empirically; no ablation shown (Section 4.2).
  • number of LoRA steps M = not reported
    A key training hyperparameter that is not stated in the paper.
  • learning rate eta = not reported
    Not specified.
  • sampling temperature T = not reported
    Only described as 'positive temperature'; actual value not given (Section 3.1).
assumptions (3)
  • domain assumption Verifier scores are reliable proxies for answer correctness on math problems
    The entire pseudo-label filtering depends on this; the paper uses Skywork PRM but does not validate its accuracy on these benchmarks (Section 3.2).
  • domain assumption Fine-tuning on verifier-selected pseudo-labels for test queries improves the model's ability on those queries in a way that reflects task competence
    The evaluation measures accuracy on the same queries used for adaptation, so this assumption is untested (Section 4.3).
  • domain assumption LoRA adaptation preserves base model capabilities and prevents catastrophic forgetting
    Claimed in Section 3.3 but no forgetting evaluation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection." pith.science (2026). https://pith.science/paper/DZOW5YAZ

@misc{pith2026250519475,
  author       = {Pith},
  title        = {Pith review of: Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZOW5YAZ}},
  note         = {Machine review of arXiv:2505.19475}
}
read the original abstract

Learning to adapt pretrained language models to unlabeled, out-of-distribution data is a critical challenge, as models often falter on structurally novel reasoning tasks even while excelling within their training distribution. We introduce a new framework called VDS-TTT - Verifier-Driven Sample Selection for Test-Time Training to efficiently address this. We use a learned verifier to score a pool of generated responses and select only from high ranking pseudo-labeled examples for fine-tuned adaptation. Specifically, for each input query our LLM generates N candidate answers; the verifier assigns a reliability score to each, and the response with the highest confidence and above a fixed threshold is paired with its query for test-time training. We fine-tune only low-rank LoRA adapter parameters, ensuring adaptation efficiency and fast convergence. Our proposed self-supervised framework is the first to synthesize verifier driven test-time training data for continuous self-improvement of the model. Experiments across three diverse benchmarks and three state-of-the-art LLMs demonstrate that VDS-TTT yields up to a 32.29% relative improvement over the base model and a 6.66% gain compared to verifier-based methods without test-time training, highlighting its effectiveness and efficiency for on-the-fly large language model adaptation.

Figures

Figures reproduced from arXiv: 2505.19475 by the authors.

Figure 1
Figure 1. In our VDS-TTT framework, we proceed through three sequential stages. First, Candidate Generation for Self-Annotation, where each input question Qi is passed to the pretrained LLM to produce a set of N candidate responses {r1, ..., rN }. Second, Confidence-Guided Annotation, in which a verifier assigns a reliability score to each rj , and we select the response r ∗ only if its score exceeds a predefined threshold τ … view at source ↗
Figure 2
Figure 2. Three instances of TTT loss curves improvements, as the model becomes increasingly capable of generating higher-quality task-relevant outputs. However, due to inherent model capacity limitations, this iterative improvement eventually exhibits diminishing returns. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Iterative VDS-TTT results 5 Conclusion We have presented Verifier-Driven sample Selection Test-Time Training (VDS-TTT), a simple yet powerful framework for self-supervised continuous improvement of an LLM model. For each test question, that the LLM responds with high confidence, we choose the single best answer as per 8 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

  1. [5]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

  2. [9]

    Superhf: Supervised iterative learning from human feedback.arXiv preprint arXiv:2310.16763,

    Gabriel Mukobi, Peter Chatain, Su Fong, Robert Windesheim, Gitta Kutyniok, Kush Bhatia, and Silas Alberti. Superhf: Supervised iterative learning from human feedback.arXiv preprint arXiv:2310.16763,

    arXiv

  3. [10]

    The entropy enigma: Success and failure of entropy minimization.arXiv preprint arXiv:2405.05012,

    Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization.arXiv preprint arXiv:2405.05012,

    arXiv

  4. [11]

    The effect of sampling temperature on problem solving in large language models

    Matthew Renze. The effect of sampling temperature on problem solving in large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356,

    work page 2024

  5. [12]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  6. [14]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  7. [16]

    Continual learning for large language models: A survey.arXiv preprint arXiv:2402.01364,

    Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey.arXiv preprint arXiv:2402.01364,

    arXiv

  8. [17]

    Beyond model adaptation at test time: A survey.arXiv preprint arXiv:2411.03687,

    Zehao Xiao and Cees GM Snoek. Beyond model adaptation at test time: A survey.arXiv preprint arXiv:2411.03687,

    arXiv

Show all 18 references
  1. [18]

    Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084,

    Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning.arXiv preprint arXiv:2504.16084,

    arXiv

  2. [1988]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  3. [1992]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390,

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390,

    arXiv

  4. [2017]

    Scaling test-time compute without verification or rl is suboptimal.arXiv preprint arXiv:2502.12118,

    Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal.arXiv preprint arXiv:2502.12118,

    arXiv

  5. [2020]

    Tent: Fully test-time adaptation by entropy minimization.arXiv preprint arXiv:2006.10726,

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization.arXiv preprint arXiv:2006.10726,

    2006 arXiv

  6. [2021]

    Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998,

    9 Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998,

    arXiv

  7. [2022]

    Open-reasoner- zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290,

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner- zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290,

    arXiv

  8. [2023]

    Test-time training on nearest neighbors for large language models.arXiv preprint arXiv:2305.18466,

    Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models.arXiv preprint arXiv:2305.18466,

    arXiv

  9. [2024]

    Test-time computing: from system-1 thinking to system-2 thinking.arXiv preprint arXiv:2501.02497,

    Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Jia Xu, Linjian Mo, and Min Zhang. Test-time computing: from system-1 thinking to system-2 thinking.arXiv preprint arXiv:2501.02497,

    arXiv

  10. [2025]

    Efficiently learning at test-time: Active fine-tuning of llms.arXiv preprint arXiv:2410.08020,

    Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms.arXiv preprint arXiv:2410.08020,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.