Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Test-Time Learning for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Lowering test-input perplexity lifts LLM accuracy by 20%

desk verdict The headline 20% improvement comes from an offline setting where the model is adapted on the exact test inputs it is later scored on, and the online results that avoid this are much smaller on several domains; the idea is interesting but the evidence overclaims. read the letter →

arxiv 2505.20633 v1 pith:4ZNVFY6H submitted 2025-05-27 cs.CL cs.AIcs.LG

Jinwu Hu , Zhitian Zhang , Guohao Chen , Xutao Wen , Chao Shuai , Wei Luo , Bin Xiao , Yuanqing Li
show 1 more author
Mingkui Tan
This is my paper
classification cs.CLcs.AIcs.LG
keywords test-timelearninglargelanguagemodelsperplexityminimizationdomainadaptationsampleselectionlow-rankdistributionshiftself-supervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a large language model can be made more accurate on a new domain without any labels by taking gradient steps that reduce the model's perplexity on the unlabeled test inputs themselves. The load-bearing idea is a local argument: if the direction that improves the model's probability of a question also improves its probability of the corresponding answer, then minimizing input perplexity will increase answer probability. The authors build a complete method, TLM, which selects the highest-perplexity test samples (the most informative ones) and updates only LoRA adapters, and they introduce a benchmark, AdaptEval, spanning domain knowledge, instruction following, and reasoning. They report performance gains of at least 20% over base LLMs on four domain-knowledge datasets, with additional gains on instruction and reasoning benchmarks.

What carries the argument

The key machinery is the cross-gradient identity derived in Eqn. (4): after one step $\Theta' = \Theta - \eta \nabla_\Theta (-\log P(x;\Theta))$, the change in output log-probability is $\log P_{\Theta'}(y|x) - \log P_{\Theta}(y|x) \approx \eta\, [\nabla_\Theta \log P(x;\Theta)]^\top \nabla_\Theta \log P_{\Theta}(y|x)$, up to $O(\eta^2)$ terms. The argument hinges on the non-negativity of that inner product for semantically aligned question-answer pairs; when it holds, the update lowers input perplexity and raises answer probability for small $\eta$. Two supporting mechanisms carry the method: the sample-selection score $S(x) = \lambda e^{\log P(x;\Theta)-\log P_0} \mathbf{1}\{P(x;\Theta) > P_0\}$, which weights high-perplexity samples more heavily, and Low-Rank Adaptation (LoRA), which restricts updates to a small parameter subspace to limit forgetting.

What would settle it

Compute the cross-gradient inner product $\langle \nabla_x, \nabla_y \rangle$ on a diverse set of question-answer pairs beyond DomainBench, especially high-perplexity ones; if a substantial fraction of these inner products are negative, then input-perplexity minimization would lower answer probability for those pairs, directly contradicting the paper's central claim. A concrete check: run TLM on a corpus of questions paired with mismatched answers and measure whether the updated model's answer perplexity increases.

Watch

Extended reading notes

Core claim

The paper's central claim is that minimizing the perplexity of unlabeled test inputs improves the model's predictions on those inputs. For a question-answer pair $(x,y)$, the paper derives a first-order Taylor expansion: a single gradient step on $-\log P(x;\Theta)$ changes $\log P_{\Theta}(y|x)$ by $\eta$ times the inner product of the gradients of the input and output log-likelihoods, up to $O(\eta^2)$ remainder. When this cross-gradient inner product is non-negative, the step is guaranteed to raise answer probability. The authors report that 98.75% of 400 DomainBench batches satisfy this non-negativity with average $+5.60$, and they show empirically that input and output perplexity move together on four vertical domains. This motivates the full TLM method: perplexity-weighted selection of high-perplexity samples and LoRA-constrained updates, evaluated on the AdaptEval benchmark.

Load-bearing premise

The method's guarantee depends on the assumption that for question-answer pairs with strong semantic alignment, the parameter direction that lowers input perplexity also raises answer probability (the cross-gradient inner product is non-negative); this is verified empirically on only 400 batches, not proven.

Editorial extensions

If this is right

  • On four domain-knowledge benchmarks (Geography, Agriculture, Medicine, Finance), TLM improves performance by at least 20% over the original LLM across Llama and Qwen model families.
  • The same method yields additional gains on instruction-following (Alpaca-GPT4, Dolly, InstructionWild) and reasoning (GSM8K, MetaMath, LogiQA) benchmarks compared to the base models.
  • Because the sample-selection weight excludes low-perplexity samples, TLM reduces training cost: in the online setting it uses 69.7% fewer backward passes than the entropy-based baselines while improving accuracy.
  • Low-rank (LoRA) updates preserve general knowledge better than full-parameter updates, as measured by keeping GSM8K accuracy near its original level during domain adaptation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the cross-gradient non-negativity holds beyond the 400 DomainBench batches examined, the same input-perplexity objective could be combined with other test-time losses (e.g., prediction consistency or confidence) without changing the argument; the paper does not explore such composite objectives.
  • The fixed threshold $P_0 = e^3$ is chosen from a sweep that shows sensitivity across datasets; an adaptive threshold per domain would likely transfer better to unseen domains, and Figure 3 suggests a plateau rather than a single optimal value.
  • The offline evaluation adapts the model on the full test set and scores it on the same set; a stricter evaluation that adapts on one batch and scores on a disjoint batch of the same distribution would clarify how much of the gain is genuine distribution adaptation versus fitting the specific test inputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TLM, a test-time learning method for large language models that adapts model parameters by minimizing the input perplexity of unlabeled test data. The method is motivated by a Taylor-expansion argument (Eq. 4) that a gradient step on -log P(x) increases log P(y|x) when a cross-gradient term is non-negative, and it adds a high-perplexity sample selection mechanism, uses LoRA for lightweight updates, and introduces the AdaptEval benchmark. The headline claim, repeated in the abstract and Section 5.2, is that TLM improves domain-knowledge adaptation by at least 20% over original LLMs.

Significance. Test-time adaptation of LLMs without labels is an active and important problem, and the idea of using input perplexity as a self-supervised adaptation signal is intuitive and potentially useful. The paper also contributes a new benchmark, experiments across multiple LLM families and sizes, both offline and online settings, quantized-LLM results, and a public code repository. However, the central quantitative claim is not supported by the paper's own tables, and the theoretical justification rests on an unverified sign assumption. These issues affect the paper's main contribution rather than its presentation only.

major comments (4)
  1. [Abstract; Section 5.2; Table 2] The claim that TLM improves DomainBench performance by at least 20% over original LLMs is contradicted by Table 2 itself. For Llama2-13B-chat on Finance, the Rouge-Lsum score moves from 0.2382 to 0.2760, a relative gain of only 15.9%; for Qwen2.5-7B-Instruct on Geography, it moves from 0.2649 to 0.3081, a relative gain of 16.3%. The statement in Section 5.2 that 'the proposed TLM achieves at least a 20.00% improvement over the original LLMs' is therefore false for at least these cells, so the abstract's central quantitative claim needs to be substantially qualified or corrected.
  2. [Supplementary C.2; Table 5] The headline improvements in Table 2 are measured in the offline setting described in Supp. C.2, where all test data is processed and the model is updated before any testing. This means the model sees the exact inputs it is later scored on. The online results in Table 5, which are closer to a realistic deployment where outputs are produced before the corresponding updates, show much smaller relative gains for Llama3-8B-Instruct: +13.8% on Geography, +5.9% on Medicine, and +5.4% on Finance, with only 1514 backward passes. The paper does not reconcile these discrepancies, and the 'at least 20%' claim is not reproduced in the online setting for these datasets.
  3. [Section 4.1; Eq. (4)] The theoretical argument that minimizing input perplexity improves output likelihood depends entirely on the non-negativity of the cross-gradient term. This is introduced as 'our core assumption' and is not derived from Assumptions 1 and 2, which are informal and do not imply the inequality. The only empirical support is a check on 400 batches from DomainBench on LLaMA3.1-8B, with no per-domain breakdown, no other model, and no discussion of the 1.25% of batches where the condition fails. Since the whole method rests on this sign condition, the paper should either prove it under stated assumptions or validate it much more broadly and show that it holds during actual online adaptation.
  4. [Section 5.3; Figure 3] The hyperparameters lambda and P0 in Eq. (6) are selected using the same evaluation data: Figure 3 shows that P0 = e3 is chosen because it gives the best or near-best results on the DomainBench test sets, and the same test sets are used for the reported comparisons. This makes the headline numbers partly a result of test-set hyperparameter selection. The authors should use a validation split or otherwise show that the chosen values are not tuned to the evaluation set.
minor comments (5)
  1. [Section 4.1, Eq. (4)] The term 'O(η2)' should be typeset as O(η^2) with a proper superscript.
  2. [Section 4.1] The model used for the cross-gradient check is called 'LLaMA3.1-8B', but the main experiments use Llama3.2-3B-Instruct, Llama3-8B-Instruct, Llama2-13B-chat, and Qwen2.5-7B-Instruct; the model naming should be made consistent and the check should be tied to the evaluated models.
  3. [Section 5.2] The text says 'our proposed TLM improves the performance of Llama3.2-8B-Instruct by 13.91%' on Alpaca-GPT4, but Table 2 reports the relevant row for Llama3-8B-Instruct; the model name appears to be a typo.
  4. [Section 4.2, Eq. (6)] The sentence 'λ and P0 are a pre-defined threshold' is grammatically and semantically imprecise: λ is a scaling factor and P0 is a threshold; the sentence should be rephrased.
  5. [Figure 1b; Observation 1] The claim of a 'strong positive correlation' between input and output perplexity is supported only by normalized curves in Figure 1b; reporting a Spearman correlation with confidence intervals would make the observation quantitative and testable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core theorem is a conditional Taylor-expansion result with an explicitly tested non-negativity assumption.

full rationale

The derivation chain is self-contained and not circular. Eq. (4) is an exact first-order Taylor expansion: after an input-perplexity gradient step, the change in output log-probability is, to first order, η times the cross-gradient inner product. The paper explicitly labels the non-negativity of that inner product as a 'core assumption' and tests it empirically on 400 DomainBench batches (98.75% non-negative, mean +5.60). The theoretical claim is therefore conditional on a stated, independently measured premise rather than a definition of the conclusion; no equation is fitted and then renamed as a prediction. The use of unlabeled test data to adapt before scoring is the intended transductive test-time protocol, and the online experiments (Table 5) provide a separate, more conservative evaluation; the gap between offline and online gains is an evaluation-validity concern, not a circularity. Self-citations to the authors' prior work (Hu et al. 2025, Wang et al. 2025) appear only as background context; the load-bearing citations (LoRA, Tent, EATA, Perplexity) are external. No circular step is present, so the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. Its free parameters are standard hyperparameters plus the selection threshold P0, which is tuned on the benchmark. The central theoretical claim relies on an unproven gradient-alignment axiom.

free parameters (3)
  • λ (sample selection scale) = 0.10
    Set as a fixed constant in Eqn. (6), no sensitivity analysis is reported.
  • P0 (perplexity threshold) = e^3
    Selected by an experiment sweeping over {e2, e3, e4, e5, e6} and choosing the value that maximizes performance on the same benchmark.
  • Learning rate = 5e-5 for DomainBench/InstructionBench, 1e-6 for ReasoningBench
    Standard hyperparameter, but it varies across benchmarks, implying tuning on the test set.
assumptions (3)
  • domain assumption Autoregressive Property (Assumption 1)
    The LLM generates each token conditioned on previous tokens and the input. Standard for decoder-only models but not formally proven.
  • domain assumption Shared Parameter Influence (Assumption 2)
    The same parameters influence both input and output perplexity. This is plausible for a single model but not formalized.
  • ad hoc to paper Non-negative cross-gradient: [∇Θ log P(x;Θ)]^T ∇Θ log PΘ(y|x) ≥ 0
    Core assumption that input and output likelihood gradients align. Empirically measured on 400 batches from DomainBench, but not proven and may not hold generally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Test-Time Learning for Large Language Models." pith.science (2026). https://pith.science/paper/4ZNVFY6H

@misc{pith2026250520633,
  author       = {Pith},
  title        = {Pith review of: Test-Time Learning for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ZNVFY6H}},
  note         = {Machine review of arXiv:2505.20633}
}
read the original abstract

While Large Language Models (LLMs) have exhibited remarkable emergent capabilities through extensive pre-training, they still face critical limitations in generalizing to specialized domains and handling diverse linguistic variations, known as distribution shifts. In this paper, we propose a Test-Time Learning (TTL) paradigm for LLMs, namely TLM, which dynamically adapts LLMs to target domains using only unlabeled test data during testing. Specifically, we first provide empirical evidence and theoretical insights to reveal that more accurate predictions from LLMs can be achieved by minimizing the input perplexity of the unlabeled test data. Based on this insight, we formulate the Test-Time Learning process of LLMs as input perplexity minimization, enabling self-supervised enhancement of LLM performance. Furthermore, we observe that high-perplexity samples tend to be more informative for model optimization. Accordingly, we introduce a Sample Efficient Learning Strategy that actively selects and emphasizes these high-perplexity samples for test-time updates. Lastly, to mitigate catastrophic forgetting and ensure adaptation stability, we adopt Low-Rank Adaptation (LoRA) instead of full-parameter optimization, which allows lightweight model updates while preserving more original knowledge from the model. We introduce the AdaptEval benchmark for TTL and demonstrate through experiments that TLM improves performance by at least 20% compared to original LLMs on domain knowledge adaptation.

Figures

Figures reproduced from arXiv: 2505.20633 by the authors.

Figure 1
Figure 1. Summary of our exploration and observations: (a) demonstrates that perplexity minimization improves the performance of LLMs, while entropy minimization (Wang et al., 2021) may harm their performance; (b) reveals that the trend of LLM’s perplexity to the input P(x) and perplexity to the output P(y|x) is the same (results are normalized), i.e., we can min Θ P(y|x; Θ) by min Θ P(x; Θ); and (c) emphasizes that training … view at source ↗
Figure 2
Figure 2. Comparison of prevent forgetting on DomainBench un￾der Llama3.1-8B-Instruct. This observation reveals that LoRA (Hu et al., 2022) prevents catastrophic forgetting more effectively than Full-Param updates across DomainBench (see Supp. B). ing degrees of training (for ease of presentation, we show the normalized results here). As shown in Figure 1b, the relationship between input perplexity P(x; Θ) and output perplexi… view at source ↗
Figure 3
Figure 3. Effects of different perplexity margins P0 in Eqn. 6. GPT4 dataset, our proposed TLM improves the perfor￾mance of Llama3.2-8B-Instruct by 13.91% (0.3752 → 0.4274), showing a relative improvement of about 113.60% (0.2001 → 0.4274) compared to Tent, demonstrating its effective adaptation to general instruction-following tasks. Superior performance on logical reasoning task. As shown in [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distributions of AdaptEval. backpropagation by learning a prompt through a derivative-free covariance matrix adaptation strategy and adjusting model activations to align with the source training domain. Looking forward, there is great potential to extend our method to …
Figure 5
Figure 5. Figure 5: Distribution of Sequence Lengths for Samples in DomainBench. linguistic styles, providing a thorough assessment of the model’s capacity to process and respond effectively to diverse instructions in real-world scenarios. • ReasoningBench. This category contains three re…
Figure 6
Figure 6. Figure 6: Distribution of Sequence Lengths for Samples in InstructionBench. 50 100 150 200 250 300 350 400 Input Sequence Length 0 50 100 150 200 250 300 350 400 Density Datasets GSM8K Logiqa MetaMath 0 2 4 6 8 10 12 14 Output Sequence Length 0 1000 2000 3000 4000 5000 Density D…
Figure 7
Figure 7. Figure 7: Distribution of Sequence Lengths for Samples in ReasoningBench. dataset is constructed by first manually creating a comprehensive set of instructions across a wide range of tasks, followed by data generation and quality assurance using GPT-4. It includes diverse task t…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A tile-based memory layout for mobile GPUs that unifies forward and backward data access, eliminating most transpose/reshape overhead and speeding LLM fine-tuning 2.2–5.7× in the paper's measurements.

  2. DeliCIR: Memory-Guided Test-Time Deliberation via Multi-Agent Collaboration for Composed Image Retrieval

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Proposes PDF, a hierarchical multi-agent Perception-to-Deliberation Framework that adds experience self-evolution and test-time scaling to composed image retrieval, claiming SOTA on CIRR, CIRCO, and FashionIQ.

  3. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Using Pass@k as an RLVR reward, with bootstrap sampling and an analytical advantage formula, improves exploration and later Pass@1 performance of reasoning LLMs.

Reference graph

Works this paper leans on

92 extracted references · 40 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    The surprising effectiveness of test-time training for abstract reasoning

    Aky \"u rek, E., Damani, M., Qiu, L., Guo, H., Kim, Y., and Andreas, J. The surprising effectiveness of test-time training for abstract reasoning. arXiv preprint arXiv:2411.07279, 2024

    arXiv 2024

  4. [4]

    M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al

    Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  5. [5]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024

    2024

  6. [6]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  7. [7]

    u hler, A., Wiewel, F., D \

    Bartler, A., B \"u hler, A., Wiewel, F., D \"o bler, M., and Yang, B. Mt3: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics, pp.\ 3080--3090. PMLR, 2022

    2022

  8. [8]

    Tackling language modelling bias in support of linguistic diversity

    Bella, G., Helm, P., Koch, G., and Giunchiglia, F. Tackling language modelling bias in support of linguistic diversity. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp.\ 562--572, 2024

    2024

Show all 92 references
  1. [9]

    A neural probabilistic language model

    Bengio, Y., Ducharme, R., and Vincent, P. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000

    2000

  2. [10]

    Parameter-free online test-time adaptation

    Boudiaf, M., Mueller, R., Ben Ayed, I., and Bertinetto, L. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8344--8353, 2022

    2022

  3. [11]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  4. [12]

    A survey on evaluation of large language models

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 0 (3): 0 1--45, 2024

    2024

  5. [13]

    Cross-device collaborative test-time adaptation

    Chen, G., Niu, S., Chen, D., Zhang, S., Li, C., Li, Y., and Tan, M. Cross-device collaborative test-time adaptation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a

    2024

  6. [14]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  7. [15]

    Towards robust and efficient cloud-edge elastic model adaptation via selective entropy distillation

    Chen, Y., Niu, S., Wang, Y., Xu, S., Song, H., and Tan, M. Towards robust and efficient cloud-edge elastic model adaptation via selective entropy distillation. In The Twelfth International Conference on Learning Representations, 2024 b

    2024

  8. [16]

    Longlora: Efficient fine-tuning of long-context large language models

    Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., and Jia, J. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2024 c

    2024

  9. [17]

    W., Sutton, C., Gehrmann, S., et al

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  10. [18]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

    2024

  11. [19]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  12. [20]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  13. [21]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    2019 arXiv

  14. [22]

    GLM : General language model pretraining with autoregressive blank infilling

    Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. GLM : General language model pretraining with autoregressive blank infilling. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computation...

    2022 doi

  15. [23]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  16. [24]

    A survey on rag meeting llms: Towards retrieval-augmented large language models

    Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., and Li, Q. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.\ 6491--6501, 2024

    2024

  17. [25]

    Fleuret, F. et al. Test time adaptation through perturbation robustness. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021

    2021

  18. [26]

    Test-time training with masked autoencoders

    Gandelsman, Y., Sun, Y., Chen, X., and Efros, A. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35: 0 29374--29385, 2022

    2022

  19. [27]

    Back to the source: Diffusion-driven adaptation to test-time corruption

    Gao, J., Zhang, J., Liu, X., Darrell, T., Shelhamer, E., and Wang, D. Back to the source: Diffusion-driven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11786--11796, 2023

    2023

  20. [28]

    Domain-specific language model pretraining for biomedical natural language processing

    Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3 0 (1): 0 1--23, 2021

    2021

  21. [29]

    and Sun, Y

    Hardt, M. and Sun, Y. Test-time training on nearest neighbors for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  22. [30]

    Deberta: Decoding-enhanced bert with disentangled attention

    He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021

    2021

  23. [31]

    Mecta: Memory-economic continual test-time model adaptation

    Hong, J., Lyu, L., Zhou, J., and Spranger, M. Mecta: Memory-economic continual test-time model adaptation. In 2023 International Conference on Learning Representations, 2023

    2023

  24. [32]

    J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al

    Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  25. [33]

    Dynamic ensemble reasoning for llm experts

    Hu, J., Wang, Y., Zhang, S., Zhou, K., Chen, G., Hu, Y., Xiao, B., and Tan, M. Dynamic ensemble reasoning for llm experts. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, 2025 a

    2025

  26. [34]

    Dynamic compressing prompts for efficient inference of large language models

    Hu, J., Zhang, W., Wang, Y., Hu, Y., Xiao, B., Tan, M., and Du, Q. Dynamic compressing prompts for efficient inference of large language models. arXiv preprint arXiv:2504.11004, 2025 b

    2025 arXiv

  27. [35]

    Efficiently learning at test-time: Active fine-tuning of llms

    H \"u botter, J., Bongni, S., Hakimi, I., and Krause, A. Efficiently learning at test-time: Active fine-tuning of llms. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024

    2024

  28. [36]

    and Matsuo, Y

    Iwasawa, Y. and Matsuo, Y. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34: 0 2427--2440, 2021

    2021

  29. [37]

    Active retrieval augmented generation

    Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\...

    2023 doi

  30. [38]

    Longrag: Enhancing retrieval-augmented generation with long-context llms

    Jiang, Z., Ma, X., and Chen, W. Longrag: Enhancing retrieval-augmented generation with long-context llms. arXiv preprint arXiv:2406.15319, 2024

    2024 arXiv

  31. [39]

    and Zuidema, W

    Jumelet, J. and Zuidema, W. Transparency at the source: Evaluating and interpreting language models with access to the true distribution. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 4354--4369, 2023

    2023

  32. [40]

    Dense passage retrieval for open-domain question answering

    Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Languag...

    2020 doi

  33. [41]

    Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models

    Kim, G., Kim, S., Jeon, B., Park, J., and Kang, J. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=vDvFT7IX4O

    2023

  34. [42]

    S., Yvon, F., Gall \'e , M., et al

    Le Scao, T., Fan, A., Akiki, C., Pavlick, E., Ili \'c , S., Hesslow, D., Castagn \'e , R., Luccioni, A. S., Yvon, F., Gall \'e , M., et al. Bloom: A 176b-parameter open-access multilingual language model. 2023

    2023

  35. [43]

    Entropy is not enough for test-time adaptation: From the perspective of disentangled factors

    Lee, J., Jung, D., Lee, S., Park, J., Shin, J., Hwang, U., and Yoon, S. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=...

    2024

  36. [44]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Associ...

    2020

  37. [45]

    u ttler, H., Lewis, M., Yih, W.-t., Rockt \

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.-t., Rockt \"a schel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 0 9459--9474, 2020 b

    2020

  38. [46]

    From classification to generation: Insights into crosslingual retrieval augmented ICL

    Li, X., Nie, E., and Liang, S. From classification to generation: Insights into crosslingual retrieval augmented ICL . In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https://openreview.net/forum?id=KLPLCXo4aD

    2023

  39. [47]

    A comprehensive survey on test-time adaptation under distribution shifts

    Liang, J., He, R., and Tan, T. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, pp.\ 1--34, 2024

    2024

  40. [48]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  41. [49]

    V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., Zettlemoyer, L., and tau Yih, W

    Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., Zettlemoyer, L., and tau Yih, W. RA - DIT : Retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Learning Representations, 2024....

    2024

  42. [50]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019

    1907 arXiv

  43. [51]

    Ttt++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34: 0 21808--21820, 2021

    Liu, Y., Kothari, P., Van Delft, B., Bellot-Gurlet, B., Mordan, T., and Alahi, A. Ttt++: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34: 0 21808--21820, 2021

    2021

  44. [52]

    J., Soneira, P

    Mirza, M. J., Soneira, P. J., Lin, W., Kozinski, M., Possegger, H., and Bischof, H. Actmad: Activation matching to align distributions for test-time-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 24152--24161, 2023

    2023

  45. [53]

    Evaluating prediction-time batch normalization for robustness under covariate shift

    Nado, Z., Padhy, S., Sculley, D., D'Amour, A., Lakshminarayanan, B., and Snoek, J. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963, 2020

    2006 arXiv

  46. [54]

    Efficient test-time model adaptation without forgetting

    Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp.\ 16888--16905. PMLR, 2022 a

    2022

  47. [55]

    Boost test-time performance with closed-loop inference

    Niu, S., Wu, J., Zhang, Y., Xu, G., Li, H., Zhao, P., Huang, J., Wang, Y., and Tan, M. Boost test-time performance with closed-loop inference. arXiv preprint arXiv:2203.10853, 2022 b

    2022 arXiv

  48. [56]

    Towards stable test-time adaptation in dynamic wild world

    Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023

    2023

  49. [57]

    Test-time model adaptation with only forward passes

    Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test-time model adaptation with only forward passes. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=qz1Vx1v9iK

    2024

  50. [58]

    Efficient diffusion-driven corruption editor for test-time adaptation

    Oh, Y., Lee, J., Choi, J., Jung, D., Hwang, U., and Yoon, S. Efficient diffusion-driven corruption editor for test-time adaptation. In European Conference on Computer Vision, pp.\ 184--201. Springer, 2025

    2025

  51. [59]

    Bleu: a method for automatic evaluation of machine translation

    Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

    2002

  52. [60]

    Check your facts and try again: Improving large language models with external knowledge and automated feedback

    Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813, 2023

    2023 arXiv

  53. [61]

    Memorag: Moving towards next-gen rag via memory-inspired knowledge discovery

    Qian, H., Zhang, P., Liu, Z., Mao, K., and Dou, Z. Memorag: Moving towards next-gen rag via memory-inspired knowledge discovery. arXiv preprint arXiv:2409.05591, 2024

    2024 arXiv

  54. [62]

    Improving language understanding by generative pre-training

    Radford, A. Improving language understanding by generative pre-training. 2018

    2018

  55. [63]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  56. [64]

    In-context retrieval-augmented language models

    Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., and Shoham, Y. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11: 0 1316--1331, 2023. doi:10.1162/tacl_a_00605. URL https://aclantho...

    2023 doi

  57. [65]

    and Gurevych, I

    Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://api.semanticscholar.org/CorpusID:201646309

    2019

  58. [66]

    Retrieve-and-sample: Document-level event argument extraction via hybrid retrieval augmentation

    Ren, Y., Cao, Y., Guo, P., Fang, F., Ma, W., and Lin, Z. Retrieve-and-sample: Document-level event argument extraction via hybrid retrieval augmentation. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Compu...

    2023 doi

  59. [67]

    Improving robustness against common corruptions by covariate shift adaptation

    Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., and Bethge, M. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33: 0 11539--11551, 2020

    2020

  60. [68]

    Prompting large language models with answer heuristics for knowledge-based visual question answering

    Shao, Z., Yu, Z., Wang, M., and Yu, J. Prompting large language models with answer heuristics for knowledge-based visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp.\ 14974--14983, 2023

    2023

  61. [69]

    Test-time prompt tuning for zero-shot generalization in vision-language models

    Shu, M., Nie, W., Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, C. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35: 0 14274--14289, 2022

    2022

  62. [70]

    S., and Choi, S

    Song, J., Lee, J., Kweon, I. S., and Choi, S. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11920--11929, 2023

    2023

  63. [71]

    Test-time training with self-supervision for generalization under distribution shifts

    Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp.\ 9229--9248. PMLR, 2020

    2020

  64. [72]

    Q., Garcia, X., Wei, J., Wang, X., Chung, H

    Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Bahri, D., Schuster, T., Zheng, S., Zhou, D., Houlsby, N., and Metzler, D. UL 2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2023. U...

    2023

  65. [73]

    J., Ting, D

    Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W. Large language models in medicine. Nature medicine, 29 0 (8): 0 1930--1940, 2023

    1930

  66. [74]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  67. [75]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  68. [76]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions

    Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association ...

    2023 doi

  69. [77]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  70. [78]

    Tent: Fully test-time adaptation by entropy minimization

    Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations, 2021

    2021

  71. [79]

    Generating long-form story using dynamic hierarchical outlining with memory-enhancement

    Wang, Q., Hu, J., Li, Z., Wang, Y., Hu, Y., Tan, M., et al. Generating long-form story using dynamic hierarchical outlining with memory-enhancement. The 2025 Annual Conference of the Nations of the Americas Chapter of the ACL, 2025 a

    2025

  72. [80]

    Self-knowledge guided retrieval augmentation for large language models

    Wang, Y., Li, P., Sun, M., and Liu, Y. Self-knowledge guided retrieval augmentation for large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=MoEfm3iPMy

    2023

  73. [81]

    Enhancing user-oriented proactivity in open-domain dialogues with critic guidance

    Wang, Y., Hu, J., Huang, Z., Lin, K., Zhang, Z., Chen, P., Hu, Y., Wang, Q., Yu, Z., Sun, B., Xing, X., Zheng, Q., and Tan, M. Enhancing user-oriented proactivity in open-domain dialogues with critic guidance. Proceedings of the Thirty-Fourth International Joint Conference on ...

    2025

  74. [82]

    H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Researc...

    2022

  75. [83]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022 b

    2022

  76. [84]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  77. [85]

    Yi, C., Chen, H., Zhang, Y., Xu, Y., Zhou, Y., and Cui, L. From question to exploration: Can classic test-time adaptation strategies be effectively applied in semantic segmentation? In Proceedings of the 32nd ACM International Conference on Multimedia, pp.\ 10085--10094, 2024

    2024

  78. [86]

    Memo: Test time robustness via adaptation and augmentation

    Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35: 0 38629--38642, 2022 a

    2022

  79. [87]

    Come: Test-time adaption by conservatively minimizing entropy

    Zhang, Q., Bian, Y., Kong, X., Zhao, P., and Zhang, C. Come: Test-time adaption by conservatively minimizing entropy. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  80. [88]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022 b

    2022 arXiv

  81. [89]

    Q., and Artzi, Y

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  82. [90]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  83. [91]

    Memorybank: Enhancing large language models with long-term memory

    Zhong, W., Guo, L., Gao, Q., Ye, H., and Wang, Y. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 19724--19731, 2024

    2024

  84. [92]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp.\ 19--27, 2015

    2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.