Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A language model's deepest attention layers already know which context sentence is the evidence; highlighting it improves answers with no training.

desk verdict A clean, training-free evidence-highlighting method that reliably improves grounded QA, but the 'secretly knows evidence' narrative overreaches because most evidence labels are just answer-string containment. read the letter →

arxiv 2502.08767 v2 pith:2NV53PTJ submitted 2025-02-12 cs.CL cs.AI

Zhining Liu , Rana Ali Amjad , Ravinarayana Adkathimar , Tianxin Wei , Hanghang Tong This is my paper · ORCID
classification cs.CLcs.AI
keywords SelfElicitevidenceelicitationattentionscoresinference-timeadaptationcontext-basedquestionansweringtraining-freegroundedgenerationretrieval-augmented
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SelfElicit claims that a language model already knows, inside its own attention patterns, which sentences in its input are the evidence it should use—even when it ultimately gives the wrong answer. The paper shows that the deepest half of the transformer layers consistently pays much higher attention to answer-bearing sentences, and turns this observation into a training-free inference procedure: generate one token, read the attention scores, highlight the sentences with the top scores using simple text markers, and let the model answer again. On four open-book QA datasets and six instruction-tuned models from three model families, this highlighting improves exact match and token F1 by 5.0% to 11.7% over direct answering, and outperforms chain-of-thought, whole-context highlighting, and generative evidence extraction while adding only a few percent of inference time. The authors further show that the highlighted sentences are indeed the relevant evidence in most cases, and that the method stays effective when the context is noisy.

What carries the argument

The load-bearing object is the sentence-level evidence score $e_i$, defined as the average, over the evidence-reading layers $L_{\mathrm{ER}}$ (the last 50% of layers), of the head-averaged attention probability assigned to sentence $s_i$ when the model produces its first token. A sentence is highlighted when $e_i \geq \alpha \max(e)$, with $\alpha = 0.5$ by default, and the markers `<start_important>` and `<end_important>` are inserted around it. This construction converts raw attention into a discrete, prompt-level intervention: no weights are changed, no extra model is trained, and only one additional token of generation is needed to obtain the scores.

What would settle it

Construct a dataset of context-question pairs where each context contains one sentence with the true answer and a second, superficially similar sentence with a plausible wrong answer, then measure whether SelfElicit's deep-layer attention assigns the higher score to the true sentence. If the method cannot separate these on held-out pairs, or if highlighting the top-scoring sentence fails to improve accuracy over highlighting a randomly chosen sentence, the claimed evidence-finding ability is not doing the causal work.

Watch

Extended reading notes

Core claim

The central claim is that decoder-only transformers have an inherent, layer-dependent evidence-finding ability: when generating the first answer token, their deeper layers distribute markedly more attention to the contextual sentences that contain the facts needed for the answer, and this is true whether or not the final answer ends up correct. The paper operationalizes this as a sentence-level evidence score obtained by averaging per-token attention over all heads in the last 50% of layers and then over the tokens of each sentence. Sentences whose score reaches half the maximum are marked with <start_important> and <end_important> tags, the prompt instructs the model to read marked sentences carefully, and the answer is regenerated. The authors report that this single extra step improves exact match and token F1 across all tested model-dataset combinations, with evidence-ranking quality (AUROC and NDCG) mostly between 80 and 95, and that the improvements hold under distractor noise and on contexts that should be rejected as unanswerable.

Load-bearing premise

The method assumes that the average head attention in the last half of the layers, taken from a single one-token forward pass, reliably identifies the sentences the model should rely on—an assumption validated empirically on these datasets rather than derived from a causal account of how attention produces answers.

Editorial extensions

If this is right

  • Explicit sentence highlighting from deep-layer attention improves exact match and token F1 consistently across all six tested models and four datasets, with gains of 5.0% to 11.7% over direct QA.
  • Attention-based evidence selection matches or beats generative evidence extraction while adding only about 3-5% inference overhead, whereas the generative baseline adds hundreds of percent.
  • The evidence scores rank true evidence sentences highly (AUROC and NDCG mostly 80-95), so the same scores could serve as a cheap saliency map for fact-grounded QA.
  • The method degrades gracefully under distractor noise: with a fixed threshold it automatically highlights a smaller fraction of the context, and it slightly improves rejection of unanswerable contexts.
  • Chain-of-thought prompting and whole-context highlighting do not produce consistent gains, indicating that fine-grained, sentence-level highlighting is the mechanism that matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: the same attention signal could be read at every decoding step, not just the first token, to re-highlight evidence dynamically during long or multi-step generations.
  • Editorial extension: because the method needs no training, it could be applied as a post-hoc diagnostic—flagging cases where the model's answer disagrees with the sentences its deep layers attend to, as a hallucination warning.
  • An untested variant: replacing the fixed threshold $\alpha$ with an adaptive threshold based on the gap between the top evidence scores could fix the multi-hop failure case the authors describe, where one hop is highlighted but the bridging sentence is missed.
  • The paper evaluates contexts up to a few thousand tokens; whether the signal survives much longer or highly duplicated contexts is an open question, since attention averages may flatten when many sentences compete.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SelfElicit, an inference-time, training-free method that uses the self-attention scores of deeper Transformer layers to identify supposedly relevant evidence sentences in a provided context, highlights those sentences with special markers, and asks the model to read them carefully before answering. The method is evaluated on four context-based QA datasets (HotpotQA, NewsQA, TriviaQA, Natural Questions) with six instruction-tuned LMs (Llama-3.1 8B/70B, Mistral 7B/12B, Qwen2.5 7B/32B). The authors report consistent Exact Match and Token F1 gains over a direct-answer baseline across all 24 model-dataset pairs, often outperforming a generative evidence-extraction baseline (PROMPT ELICIT) at much lower inference cost. They also report evidence-elicitation quality via AUROC and NDCG against ground-truth evidence labels, robustness to noise, and sensitivity analyses for the evidence-reading layers and the threshold α.

Significance. If the evidence-identification claim holds, SelfElicit is a simple, efficient, and broadly applicable method for improving grounded QA in retrieval-augmented settings, requiring no training and only one additional forward pass. The paper includes public code, detailed reproducibility notes, and a wide model-dataset coverage, which are clear strengths. The main weakness is that the quantitative evidence-identification evaluation is largely based on answer-string containment rather than human-labeled reasoning evidence, so the paper's central claim that LMs 'secretly know where relevant evidence is' is not yet established for most datasets. The QA gains could be partly an answer-hinting effect. This is a fixable concern through additional analysis on HotpotQA and, if the authors provide it, the paper would be a solid contribution.

major comments (4)
  1. [Section 4.2, Table 3, Appendix B.2] The evidence ground truth for NewsQA, TQA, and NQ is defined by answer-string containment ('a sentence is treated as ground truth evidence if it contains at least one of the correct answers'), as stated in Section 4.2 and acknowledged in Appendix B.2 to be 'not a rigorous approach.' Consequently, the AUROC/NDCG values in Table 3 mostly measure whether deep-layer attention highlights answer-bearing sentences, not whether it identifies reasoning evidence. Because the central claim is that LMs have an inherent evidence-finding ability, please provide a quantitative breakdown on HotpotQA (which has human supporting_facts annotations) separating cases where the selected evidence does and does not contain the answer string, and report QA gains separately for those cases. If gains persist when selected evidence does not contain the answer string, the evidence-finding narrative is supported; if they vanish, the method is largely answer-hinting rather than evidence-finding.
  2. [Section 4.1, Table 1, Section 4.2] No error bars, variance estimates, or significance tests are reported for any of the 24 model-dataset pairs. The text in Section 4.2 calls the gains 'significant,' but several improvements are small in absolute terms (e.g., Mistral-7B on NQ EM from 65.7 to 66.4) and no uncertainty quantification is given. Please provide bootstrap confidence intervals or paired significance tests over samples, or explicitly qualify the claim as 'consistent in sign across all pairs' rather than statistically significant. This is important because the method's main selling point is consistency across diverse settings.
  3. [Section 3.1, Eqs. (1)-(4), Algorithm 1] The sentence evidence scores are computed from a single forward pass that generates only one token. The paper does not quantify the stability of the resulting evidence scores and selected sentence sets with respect to this single-token choice. Since the entire method hinges on the reliability of these attention-based scores, please add an analysis of evidence-score stability across the first few decoding steps (or random seeds) and show that the highlighted sentences and downstream QA gains are stable.
  4. [Section 4.3 RQ4/RQ5, Section 4.1] The defaults LER = last 50% layers and α = 0.5 are chosen after inspecting the same test sets used in Table 1 (RQ4 uses HotpotQA, RQ5 uses all four datasets, both with Llama-3.1-8B). The statement in Section 4.1 that 'we do not tune the hyper-parameters' is therefore not accurate in an evaluation sense, because the settings are selected on the test data. Please either move hyperparameter selection to a development split and report results on a truly untouched test split, or make the robustness argument more direct by showing that the reported conclusions are unchanged for the full range of α ∈ [0.5, 1] and for several LER choices across all model-dataset pairs.
minor comments (5)
  1. [Figure 2] The y-axis label 'Relateive Attention' contains a typo; it should be 'Relative Attention'. Also, the definition of relative attention only appears in footnote 3; please move it into the main caption for readability.
  2. [Section 4.2] The phrase '5.0%-11.7% gain over baseline' should specify whether these are absolute percentage-point differences or relative improvements. The numbers in Table 1 (e.g., HotpotQA EM 58.9→68.5 for Llama-3.1-8B) suggest absolute EM/F1 points, but the text should be explicit.
  3. [Table 1] The 'Ranking' and 'Inference Time' average columns are not clearly described in the caption. Please state how the ranking is computed (e.g., averaged rank across datasets for EM and F1) and whether the reported inference time is per sample.
  4. [Appendix C.3] The 'Potential Risks' section contains a very long chain of citations that are not directly related to the paper's method or evaluation. Consider trimming this to a single sentence about known RAG risks and mitigation strategies.
  5. [Section 3.2] The SELF ELICIT prompt template uses the placeholder '{Original QA Instructions}', but the baseline template in Section 2 is a complete direct-answer instruction. It would be clearer to state explicitly that τSEQA is constructed by taking the baseline instructions verbatim and appending the marker guidance.

Circularity Check

1 steps flagged · score 4.0 of 10

Evidence-finding claim partially reduces to answer-string localization; QA gains remain externally benchmarked.

  1. renaming known result [Section 4.2 (RQ2) and Appendix B.2; Table 3]
    "For quantitative evaluation, we assess the accuracy of evidence elicitation by checking whether SELF ELICIT assigns higher evidence scores to ground-truth evidence sentences. Specifically, for HotpotQA, we use the "supporting_facts" annotations to derive ground-truth evidence labels, while for other datasets, a sentence is treated as ground truth evidence if it contains at least one of the correct answers."

    For three of the four datasets, the paper defines "ground-truth evidence" as any sentence containing a correct answer string. The Table 3 AUROC/NDCG scores therefore measure whether deep-layer attention localizes answer-bearing sentences, not whether it identifies multi-step reasoning evidence.

full rationale

The method's derivation is not itself circular: attention scores are obtained from a single forward pass (Eq. 1-2), aggregated over evidence-reading layers (Eq. 3), thresholded (Eq. 4), and used only to highlight sentences; no label or answer is fed back into the score computation, and the QA gains in Table 1 are measured against standard EM/F1 on external benchmarks. The main circularity is confined to RQ2/Table 3: for NewsQA, TQA, and NQ, the ground-truth evidence labels are constructed from answer containment, so the reported evidence-accuracy numbers mostly measure answer-string localization rather than evidence relevance. HotpotQA's human supporting_facts annotations and the independent QA improvements provide real content, so the paper is only partially circular. A separate, non-circularity concern is that LER = last 50% layers and alpha = 0.5 were selected after inspecting the same benchmarks (Table 4, Figure 4) and then described as fixed defaults; that is benchmark selection rather than construction-level circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. Its burden is two hyperparameters (alpha and LER) selected on evaluation data, and several assumptions about attention semantics and evidence labeling. The answer-containment labeling assumption is the most likely to inflate reported evidence scores, while the attention-reliability assumption is the most likely to limit generalization to new tasks.

free parameters (2)
  • Evidence threshold alpha = 0.5
    Chosen as default based on RQ5 experiments on the evaluation datasets, balancing evidence precision and coverage. The paper shows robustness for alpha in [0.5, 1] but the selection uses the same data on which final results are reported.
  • Evidence-reading layers LER = last 50% of layers
    Selected based on RQ4 analysis on HotpotQA, where the last-half layers gave the best AUROC and QA performance. Applied universally across datasets and models.
assumptions (4)
  • domain assumption Average attention over all heads in a layer is a valid measure of token importance.
    Used in Equations (1)-(3) to convert token-level attention into sentence-level evidence scores. No head-specific analysis is performed.
  • domain assumption Deeper-layer attention reliably identifies evidence sentences even when the model answers incorrectly.
    This is the central empirical observation in Figure 2 and Appendix B.2. It is shown for several models but not proven or tested across all possible contexts.
  • domain assumption Highlighting sentences with text markers preserves semantic content and guides the language model without harmful side effects.
    Supported indirectly by the FULL ELICIT baseline (which does not help) and by the attention shift measured in Table 7, but no direct causal test is provided.
  • domain assumption For datasets without supporting-fact annotations, a sentence is evidence if it contains a correct answer.
    Used to compute AUROC and NDCG in Table 3 for NewsQA, TQA, and NQ. This is a coarse proxy and may overstate evidence-identification quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence." pith.science (2026). https://pith.science/paper/2NV53PTJ

@misc{pith2026250208767,
  author       = {Pith},
  title        = {Pith review of: SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2NV53PTJ}},
  note         = {Machine review of arXiv:2502.08767}
}
read the original abstract

Providing Language Models (LMs) with relevant evidence in the context (either via retrieval or user-provided) can significantly improve their ability to provide better-grounded responses. However, recent studies have found that LMs often struggle to fully comprehend and utilize key evidence from the context, especially when it contains noise and irrelevant information, an issue common in real-world scenarios. To address this, we propose SelfElicit, an inference-time approach that helps LMs focus on key contextual evidence through self-guided explicit highlighting. By leveraging the inherent evidence-finding capabilities of LMs using the attention scores of deeper layers, our method automatically identifies and emphasizes key evidence within the input context, facilitating more accurate and grounded responses without additional training or iterative prompting. We demonstrate that SelfElicit brings consistent and significant improvement on multiple evidence-based QA tasks for various LM families while maintaining computational efficiency. Our code and documentation are available at https://github.com/ZhiningLiu1998/SelfElicit.

Figures

Figures reproduced from arXiv: 2502.08767 by the authors.

Figure 1
Figure 1. SELFELICIT workflow on a real example with Llama3.1-8B. By locating and explicitly highlighting the initially overlooked 2nd-hop evidence (“SAS was founded in 1941 ...") within the context, SELFELICIT guides the model to arrive at the correct answer “1941”. et al., 2024; Cuconasu et al., 2024), which is usually inevitable in practice (Gao et al., 2023). Recently, improved prompting (Zhou et al., 2023) or decoding (S… view at source ↗
Figure 2
Figure 2. Relative attention3 to the evidence/non-evidence sections (y-axis) across the layers (x-axis) for different LM families on HotpotQA. Deeper layers pay much greater attention to crucial evidence (green lines) in the context, even when LM responds incorrectly (dashed lines). Best viewed in color. how a¯ (ℓ) i varies across layers for evidence vs non￾evidence sentences in the context when generating the first response … view at source ↗
Figure 3
Figure 3. SELFELICIT demonstrates robust advantage even in the presence of noisy context ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Impact of elicit threshold α (x-axis) on the QA performance gain (blue bars, left y-axis) and evidence elicit ratio (orange lines, right y-axis) of SELFELICIT on four QA tasks. Best viewed in color. α indicating clearly that evidence elicitation upto a threshold helps …
Figure 5
Figure 5. Figure 5: Across different LM families and datasets, the deep attention layers highlight the crucial evidence [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling

    cs.CV 2025-09 unverdicted novelty 7.0 of 10

    HiDe uses token-wise attention decoupling and layout-preserving decoupling to build compact crops that push Qwen2.5-VL and InternVL3 to state-of-the-art scores on high-resolution VQA benchmarks.

  2. On Improving Faithfulness of Podcasts from Documents

    cs.CL 2026-07 conditional novelty 6.0 of 10

    AI-generated podcasts often add unsupported claims; a turn-level detector plus rewrite pass improves measured faithfulness across five models and in- and out-of-domain documents.

  3. AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism

    cs.CL 2025-06 conditional novelty 6.0 of 10

    AdaDecode speeds up LLM generation by predicting tokens at early layers when confidence is high, running the skipped layers in parallel, and verifying the output exactly matches standard decoding.

Reference graph

Works this paper leans on

96 extracted references · 38 canonical work pages · cited by 3 Pith papers

  1. [1]

    Mengting Ai, Tianxin Wei, Yifan Chen, Zeming Guo, and Jingrui He. 2023. Mlp fusion: Towards efficient fine-tuning of dense and mixture-of-experts language models. arXiv e-prints, pages arXiv--2307

    2023

  2. [2]

    Mengting Ai, Tianxin Wei, Yifan Chen, Zhichen Zeng, Ritchie Zhao, Girish Varatkar, Bita Darvish Rouhani, Xianfeng Tang, Hanghang Tong, and Jingrui He. 2025. Resmoe: Space-efficient compression of mixture of experts llms via residual restoration. arXiv preprint arXiv:2503.06881

    work page Pith review arXiv 2025

  3. [3]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2310.11511 Self-rag: Learning to retrieve, generate, and critique through self-reflection . arXiv preprint arXiv:2310.11511

    arXiv 2023

  4. [4]

    Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. 2021. Ee-net: Exploitation-exploration neural networks in contextual bandits. arXiv preprint arXiv:2110.03177

    arXiv 2021

  5. [5]

    Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. 2023. Adaptive test-time personalization for federated learning. Advances in Neural Information Processing Systems, 36:77882--77914

    2023

  6. [6]

    Eunice Chan, Zhining Liu, Ruizhong Qiu, Yuheng Zhang, Ross Maciejewski, and Hanghang Tong. 2024. Group fairness via group consensus. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1788--1808

    2024

  7. [7]

    Lingjie Chen, Ruizhong Qiu, Siyu Yuan, Zhining Liu, Tianxin Wei, Hyunsik Yoo, Zhichen Zeng, Deqing Yang, and Hanghang Tong. 2024 a . Wapiti: A watermark for finetuned open-source llms. arXiv preprint arXiv:2410.06467

    arXiv 2024

  8. [8]

    Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. 2024 b . In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In Forty-first International Conference on Machine Learning

    2024

Show all 96 references
  1. [9]

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2024. https://arxiv.org/pdf/2309.03883.pdf Dola: Decoding by contrasting layers improves factuality in large language models . In International Conference on Learning Representations (ICLR)

    2024 arXiv

  2. [10]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference o...

    2024

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,...

    2024

  5. [13]

    Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. In EMNLP 2019 MRQA Workshop, page 1

    2019

  6. [14]

    Torvik, and Jingrui He

    Dongqi Fu, Liri Fang, Zihao Li, Hanghang Tong, Vetle I. Torvik, and Jingrui He. 2024. https://doi.org/10.48550/ARXIV.2410.12126 Parametric graph representations in the era of foundation models: A survey and position . CoRR, abs/2410.12126

    2024 doi arXiv

  7. [15]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  8. [16]

    Kai Guo, Xiaofeng Cao, Zhining Liu, and Yi Chang. 2023. Taming over-smoothing representation on heterophilic graphs. Information Sciences, 647:119463

    2023

  9. [17]

    Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. 2024. Overthinking the truth: Understanding how language models process false demonstrations. In The Twelfth International Conference on Learning Representations

    2024

  10. [18]

    Xinrui He, Tianxin Wei, and Jingrui He. 2023. Robust basket recommendation via noise-tolerated graph contrastive learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 709--719

    2023

  11. [19]

    Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear, 7(1):411--420

    2017

  12. [20]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. https://arxiv.org/abs/2311.05232 A survey on hallucination in large language models: Principles, taxonomy, challenges, and open qu...

    2023 arXiv

  13. [21]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://arxiv.org/abs/2202.03629 Survey of hallucination in natural language generation . In ACM Computing Surveys

    2023 arXiv

  14. [22]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  15. [23]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023 b . https://arxiv.org/abs/2305.06983 Active retrieval augmented generation . In Empirical Methods in Natural Language Processing (EMNLP)

    2023 arXiv

  16. [24]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. https://arxiv.org/abs/1705.03551 Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension . In Association for Computational Linguistics (ACL)

    2017 arXiv

  17. [25]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. https://arxiv.org/abs/2307.10169 Challenges and applications of large language models . In arXiv preprint arXiv:2307.10169

    2023 arXiv

  18. [26]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  19. [27]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 a . Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  20. [28]

    Yunzhe Li, Junting Wang, Hari Sundaram, and Zhining Liu. 2025. A zero-shot generalization framework for llm-driven cross-domain sequential recommendation. arXiv preprint arXiv:2501.19232

    2025 arXiv

  21. [29]

    Zihao Li, Lecheng Zheng, Bowen Jin, Dongqi Fu, Baoyu Jing, Yikun Ban, Jingrui He, and Jiawei Han. 2024 b . https://doi.org/10.48550/ARXIV.2412.08174 Can graph neural networks learn language with extremely weak text supervision? CoRR, abs/2412.08174

    2024 doi arXiv

  22. [30]

    Xiao Lin, Jian Kang, Weilin Cong, and Hanghang Tong. 2024 a . Bemap: Balanced message passing for fair graph neural network. In Learning on Graphs Conference, pages 37--1. PMLR

    2024

  23. [31]

    Xiao Lin, Zhining Liu, Dongqi Fu, Ruizhong Qiu, and Hanghang Tong. 2024 b . Backtime: Backdoor attacks on multivariate time series forecasting. arXiv preprint arXiv:2410.02195

    2024 arXiv

  24. [32]

    Lihui Liu, Boxin Du, Yi Ren Fung, Heng Ji, Jiejun Xu, and Hanghang Tong. 2021 a . Kompare: A knowledge graph comparative reasoning system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '21, page 3308–3318, New York, NY, USA. Assoc...

    2021

  25. [33]

    Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong. 2022. Joint knowledge graph completion and question answering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 1098–1108, New York, NY, USA. Association for ...

    2022

  26. [34]

    Lihui Liu, Blaine Hill, Boxin Du, Fei Wang, and Hanghang Tong. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.48 Conversational question answering with language models generated reformulations over knowledge graph . In Findings of the Association for Computational Ling...

    2024 doi

  27. [35]

    Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tie-Yan Liu. 2020 a . Self-paced ensemble for highly imbalanced massive data classification. In 2020 IEEE 36th international conference on data engineering (ICDE), pages 841--852. IEEE

    2020

  28. [36]

    Zhining Liu, Jian Kang, Hanghang Tong, and Yi Chang. 2021 b . Imbens: Ensemble class-imbalanced learning in python. arXiv preprint arXiv:2111.12776

    2021 arXiv

  29. [37]

    Zhining Liu, Ruizhong Qiu, Zhichen Zeng, Hyunsik Yoo, David Zhou, Zhe Xu, Yada Zhu, Kommy Weldemariam, Jingrui He, and Hanghang Tong. 2024 b . Class-imbalanced graph learning without class rebalancing. In Forty-first International Conference on Machine Learning

    2024

  30. [38]

    Zhining Liu, Ruizhong Qiu, Zhichen Zeng, Yada Zhu, Hendrik Hamann, and Hanghang Tong. 2024 c . Aim: Attributing, interpreting, mitigating data unfairness. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2014--2025

    2024

  31. [39]

    Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. 2020 b . Mesa: boost ensemble imbalanced learning with meta-sampler. Advances in neural information processing systems, 33:14463--14474

    2020

  32. [40]

    OpenAI. 2022. Introducing chatgpt. URL https://openai.com/blog/chatgpt

    2022

  33. [41]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT-4 technical report . arXiv preprint arXiv:2303.08774

    2023 arXiv

  34. [42]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  35. [43]

    Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427

    2023 arXiv

  36. [44]

    Ruizhong Qiu, Jun-Gi Jang, Xiao Lin, Lihui Liu, and Hanghang Tong. 2024 a . TUCKET : A tensor time series data structure for efficient and accurate factor analysis over time ranges. Proceedings of the VLDB Endowment, 17(13)

    2024

  37. [45]

    Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. 2022. DIMES : A differentiable meta solver for combinatorial optimization problems. In Advances in Neural Information Processing Systems, volume 35, pages 25531--25546

    2022

  38. [46]

    Ruizhong Qiu and Hanghang Tong. 2024. Gradient compressed sensing: A query-efficient gradient estimator for high-dimensional zeroth-order optimization. In Proceedings of the 41st International Conference on Machine Learning

    2024

  39. [47]

    Ruizhong Qiu, Dingsu Wang, Lei Ying, H Vincent Poor, Yifang Zhang, and Hanghang Tong. 2023. Reconstructing graph diffusion history from a single snapshot. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1978--1988

    2023

  40. [48]

    Ruizhong Qiu, Zhe Xu, Wenxuan Bao, and Hanghang Tong. 2024 b . Ask, and it shall be given: On the Turing completeness of prompting. arXiv, 2411.01992

    2024 arXiv

  41. [49]

    Ruizhong Qiu, Weiliang Will Zeng, Hanghang Tong, James Ezick, and Christopher Lott. 2024 c . How efficient is LLM -generated code? A rigorous & high-standard benchmark. arXiv, 2406.06647

    2024 arXiv

  42. [50]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. https://arxiv.org/abs/2302.00083 In-context retrieval-augmented language models . In Association for Computational Linguistics (ACL)

    2023 arXiv

  43. [51]

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingu...

    2024

  44. [52]

    Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. ACL 2017, page 191

    2017

  45. [53]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LLion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://arxiv.org/abs/1706.03762 Attention is all you need . In Advances in Neural Information Processing Systems (NeurIPS)

    2017 arXiv

  46. [54]

    Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. 2023 a . https://arxiv.org/abs/2310.07521 Survey on factuality in large language models: Knowledge, retrieval and domain-specificity . In a...

    2023 arXiv

  47. [55]

    Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023 b . Learning to filter context for retrieval-augmented generation. arXiv preprint arXiv:2311.08377

    2023 arXiv

  48. [56]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 a . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  49. [57]

    Tianxin Wei, Yifan Chen, Xinrui He, and Jingrui He. 2025 a . https://dl.acm.org/doi/10.1145/3690624.3709280 Connecting domains and contrasting samples: A ladder for domain generalization

    2025

  50. [58]

    Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. 2021. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 1791--1800

    2021

  51. [59]

    Tianxin Wei, Zeming Guo, Yifan Chen, and Jingrui He. 2023. Ntk-approximating mlp fusion for efficient language model fine-tuning. In International Conference on Machine Learning, pages 36821--36838. PMLR

    2023

  52. [60]

    Tianxin Wei and Jingrui He. 2022. Comprehensive fair meta-learned recommender system. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1989--1999

    2022

  53. [61]

    Tianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng, Zhengyang Wang, Jianhui Sun, Qingyu Yin, Hanqing Lu, Suhang Wang, Jingrui He, et al. 2024 a . Towards unified multi-modal personalization: Large vision-language models for generative recommendation and beyond. arXiv preprint arXiv...

    2024 arXiv

  54. [62]

    Tianxin Wei, Ruizhong Qiu, Yifan Chen, Yunzhe Qi, Jiacheng Lin, Wenju Xu, Sreyashi Nag, Ruirui Li, Hanqing Lu, Zhengyang Wang, Chen Luo, Hui Liu, Suhang Wang, Jingrui He, Qi He, and Xianfeng Tang. 2024 b . https://openreview.net/pdf?id=O13fIFEB81 Robust watermarking for diffus...

    2024

  55. [63]

    Tianxin Wei, Ziwei Wu, Ruirui Li, Ziniu Hu, Fuli Feng, Xiangnan He, Yizhou Sun, and Wei Wang. 2020. Fast adaptation for cold-start collaborative filtering with meta-learning. In 2020 IEEE International Conference on Data Mining (ICDM), pages 661--670. IEEE

    2020

  56. [64]

    Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. 2022 b . Augmentations in hypergraph contrastive learning: Fabricated and generative. Advances in neural information processing systems, 35:1909--1922

    2022

  57. [65]

    Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025 b . https://openreview.net/forum?id=P1qhkp8gQT Instruct RAG : Instructing retrieval-augmented generation via self-synthesized rationales . In The Thirteenth International Conference on Learning Representations

    2025

  58. [66]

    Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. 2025 c . https://arxiv.org/abs/2505.16421 Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning . Preprint, a...

    2025

  59. [67]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  60. [68]

    Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302

    2024 arXiv

  61. [69]

    Slog: An inductive spectral graph neural network beyond polynomial filter

    Haobo Xu, Yuchen Yan, Dingsu Wang, Zhe Xu, Zhichen Zeng, Tarek F Abdelzaher, Jiawei Han, and Hanghang Tong. Slog: An inductive spectral graph neural network beyond polynomial filter. In Forty-first International Conference on Machine Learning

  62. [70]

    Zhe Xu, Ruizhong Qiu, Yuzhong Chen, Huiyuan Chen, Xiran Fan, Menghai Pan, Zhichen Zeng, Mahashweta Das, and Hanghang Tong. 2024. Discrete-state continuous-time diffusion for graph generation. In Advances in Neural Information Processing Systems, volume 37

    2024

  63. [71]

    Yuchen Yan, Yuzhong Chen, Huiyuan Chen, Xiaoting Li, Zhe Xu, Zhichen Zeng, Lihui Liu, Zhining Liu, and Hanghang Tong. 2024 a . Thegcn: Temporal heterophilic graph convolutional network. arXiv preprint arXiv:2412.16435

    2024 arXiv

  64. [72]

    Yuchen Yan, Yuzhong Chen, Huiyuan Chen, Minghua Xu, Mahashweta Das, Hao Yang, and Hanghang Tong. 2023 a . From trainable negative depth to edge heterophily in graphs. Advances in Neural Information Processing Systems, 36:70162--70178

    2023

  65. [73]

    Yuchen Yan, Yongyi Hu, Qinghai Zhou, Lihui Liu, Zhichen Zeng, Yuzhong Chen, Menghai Pan, Huiyuan Chen, Mahashweta Das, and Hanghang Tong. 2024 b . Pacer: Network embedding from positional to structural. In Proceedings of the ACM on Web Conference 2024, pages 2485--2496

    2024

  66. [74]

    Yuchen Yan, Yongyi Hu, Qinghai Zhou, Shurang Wu, Dingsu Wang, and Hanghang Tong. 2024 c . Topological anonymous walk embedding: A new structural node embedding approach. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 2796--2806

    2024

  67. [75]

    Yuchen Yan, Baoyu Jing, Lihui Liu, Ruijie Wang, Jinning Li, Tarek Abdelzaher, and Hanghang Tong. 2023 b . Reconciling competing sampling strategies of network embedding. Advances in Neural Information Processing Systems, 36:6844--6861

    2023

  68. [76]

    Yuchen Yan, Lihui Liu, Yikun Ban, Baoyu Jing, and Hanghang Tong. 2021. Dynamic knowledge graph alignment. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4564--4572

    2021

  69. [77]

    Yuchen Yan, Qinghai Zhou, Jinning Li, Tarek Abdelzaher, and Hanghang Tong. 2022. Dissecting cross-layer dependency inference on multi-layered inter-dependent networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 2341--2351

    2022

  70. [78]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  71. [79]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. https://arxiv.org/abs/1809.09600 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . In Empirical Methods in Natural Language Pr...

    2018 arXiv

  72. [80]

    Hangting Ye, Zhining Liu, Wei Cao, Amir M Amiri, Jiang Bian, Yi Chang, Jon D Lurie, Jim Weinstein, and Tie-Yan Liu. 2023. Web-based long-term spine treatment outcome forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3082--3092

    2023

  73. [81]

    Hyunsik Yoo, SeongKu Kang, Ruizhong Qiu, Charlie Xu, Fei Wang, and Hanghang Tong. 2025 a . Embracing plasticity: Balancing stability and plasticity in continual recommender systems. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in In...

    2025

  74. [82]

    Hyunsik Yoo, Ruizhong Qiu, Charlie Xu, Fei Wang, and Hanghang Tong. 2025 b . Generalizable recommender system during temporal popularity distribution shifts. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    2025

  75. [83]

    Hyunsik Yoo, Zhichen Zeng, Jian Kang, Ruizhong Qiu, David Zhou, Zhining Liu, Fei Wang, Charlie Xu, Eunice Chan, and Hanghang Tong. 2024. Ensuring user-side fairness in dynamic recommender systems. In Proceedings of the ACM on Web Conference 2024, pages 3667--3678

    2024

  76. [84]

    Erxin Yu, Jing Li, and Chunpu Xu. 2024. P op ALM : Popularity-aligned language models for social media trendy response prediction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), page...

    2024

  77. [85]

    Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. 2024. https://arxiv.org/abs/2309.15098 Attention satisfies: A constraint-satisfaction lens on factual errors of language models . In International C...

    2024 arXiv

  78. [86]

    Zhichen Zeng, Boxin Du, Si Zhang, Yinglong Xia, Zhining Liu, and Hanghang Tong. 2024 a . Hierarchical multi-marginal optimal transport for network alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16660--16668

    2024

  79. [87]

    Zhichen Zeng, Ruizhong Qiu, Zhe Xu, Zhining Liu, Yuchen Yan, Tianxin Wei, Lei Ying, Jingrui He, and Hanghang Tong. 2024 b . Graph mixup on approximate gromov--wasserstein geodesics. In Forty-first International Conference on Machine Learning

    2024

  80. [88]

    Zhichen Zeng, Si Zhang, Yinglong Xia, and Hanghang Tong. 2023 a . Parrot: Position-aware regularized optimal transport for network alignment. In Proceedings of the ACM Web Conference 2023, pages 372--382

    2023

  81. [89]

    Zhichen Zeng, Ruike Zhu, Yinglong Xia, Hanqing Zeng, and Hanghang Tong. 2023 b . Generative graph dictionary learning. In International Conference on Machine Learning, pages 40749--40769. PMLR

    2023

  82. [90]

    Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, and Dong Yu. 2025. Improving llm general preference alignment via optimistic online mirror descent. arXiv preprint arXiv:2502.16852

    2025 arXiv

  83. [91]

    Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024. Enhancing contextual understanding in large language models through contrastive decoding. arXiv preprint arXiv:2405.02750

    2024 arXiv

  84. [92]

    Lecheng Zheng, Baoyu Jing, Zihao Li, Hanghang Tong, and Jingrui He. 2024 a . https://doi.org/10.1145/3637528.3671454 Heterogeneous contrastive learning for foundation models and beyond . In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, K...

    2024

  85. [93]

    Lecheng Zheng, Baoyu Jing, Zihao Li, Zhichen Zeng, Tianxin Wei, Mengting Ai, Xinrui He, Lihui Liu, Dongqi Fu, Jiaxuan You, Hanghang Tong, and Jingrui He. 2024 b . https://doi.org/10.48550/ARXIV.2412.21151 Pyg-ssl: A graph self-supervised learning toolkit . CoRR, abs/2412.21151

    2024 doi arXiv

  86. [94]

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544--14556

    2023

  87. [95]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  88. [96]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.