Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education?

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Changing one textbook fact drops retrieval-augmented question answering by 22–27 accuracy points, with the largest failures on questions that blend context and parametric knowledge.

desk verdict A useful, well-built benchmark with a solid core finding, but the headline 22-27 point drop is confounded by rewrite artifacts and needs a control. read the letter →

arxiv 2412.08985 v4 pith:YGGIYCOV submitted 2024-12-12 cs.CL

Tianshi Zheng , Weihan Li , Jiaxin Bai , Weiqi Wang , Yangqiu Song This is my paper · ORCID
classification cs.CL
keywords knowledgeshiftsretrieval-augmentedgenerationK-12educationalQAcontextutilizationintegrationhypotheticalupdatequestionansweringbenchmarkLLMrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces KnowShiftQA, a multiple-choice benchmark of 3,005 questions built from middle-school textbooks in physics, chemistry, biology, geography, and history. Its central move is a "hypothetical knowledge update": every question's source paragraph is rewritten so that one authoritative fact is replaced by a plausible alternative, creating a paired before/after test of the same reasoning task. The paper claims that retrieval-augmented generation (RAG) systems lose 22–27 accuracy points on these updated questions, and that the dominant failure is not retrieval or multi-hop reasoning but "knowledge integration"—combining a fact found in the retrieved context with the model's own parametric knowledge, which is exactly what implicit question types demand. A sympathetic reader would care because a deployed educational QA system must answer from the current textbook even when that textbook disagrees with what the LLM learned during training.

What carries the argument

The load-bearing mechanism is the paired before/after document produced by the hypothetical knowledge update: a human-curated rewrite of an open-source textbook paragraph in which every occurrence of the original answer is replaced with a plausible but incorrect alternative and surrounding statements are adjusted so the paragraph remains coherent and consistent. This pairing gives the same question a version in which the LLM's parametric knowledge agrees with the context and a version in which the two conflict, so the only systematic difference is the knowledge shift. The second piece is the question typology, which sorts questions into five types—Simple Direct, Multi-hop Direct, Multi-hop Distant, Multi-hop Implicit, and Distant Implicit—that separate basic recall, multi-hop reasoning, distant context utilization, parametric-context integration, and their combination; the last two are the ones that isolate the paper's central failure mode.

What would settle it

Run the same 3,005 questions through the same rewriting pipeline but with the original answer restored (or with a random fact swapped in), keeping all other edits; if accuracy still drops substantially, the reported loss is not specific to knowledge shifting. Alternatively, have human raters try to distinguish updated from original paragraphs—if they can reliably tell them apart, the naturalness control fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that RAG systems are fragile precisely where they are supposed to add value: when the authoritative source conflicts with the model's internal knowledge. Across two LLMs (Llama3-8b and GPT-4o) and two retrievers (Ada-002 and a BM25+Ada-002 hybrid rerank), accuracy drops from 87.49 to 62.60, from 88.49 to 66.02, from 96.57 to 69.65, and from 97.10 to 73.71 after the hypothetical knowledge update—a 22–27 point decline. The paper attributes this decline to the two "implicit" question types, where the updated fact sits inside a multi-hop chain and must be fused with a fact the model already knows; even the strongest tested model, o1-preview, reaches only 78.86% on Distant Implicit questions, while a small model like Mistral-7b falls to 33.98%.

Load-bearing premise

Everything in the 22–27 point drop depends on updated textbook passages being exactly as natural and coherent as the originals apart from the swapped fact.

Editorial extensions

If this is right

  • RAG deployments over authoritative but evolving sources should budget for a roughly 20-point accuracy loss when the source contradicts the model's training knowledge, not a small edge-case effect.
  • Retrieval quality is not the binding constraint in this setting; BM25 and the hybrid rerank already retrieve the right paragraph most of the time, so the loss concentrates in generation.
  • Implicit question types are the diagnostic to use in future robustness evaluations; direct and multi-hop direct scores overstate how well a system will handle knowledge shifts.
  • Fine-tuning dense retrievers on the target corpus and using a locate-and-answer prompt each help, but neither closes the gap, indicating the bottleneck is the model's ability to override its parametric prior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A matched control—rewriting paragraphs the same way but leaving the answer unchanged—would isolate how much of the 22–27 point drop is caused by knowledge shift versus by the rewrite itself.
  • The benchmark could be extended to real temporal shifts such as populations, capitals, or currency data, where the updated fact is true in the world; the paper flags this as future work and notes the sparsity and noise of such data.
  • Because the paper finds that even strong models second-guess the provided document on simple direct questions, an intervention that calibrates the model to treat the retrieved document as authoritative—such as explicit document-grounded verification—would be a direct test of the claimed integration bottleneck.
  • The dataset's question typology could transfer to other high-stakes domains where a source of record is authoritative, such as medical guidelines or legal texts, to test whether the same integration failure appears outside education.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces KnowShiftQA, a dataset of 3,005 multiple-choice questions across Physics, Chemistry, Biology, Geography, and History, built by applying hypothetical knowledge updates to textbook paragraphs and answers. The authors design a five-type question typology (Simple Direct, Multi-hop Direct, Multi-hop Distant, Multi-hop Implicit, Distant Implicit) to separate context-utilization from knowledge-integration failures. They evaluate 10 retrieval configurations and 16 LLMs under a zero-shot Locate-and-Answer prompting protocol, and report a 22–27 accuracy-point drop for four RAG systems (Llama3-8b and GPT-4o, each with Ada-002 and with hybrid reranking) after the hypothetical updates. The paper concludes that most RAG systems degrade substantially under knowledge shifts and that the dominant difficulty is integrating contextual textbook facts with parametric LLM knowledge on implicit question types.

Significance. If the central result holds, KnowShiftQA is a useful and carefully constructed benchmark for a real deployment concern: RAG systems that answer K-12 questions from authoritative textbooks will encounter discrepancies between textbook content and an LLM's parametric knowledge. The dataset is open-source, the curation pipeline is human-verified at multiple stages, the question typology is intuitive, and the evaluation covers a broad set of modern LLMs and retrieval methods. The paper also states its limitations explicitly, which is a strength. However, the quantitative headline (the 22–27 point drop) rests on only four system configurations, and the before/after comparison may not isolate knowledge discrepancy from rewriting artifacts; these issues must be addressed before the central claim is fully supported.

major comments (4)
  1. [Section 3.3, Table 4] The central claim that 'most RAG systems suffer a substantial performance drop' is supported by only four configurations: two LLMs (Llama3-8b and GPT-4o) combined with two retrievers (Ada-002 and Hybrid Rerank). No variance, confidence intervals, or significance tests are reported for these 22–27 point gaps. Given the strong wording of the abstract and conclusion, either expand the evaluation to more diverse RAG configurations or temper the claim to 'the tested RAG systems.' Reporting per-question resampling or bootstrap intervals would also make the magnitude of the drop more interpretable.
  2. [Section 2.1 and Appendix D, Table 5] The before/after comparison is intended to isolate the effect of a knowledge discrepancy, but the update procedure changes more than the target fact. In the halophile example in Table 5, the word 'halophiles' is retained but redefined from 'salt-loving' to 'pressure-loving,' and its habitat is moved from the Great Salt Lake and Dead Sea to the Mariana Trench. This introduces an internal lexical contradiction (the etymology no longer matches the meaning) and involves multi-sentence rewriting. The paper's only safeguard is 'extensive human curation and verification' (Section 2.1), with no quantitative naturalness or equivalence check reported. Consequently, the 22–27 point drop in Table 4 may be inflated by reduced fluency, coherence, or consistency of the updated passages rather than by knowledge discrepancy per se. To support the causal attribution, add a control condition in which paragraphs are rewritten with comparable editing effort but without changing the target fact, or a matched-fluency condition, and show that the accuracy gap persists.
  3. [Table 2 and Appendix C] The fine-tuned retrieval models (Contriever fine-tuned and Con.-msmarco fine-tuned) are described as fine-tuned 'on the KNOWSHIFTQA dataset,' but no train/test split is stated. If fine-tuning and evaluation use the same corpus, the reported R@1 values of 84.19 and 87.95 reflect in-domain learning and are not directly comparable to zero-shot baselines such as BM25 or Ada-002. Specify the number of documents and questions used for training versus evaluation, and, if feasible, evaluate on held-out documents or questions to ensure the comparison is fair.
  4. [Section 3.1, Table 2] The Hybrid Rerank result is reported with k=6 as 'the optimal hyperparameter setting,' but no sensitivity analysis or variance estimate is provided. For a benchmark that aims to measure robustness, showing how retrieval quality varies with k (e.g., k = 3, 6, 10) and reporting standard errors across query subsets would strengthen the methodological contribution and make the improvement over BM25 and Ada-002 more credible.
minor comments (5)
  1. [Section 3.2, Table 3] The LLM evaluation does not report decoding parameters such as temperature, top-p, or number of samples. Adding these details to Appendix C would improve reproducibility.
  2. [Table 1] In the 'Simple Direct' example, the layout 'NV goggles - detect - Infrared light / Ultraviolet' is visually ambiguous; clarify which row is the question, which is the original fact, and which is the updated fact (e.g., by using separate columns with explicit labels).
  3. [Appendix B] The curation pipeline reports that '90.5% of these queries were retained or underwent minor refinements' and an 'overall success rate of 86.4%,' but the relationship between these two figures is not explained. Clarify whether they refer to different pipeline stages and how they were computed.
  4. [Section 2.2, Table 1] The definition of 'Multi-hop Implicit' says the updated facts are 'embedded within the middle of the multi-hop reasoning chain,' but in the example 'Who discovered the radioactive element that is commonly found in uranium ores?' the updated fact (uranium ores containing thorium rather than polonium) functions more as a distal premise than as a middle hop. Suggest rewording the definition to cover indirect reference to the updated fact at any position in the reasoning chain.
  5. [References] The reference list contains two entries for 'OpenAI. 2022' (ChatGPT and the embedding model); these should be disambiguated with distinct year suffixes or different labels to avoid citation ambiguity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the before/after performance drop is an empirical measurement, and the question typology is defined in-paper without fitted parameters or imported uniqueness claims.

full rationale

The paper's central claim is empirical: RAG accuracy drops 22-27 points when evaluated on paragraphs modified by hypothetical knowledge updates (Table 4). This drop is measured, not derived from the construction. No fitted parameter is renamed as a prediction, and no equation reduces the outcome to its inputs. The question typology is defined in Section 2.2 and Table 1 with explicit reasoning patterns and examples; the footnote that the typology is 'motivated by studies in complex logical reasoning' cites prior work by the same authors, but that citation is not load-bearing because the definitions and examples are self-contained in the paper. The concern that the updated paragraphs may contain rewriting artifacts (e.g., the Appendix D halophile example) is a validity threat about experimental control, not circularity: it questions whether the measured drop is attributable to knowledge shift, but it does not show that the result is equivalent to the input by construction. The Limitations section openly acknowledges the hypothetical-update simulation, and the Ethics statement explicitly says the dataset is not suitable for assessing factual accuracy, further indicating the authors do not hide the constructed nature of the scenario. Overall, the derivation chain is self-contained and the findings are externally falsifiable, so circularity is minimal.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central findings rest mainly on data-construction and evaluation assumptions rather than on fitted theoretical parameters. The free parameters are benchmark hyperparameters; the axioms concern the coherence of the hypothetical updates, the validity of the typology, and the evaluation protocol.

free parameters (3)
  • Hybrid Rerank top-k = 6
    Section 3.1 states k=6 as the optimal hyperparameter for the Ada-002 + BM25 ensemble; retrieval and RAG results depend on this.
  • Minimum document length for Multi-hop Distant = 200 words
    Appendix B assigns Distant questions only to documents with at least 200 words; the distribution and difficulty of question types depend on this threshold.
  • Decoding temperature and sampling parameters = not reported
    Appendix C lists models but no decoding configuration; QA accuracy across Tables 3, 4, and 9 depends on this unstated setting.
assumptions (4)
  • domain assumption Hypothetically updated textbook paragraphs are coherent and differ from originals only in the modified facts.
    Section 2.1 and Appendix D rely on human curation and verification to guarantee this; no quantitative naturalness check is provided.
  • domain assumption The five-type typology isolates context utilization from parametric-knowledge integration, and implicit questions require only head-knowledge.
    Section 2.2 invokes Sun et al. (2024) to restrict parametric knowledge to head entities; if models lack head knowledge, integration conclusions are confounded.
  • domain assumption Retrieval Recall@1/5 and the Locate-and-Answer prompt adequately represent the context available to the reader in RAG.
    Sections 3.1 and 3.2 evaluate retrieval and QA separately; the end-to-end link is assumed for the RAG drop measurement.
  • domain assumption Fine-tuned retrievers are evaluated without train/test leakage.
    Appendix C says Contriever models were fine-tuned on the dataset; the paper does not state the split, so top retrieval scores assume generalization rather than memorization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education?." pith.science (2026). https://pith.science/paper/YGGIYCOV

@misc{pith2026241208985,
  author       = {Pith},
  title        = {Pith review of: KnowShiftQA: How Robust are RAG Systems when Textbook Knowledge Shifts in K-12 Education?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGGIYCOV}},
  note         = {Machine review of arXiv:2412.08985}
}
read the original abstract

Retrieval-Augmented Generation (RAG) systems show remarkable potential as question answering tools in the K-12 Education domain, where knowledge is typically queried within the restricted scope of authoritative textbooks. However, discrepancies between these textbooks and the parametric knowledge inherent in Large Language Models (LLMs) can undermine the effectiveness of RAG systems. To systematically investigate RAG system robustness against such knowledge discrepancies, we introduce KnowShiftQA. This novel question answering dataset simulates these discrepancies by applying deliberate hypothetical knowledge updates to both answers and source documents, reflecting how textbook knowledge can shift. KnowShiftQA comprises 3,005 questions across five subjects, designed with a comprehensive question typology focusing on context utilization and knowledge integration. Our extensive experiments on retrieval and question answering performance reveal that most RAG systems suffer a substantial performance drop when faced with these knowledge discrepancies. Furthermore, questions requiring the integration of contextual (textbook) knowledge with parametric (LLM) knowledge pose a significant challenge to current LLMs.

Figures

Figures reproduced from arXiv: 2412.08985 by the authors.

Figure 1
Figure 1. An illustration of knowledge discrepancy in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Distribution of five subjects and their corresponding topics included in the K [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. An overview of the data curation pipeline of the K [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing Transformers for Generalizable First-Order Logical Entailment

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Transformers with relative positional encoding beat KGQA baselines, and adding logic-aware attention (TEGA) improves out-of-distribution performance on a new 55-type benchmark.

Reference graph

Works this paper leans on

68 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Asad Abdi, Norisma Idris, and Zahrah Ahmad. 2018. https://doi.org/10.1007/s00500-016-2328-2 Qapd: an ontology-based question answering system in the physics domain . Soft Computing, 22:213--230

    doi:10.1007/s00500-016-2328-2 2018

  4. [4]

    Agarwal, N

    A. Agarwal, N. Sachdeva, R. K. Yadav, V. Udandarao, V. Mittal, A. Gupta, and A. Mathur. 2019. https://doi.org/10.1109/ICASSP.2019.8683538 Eduqa: Educational domain question answering system using conceptual network mapping . In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8137--8141

    arXiv 2019

  5. [5]

    Anthropic. 2024 a . https://www.anthropic.com/news/claude-3-family Blog: Introducing the next generation of claude

    work page 2024

  6. [6]

    Anthropic. 2024 b . https://www.anthropic.com/news/claude-3-5-sonnet Introducing claude 3.5 sonnet

    work page 2024

  7. [7]

    Arbesman

    S. Arbesman. 2012. https://books.google.com.hk/books?id=jWKCtwAACAAJ The Half-life of Facts: Why Everything We Know Has an Expiration Date . Human body. Current

    work page 2012

  8. [8]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. http://arxiv.org/abs/2310.11511 Self-rag: Learning to retrieve, generate, and critique through self-reflection

    arXiv 2023

Show all 68 references
  1. [9]

    Jiaxin Bai, Wei Fan, Qi Hu, Qing Zong, Chunyang Li, Hong Ting Tsang, Hongyu Luo, Yauwai Yim, Haoyu Huang, Xiao Zhou, Feng Qin, Tianshi Zheng, Xi Peng, Xin Yao, Huiwen Yang, Leijie Wu, Yi Ji, Gong Zhang, Renhai Chen, and Yangqiu Song. 2025 a . http://arxiv.org/abs/2505.23628 Au...

    2025 arXiv

  2. [10]

    Jiaxin Bai, Yicheng Wang, Tianshi Zheng, Yue Guo, Xin Liu, and Yangqiu Song. 2024. http://arxiv.org/abs/2312.15643 Advancing abductive reasoning in knowledge graphs through complex logical hypothesis generation

    2024 arXiv

  3. [11]

    Jiaxin Bai, Zihao Wang, Yukun Zhou, Hang Yin, Weizhi Fei, Qi Hu, Zheye Deng, Jiayang Cheng, Tianshi Zheng, Hong Ting Tsang, Yisen Gao, Zhongwei Xie, Yufei Li, Lixin Fan, Binhang Yuan, Wei Wang, Lei Chen, Xiaofang Zhou, and Yangqiu Song. 2025 b . http://arxiv.org/abs/2501.14224...

    2025 arXiv

  4. [12]

    Jiaxin Bai, Tianshi Zheng, and Yangqiu Song. 2023. http://arxiv.org/abs/2302.13114 Sequential query encoding for complex query answering on knowledge graphs

    2023 arXiv

  5. [13]

    Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. http://arxiv.org/abs/2108.06314 A dataset for answering time-sensitive questions

    2021 arXiv

  6. [14]

    Zheye Deng, Chunkit Chan, Weiqi Wang, Yuxi Sun, Wei Fan, Tianshi Zheng, Yauwai Yim, and Yangqiu Song. 2024. http://arxiv.org/abs/2404.14215 Text-tuple-table: Towards information integration in text-to-table generation via global tuple extraction

    2024 arXiv

  7. [15]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. http://arxiv.org/abs/2404.16130 From local to global: A graph rag approach to query-focused summarization

    2024 arXiv

  8. [16]

    Andy Extance. 2023. https://doi.org/10.1038/d41586-023-03507-3 ChatGPT has entered the classroom: how LLMs could transform education . Nature, 623:474--477

    2023 doi

  9. [17]

    Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. http://arxiv.org/abs/2109.10086 Splade v2: Sparse lexical and expansion model for information retrieval

    2021 arXiv

  10. [18]

    Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. http://arxiv.org/abs/2311.13160 Large language models in education: Vision and opportunities

    2023 arXiv

  11. [19]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. http://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey

    2024 arXiv

  12. [20]

    Google. 2024 a . https://deepmind.google/technologies/gemini/flash/ Gemini flash - google deepmind

    2024

  13. [21]

    Google. 2024 b . https://deepmind.google/technologies/gemini/pro/ Gemini pro - google deepmind

    2024

  14. [22]

    Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. http://arxiv.org/abs/2405.14831 Hipporag: Neurobiologically inspired long-term memory for large language models

    2024 arXiv

  15. [23]

    Yikun Han, Chunjiang Liu, and Pengfei Wang. 2023. http://arxiv.org/abs/2310.11703 A comprehensive survey on vector database: Storage and retrieval technique, challenge

    2023

  16. [24]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. http://arxiv.org/abs/2112.09118 Unsupervised dense information retrieval with contrastive learning

    2022 arXiv

  17. [25]

    Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig

    Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. http://arxiv.org/abs/2305.06983 Active retrieval augmented generation

    2023 arXiv

  18. [26]

    Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sail...

    2023

  19. [27]

    Bor-Chen Kuo, Frederic T. Y. Chang, and Zong-En Bai. 2023. https://ceur-ws.org/Vol-3487/paper6.pdf Leveraging llms for adaptive testing and learning in taiwan adaptive learning platform (talp) . In Empowering Education with LLMs -- the Next-Gen Interface and Content Generation...

    2023

  20. [28]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  21. [29]

    Chunyang Li, Weiqi Wang, Tianshi Zheng, and Yangqiu Song. 2025. http://arxiv.org/abs/2502.16169 Patterns over principles: The fragility of inductive reasoning in llms under noisy observations

    2025 arXiv

  22. [30]

    Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. http://arxiv.org/abs/2211.05110 Large language models with controllable working memory . arXiv preprint arXiv:2211.05110

    2022 arXiv

  23. [31]

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. http://arxiv.org/abs/2305.14283 Query rewriting for retrieval-augmented large language models

    2023 arXiv

  24. [32]

    Meta. 2024. https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama 3: The most capable openly available llm to date

    2024

  25. [33]

    Mistral. 2023 a . https://docs.mistral.ai/capabilities/embeddings/ Embeddings | mistral ai large language models

    2023

  26. [34]

    Mistral. 2023 b . https://mistral.ai/news/announcing-mistral-7b/ Mistral 7b | frontier ai in your hands

    2023

  27. [35]

    Mistral. 2023 c . https://mistral.ai/news/mixtral-of-experts/ Mixtral of experts | frontier ai in your hands

    2023

  28. [36]

    Mistral. 2024 a . https://mistral.ai/news/september-24-release/ Ai in abundance | mistral ai | frontier ai in your hands

    2024

  29. [37]

    Mistral. 2024 b . https://mistral.ai/news/mistral-large-2407/ Large enough | mistral ai | frontier ai in your hands

    2024

  30. [38]

    Nye, Dillon Mee, and Mark G

    Benjamin D. Nye, Dillon Mee, and Mark G. Core. 2023. https://ceur-ws.org/Vol-3487/paper4.pdf Generative large language models for dialog-based tutoring: An early consideration of opportunities and concerns . In Empowering Education with LLMs – the Next-Gen Interface and Conten...

    2023

  31. [39]

    OpenAI . 2022. https://openai.com/chatgpt Chatgpt: Optimizing language models for dialogue

    2022

  32. [40]

    OpenAI. 2022. https://openai.com/index/new-and-improved-embedding-model/ New and improved embedding model

    2022

  33. [41]

    OpenAI. 2023 a . http://arxiv.org/abs/2303.08774 Gpt-4 technical report

    2023 arXiv

  34. [42]

    OpenAI. 2023 b . https://openai.com/index/new-models-and-developer-products-announced-at-devday New models and developer products announced at devday

    2023

  35. [43]

    OpenAI. 2024 a . https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  36. [44]

    OpenAI. 2024 b . https://openai.com/o1/ Introducing openai o1

    2024

  37. [45]

    Leigh Patel. 2015. https://doi.org/10.4324/9781315658551 Decolonizing Educational Research: From Ownership to Answerability , 1 edition. Routledge, New York

    2015 doi

  38. [46]

    Provenzo, A.N

    E.F. Provenzo, A.N. Shaver, and M. Bello. 2011. https://books.google.com.hk/books?id=fxasAgAAQBAJ The Textbook as Discourse: Sociocultural Dimensions of American Schoolbooks . Taylor & Francis

    2011

  39. [47]

    Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. https://openreview.net/forum?id=bNt7oajl2a Phenomenal yet puzzling: Testing inductive reasoning capabilities of langua...

    2024

  40. [48]

    Srikrishna Raamadhurai, Ryan Baker, and Vikraman Poduval. 2019. https://doi.org/10.18653/v1/W19-4435 Curio S mart C hat : A system for natural language question answering for self-paced k-12 learning . In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Buil...

    2019 doi

  41. [49]

    Stephen E Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. In Overview of the Third Text REtrieval Conference (TREC-3). NIST

    1994

  42. [50]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. http://arxiv.org/abs/2305.15294 Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy

    2023 arXiv

  43. [51]

    Teotino Gomes Soares, Azhari Azhari, Nur Rokhman, and E Wonarko. 2021. Education question answering systems: a survey. In Proceedings of The International MultiConference of Engineers and Computer Scientists

    2021

  44. [52]

    Karen Sp \"a rck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11--21

    1972

  45. [53]

    Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.18 Head-to-tail: How knowledgeable are large language models ( LLM s)? A . K . A . will LLM s replace knowledge graphs? In Proceedings of the 2024 Conference of the Nor...

    2024 doi

  46. [54]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. http://arxiv.org/abs/2104.08663 Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models

    2021 arXiv

  47. [55]

    Yu, and Qingsong Wen

    Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. 2024. http://arxiv.org/abs/2403.18105 Large language models for education: A survey and outlook

    2024 arXiv

  48. [56]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. http://arxiv.org/abs/2206.07682 Emergent...

    2022 arXiv

  49. [57]

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. http://arxiv.org/abs/2403.08319 Knowledge conflicts for llms: A survey

    2024 arXiv

  50. [58]

    Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Ga s evi \'c . 2024. Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educationa...

    2024

  51. [59]

    Yauwai Yim, Chunkit Chan, Tianyu Shi, Zheye Deng, Wei Fan, Tianshi Zheng, and Yangqiu Song. 2024. http://arxiv.org/abs/2408.02559 Evaluating and enhancing llms agent based on theory of mind in guandan: A multi-player cooperative game under imperfect information

    2024 arXiv

  52. [60]

    Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Nianyi Lin, Kaifeng Yun, Linlu Gong, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan, ...

    2024 arXiv

  53. [61]

    Michael J. Q. Zhang and Eunsol Choi. 2024. http://arxiv.org/abs/2305.14824 Mitigating temporal misalignment by discarding outdated facts

    2024 arXiv

  54. [62]

    Tianshi Zheng, Jiaxin Bai, Yicheng Wang, Tianqing Fang, Yue Guo, Yauwai Yim, and Yangqiu Song. 2024. http://arxiv.org/abs/2407.20564 Clr-fact: Evaluating the complex logical reasoning capability of large language models over factual knowledge

    2024 arXiv

  55. [63]

    Wong, and Simon See

    Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, and Simon See. 2025 a . http://arxiv.org/abs/2504.05081 The curse of cot: On the limitations of chain-of-thought in in-context learning

    2025

  56. [64]

    Wong, and Simon See

    Tianshi Zheng, Jiayang Cheng, Chunyang Li, Haochen Shi, Zihao Wang, Jiaxin Bai, Yangqiu Song, Ginny Y. Wong, and Simon See. 2025 b . http://arxiv.org/abs/2502.11176 Logidynamics: Unraveling the dynamics of logical inference in large language model reasoning

    2025

  57. [65]

    Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang, and Yangqiu Song. 2025 c . http://arxiv.org/abs/2505.13259 From automation to autonomy: A survey on large language models in scientific discovery

    2025

  58. [66]

    Tianshi Zheng, Jiazheng Wang, Zihao Wang, Jiaxin Bai, Hang Yin, Zheye Deng, Yangqiu Song, and Jianxin Li. 2025 d . http://arxiv.org/abs/2501.00759 Enhancing transformers for generalizable first-order logical entailment

    2025 arXiv

  59. [67]

    Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. http://arxiv.org/abs/2303.11315 Context-faithful prompting for large language models

    2023 arXiv

  60. [68]

    Qing Zong, Zhaowei Wang, Tianshi Zheng, Xiyu Ren, and Yangqiu Song. 2024. http://arxiv.org/abs/2412.20251 Comparisonqa: Evaluating factuality robustness of llms through knowledge frequency control and uncertainty

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.