Pith. sign in

REVIEW 3 major objections 5 minor 47 references

CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that large language models are systematically unreliable at commonsense reasoning about obscure, long-tail entities, and introduces CoLoTa, a 3,300-query dataset with Wikidata-backed facts, inference rules, and reasoning…

desk verdict CoLoTa is a valuable new benchmark, but the paper's causal claim about long-tail entity popularity is confounded by simultaneous query rewrites. read the letter →

arxiv 2504.14462 v1 pith:2YSAYIPV submitted 2025-04-20 cs.CL

Armin Toroghi , Willis Guo , Scott Sanner This is my paper · ORCID
classification cs.CL
keywords commonsensereasoninglong-tailknowledgeentity-basedhallucinationLLMevaluationbenchmarkdatasetgraphquestionansweringWikidata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the commonsense reasoning ability of large language models depends heavily on how famous the entities in the question are. To test this, it builds CoLoTa, a 3,300-query dataset that rewrites questions from two existing benchmarks, replacing well-known entities with obscure ones that still have full factual support in the Wikidata knowledge graph. Across the models tested, accuracy, factual precision, and reasoning validity all drop on the long-tail versions, while the models still answer at nearly the same rate. That combination, wrong answers delivered confidently, is what the authors identify as hallucination under commonsense reasoning, and it is the phenomenon the dataset is designed to expose and measure.

What carries the argument

The central mechanism is the parallel query pair: each CoLoTa entry takes an original StrategyQA question or CREAK claim and rewrites it around a long-tail Wikidata entity, so the factual information needed to answer is still present in the graph but likely absent from model training. Each entry also carries an explicit commonsense inference rule, an ordered decomposition into reasoning steps, and the relevant Wikidata sub-graph with entity QIDs, turning the task into one that can be scored for factual support (FActScore) and reasoning validity separately. Entity substitution is done by SPARQL queries over Wikidata that find candidate entities with similar properties and much smaller triple counts, which operationalizes 'long-tail' as low Wikidata popularity.

What would settle it

Run a controlled parallel pair where the identical natural-language query is evaluated once with a popular entity and once with an obscure entity that has comparable Wikidata support, holding wording, sentence length, and commonsense steps fixed; if accuracy, FActScore, and reasoning score show no systematic gap across models, the central claim is refuted. Alternatively, if providing the relevant Wikidata triples to the model restores performance to the popular-entity level, the observed gap would be attributable to missing facts rather than to reasoning failure over long-tail knowledge.

Watch

Extended reading notes

Core claim

CoLoTa demonstrates that state-of-the-art LLMs are systematically less reliable when commonsense reasoning must be grounded in obscure rather than popular entities. For every model and prompting strategy tested, accuracy on the rewritten long-tail queries falls well below accuracy on the original popular-entity queries, with drops from 0.15 to 0.42 depending on model and task. The drop is not mainly a refusal to answer: answer rates stay high for the strongest models, while FActScore and a novel reasoning score fall sharply, indicating that correct-sounding answers rest on unsupported facts and invalid inference steps. The same degradation appears in LLM-based KGQA pipelines, which suggests that existing KGQA methods do not combine graph facts with commonsense inference.

Load-bearing premise

The paper's claim depends on the assumption that the performance gap between original and CoLoTa queries is caused by entity popularity, even though the rewritten queries were also made more natural and sometimes added extra reasoning steps; if those wording and indirection changes drive the gap, the long-tail story weakens.

Editorial extensions

If this is right

  • Real deployments that ask models about rare people, places, or products should expect confident fabricated facts and faulty commonsense steps, not calibrated refusals.
  • Benchmarking LLM commonsense on famous entities overstates ability; evaluation should include long-tail counterparts as a standard practice.
  • Because every CoLoTa query is backed by a Wikidata subgraph, the benchmark can test whether supplying those triples to a model closes the accuracy gap.
  • The results imply KGQA methods need a commonsense reasoning component beyond semantic parsing, since graph retrieval alone does not answer these queries.
  • The reasoning-score drop implies improving factual recall alone will not fix long-tail commonsense; the inference itself needs attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A controlled variant of CoLoTa that changes only entity names while holding wording identical would isolate how much of the drop is due to popularity rather than the accompanying rewrites.
  • The parallel structure could be extended to other reasoning types, such as physical or social commonsense, to test whether the long-tail effect generalizes beyond the skills sampled from StrategyQA and CREAK.
  • The dataset's QIDs and reasoning steps make it a plausible testbed for retrieval-augmented systems; one could predict that supplying the relevant subgraph reduces hallucinations but may not fix invalid inference steps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CoLoTa, a dataset of 3,300 entity-based commonsense reasoning queries (half question answering, half claim verification) formed by rewriting StrategyQA and CREAK queries, replacing popular entities with long-tail Wikidata entities, and annotating each entry with anchor-entity QIDs, a relevant Wikidata sub-graph, an inference rule, and reasoning steps. The authors evaluate five LLMs under zero-shot and few-shot Chain-of-Thought prompting plus two LLM-based KGQA systems on both the original queries and their CoLoTa counterparts. They report consistent drops in accuracy, answer rate, FActScore, and reasoning score for CoLoTa and interpret these as evidence that LLMs are unreliable for commonsense reasoning about long-tail entities. They further propose CoLoTa as a benchmark for both LLM commonsense reasoning and KGQA.

Significance. If the central claim were fully established, the paper would provide a valuable benchmark and a systematic demonstration of a significant LLM failure mode. The dataset construction has real strengths: queries are anchored to Wikidata QIDs, factual support is explicitly verified, inference rules and reasoning steps are supplied, and the popularity distributions confirm the intended long-tail focus. The availability of the data and evaluation code is a plus, as is the breadth of baselines covering both general LLMs and KGQA methods. However, the current evidence does not isolate entity popularity as the cause of the performance drop, because the paired comparison also changes question wording, removes implicit assumptions, and deliberately adds reasoning indirection. The dataset may still be useful as a hard benchmark, but the paper's stronger causal claims about long-tail knowledge are not yet supported.

major comments (3)
  1. [Section 3.2; Tables 5 and 6] The central comparison between Original and CoLoTa queries does not isolate entity popularity as the cause of the observed drops. Section 3.2 states that queries were rewritten following Linjordet and Balog's naturalness scheme, that incorrect implicit assumptions were removed, and that "we also make some queries more challenging by adding commonsense reasoning indirection." The examples in Table 2 illustrate changes beyond entity substitution: modality changes ("Did" to "Could"), property changes ("explain mental auditing" to "familiar with mental auditing"), and the introduction of intermediate entities (Figure 1). Therefore the accuracy, FActScore, and reasoning-score gaps in Tables 5 and 6 could be due to differences in question difficulty or required reasoning complexity rather than long-tail entity knowledge. This directly affects the paper's conclusion in Section 4.3 that "LLMs can adeptly perform commonsense reasoning about popular entities ... but struggle with long-tail ones." I recommend either adding a control condition that replaces only the entities while preserving all other lexical and logical properties, or substantially tempering the causal interpretation and reframing CoLoTa as a difficulty-controlled benchmark.
  2. [Section 4.2 and Table 6] The manual evaluation of FActScore and reasoning score lacks the details needed to support the hallucination and reasoning-error claims. The authors state that calculating these metrics requires manual verification of all intermediate steps, but they do not report the number of responses annotated, the annotation protocol, or inter-annotator agreement. Without reliability information, the reported score gaps (e.g., OpenAI-o1 FActScore dropping from 0.98 to 0.58 on the question-answering task with few-shot CoT) cannot be distinguished from annotation noise. In addition, the definition of the reasoning score (V(s_i)=1 iff s_i can be logically deduced from all previous steps) checks only internal logical consistency and does not require the premises to be factually true, so a chain of hallucinated but internally coherent steps could receive full credit. The metric should be clarified, and preferably revised to require both factual support and logical validity.
  3. [Section 3.2] The paper does not describe how the ground-truth answers of the rewritten queries were validated. Since the rewriting removed implicit assumptions and changed the meaning of some queries (for example, "Could Tom Cruise explain mental auditing?" becomes "Is it likely for Julia Nickson-Soul to be familiar with mental auditing?"), the original answers may not carry over automatically. The authors should report an independent validation of the answer labels for CoLoTa, such as a second annotator checking the inference rule and the resulting answer, or at least an agreement measure. This is important because the dataset is proposed as a benchmark and the correctness of the labels underpins all conclusions.
minor comments (5)
  1. [Table 5] Accuracy and answer-rate differences are reported without confidence intervals or significance tests; given that some drops are small (e.g., Gemini-1.5 Flash 0.20 on several rows), a statistical test would help the reader assess whether the paired differences are reliable.
  2. [Section 3.1, Eq. (1)] The formal definition of the inference rule contains undefined notation and the equation appears corrupted in the rendered text (e.g., "› « |P|Û" and the placement of operators). Please fix the typesetting and explicitly define all symbols, including the range of the index i and the semantics of the comparison operator.
  3. [Figure 1] Some labels in the KG subgraph appear to be typos or artifacts (e.g., "Divingsharesborder with" and "Toronto sport"); please verify the triples and correct any erroneous labels.
  4. [References] There are citation formatting issues, notably the malformed reference "[32? ]" in Section 2.2 and the duplicate/reordered entries for Sun et al. (2023 and 2024); please standardize all references.
  5. [Section 4.1] The choice of k=2 for few-shot Chain-of-Thought prompting is not justified; please discuss whether results are sensitive to the number of examples or provide a small sensitivity analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CoLoTa benchmark is an empirical dataset construction, not a derivation that reduces to its inputs.

full rationale

The paper's central claim is empirical: LLMs show higher error and hallucination rates on CoLoTa's long-tail queries than on the original popular-entity queries (Tables 5 and 6). This is a benchmark evaluation, not a mathematical derivation. CoLoTa queries are formed by rewriting StrategyQA and CREAK queries and replacing popular entities with obscure Wikidata entities, but the ground-truth answers and reasoning annotations come from human annotation and external Wikidata facts, not from the models being evaluated. No parameter is fitted to a subset of the evaluation data and then renamed as a prediction; the reported accuracies, FActScores, and reasoning scores are measured against independently annotated labels. The only 'prediction' is the performance of the models on held-out queries, which is exactly the proper use of a benchmark. The fact that CoLoTa is deliberately constructed to be challenging ('We also make some queries more challenging by adding commonsense reasoning indirection,' Section 3.2) is a design property, not circularity. The additional rewriting and removal of 'incorrect implicit assumptions' in Section 3.2 do mean the Original-vs-CoLoTa gap may not isolate entity popularity as the sole cause of the drop; that is a potential confounding-variable concern about causal attribution, but it is not a circularity of the kind where a result is true by construction or by self-citation. The paper's own prior works appear in the reference list but are not invoked as load-bearing evidence for the main empirical claim. Therefore, no specific circular step can be identified under the required standard, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The benchmark's usefulness depends on the correctness of its ground-truth answers and inference rules, on Wikidata's accuracy, and on the experiment's ability to attribute the performance gap to long-tail entity popularity. None of these are machine-checked; all are paper-internal or annotation-based assumptions.

assumptions (5)
  • domain assumption The manually authored inference rules and ground-truth answers for CoLoTa queries are correct.
    Section 3.1 presents inference rules as natural-language FOL expressions written by the authors for each query. No inter-annotator agreement or external validation is reported for the final labels, so an incorrect rule would propagate to both the answer and the reasoning evaluation.
  • domain assumption The original StrategyQA and CREAK answers remain valid after entity substitution and rewriting.
    Section 3.2 replaces popular entities with similar long-tail counterparts and modifies wording. The paper does not describe a separate step that re-derives the answer label from the substituted facts, so answer transfer is assumed.
  • domain assumption Wikidata contains accurate and sufficient factual support for each CoLoTa query.
    Query selection in Section 3.2 uses two annotators to verify that necessary facts exist in Wikidata. Wikidata itself can contain errors, outdated statements, or missing qualifiers, and no audit of triple correctness is reported.
  • domain assumption The Original versus CoLoTa comparison isolates long-tail entity popularity as the cause of performance differences.
    The comparison assumes all other factors are equal, but Section 3.2 states that CoLoTa queries add commonsense reasoning indirection and improve wording, so the design does not control for reasoning complexity.
  • domain assumption FActScore and the manual Reasoning score are valid and reliable measures of hallucination and reasoning errors.
    Section 4.2 defines these metrics, but the paper reports no inter-annotator agreement, no response-level sample size for Table 6, and no validation that atomic fact decomposition is stable across annotators.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge." pith.science (2026). https://pith.science/paper/2YSAYIPV

@misc{pith2026250414462,
  author       = {Pith},
  title        = {Pith review of: CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YSAYIPV}},
  note         = {Machine review of arXiv:2504.14462}
}
read the original abstract

The rise of Large Language Models (LLMs) has redefined the AI landscape, particularly due to their ability to encode factual and commonsense knowledge, and their outstanding performance in tasks requiring reasoning. Despite these advances, hallucinations and reasoning errors remain a significant barrier to their deployment in high-stakes settings. In this work, we observe that even the most prominent LLMs, such as OpenAI-o1, suffer from high rates of reasoning errors and hallucinations on tasks requiring commonsense reasoning over obscure, long-tail entities. To investigate this limitation, we present a new dataset for Commonsense reasoning over Long-Tail entities (CoLoTa), that consists of 3,300 queries from question answering and claim verification tasks and covers a diverse range of commonsense reasoning skills. We remark that CoLoTa can also serve as a Knowledge Graph Question Answering (KGQA) dataset since the support of knowledge required to answer its queries is present in the Wikidata knowledge graph. However, as opposed to existing KGQA benchmarks that merely focus on factoid questions, our CoLoTa queries also require commonsense reasoning. Our experiments with strong LLM-based KGQA methodologies indicate their severe inability to answer queries involving commonsense reasoning. Hence, we propose CoLoTa as a novel benchmark for assessing both (i) LLM commonsense reasoning capabilities and their robustness to hallucinations on long-tail entities and (ii) the commonsense reasoning capabilities of KGQA methods.

Figures

Figures reproduced from arXiv: 2504.14462 by the authors.

Figure 1
Figure 1. Constituents of an entry from CoLoTa: (i) an entity-based commonsense reasoning query (ii) Wikidata QIDs of the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Distribution of reasoning skills in claim verifica [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Distribution of popularity of the entities targeted [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Calculation of FActScore and Reasoning score for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 27 canonical work pages

  1. [1]

    Marco Baroni, Armand Joulin, Allan Jabri, German Kruszewski, Angeliki Lazari- dou, Klemen Simonic, and Tomas Mikolov. 2017. CommAI: Evaluating the first steps towards a useful general AI. arXiv preprint arXiv:1701.08954 (2017)

    arXiv 2017

  2. [2]

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Seman- tic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing . 1533–1544

    work page 2013

  3. [3]

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL, 1...

    work page 2013

  4. [4]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , Vol. 34. 7432–7439

    2020

  5. [5]

    Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor

  6. [6]

    Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58, 9 (2015), 92–103

    work page 2015

  7. [7]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Trans. Assoc. Comput. Linguistics 9 (2021), 346–361. doi:10.1162/TACL_A_00370

    doi:10.1162/tacl_a_00370 2021

  8. [8]

    Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond IID: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021 . 3477–3488

    work page 2021

Show all 47 references
  1. [9]

    Sadler, Percy Liang, Xifeng Yan, and Yu Su

    Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases. InWWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, Jure...

    2021

  2. [10]

    Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. 2024. Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18126–18134

    2024

  3. [11]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel

  4. [12]

    Gregory Karvounarakis, Sofia Alexaki, Vassilis Christophides, Dimitris Plex- ousakis, and Michel Scholl. 2002. RQL: a declarative query language for RDF. In Proceedings of the 11th international conference on World Wide Web . 592–603

    2002

  5. [13]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213

    2022

  6. [14]

    Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web 6, 2 (2015), 167–195

    2015

  7. [15]

    Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023. Few- shot In-context Learning on Knowledge Base Question Answering. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6966–6980

    2023

  8. [16]

    Trond Linjordet and Krisztian Balog. 2022. Would you ask it that way? measuring and improving question naturalness for knowledge graph question answering. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . 3090–3098

    2022

  9. [17]

    Hugo Liu and Push Singh. 2004. ConceptNet—a practical commonsense reasoning tool-kit. BT technology journal 22, 4 (2004), 211–226

    2004

  10. [18]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Compu...

    2023

  11. [19]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine- grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empiri...

    2023

  12. [20]

    Robert C Moore. 1982. The role of logic in knowledge representation and common- sense reasoning. SRI International. Artificial Intelligence Center

    1982

  13. [21]

    Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. 2021. Creak: A dataset for commonsense reasoning over entity knowledge. arXiv preprint arXiv:2109.01653 (2021)

    2021 arXiv

  14. [22]

    Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Dur- rett. 2021. CREAK: A Dataset for Commonsense Reasoning over En- tity Knowledge. In Proceedings of the Neural Information Processing Sys- tems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Bench- marks 20...

    2021

  15. [23]

    Nidhi Rastogi and Mohammed J Zaki. 2020. Personal health knowledge graphs for patients. arXiv preprint arXiv:2004.00071 (2020)

    2020 arXiv

  16. [24]

    Shaina Raza, Mizanur Rahman, Safiullah Kamawal, Armin Toroghi, Ananya Raval, Farshad Navah, and Amirmohammad Kazemeini. 2024. A comprehensive review of recommender systems: Transitioning from theory to practice. arXiv preprint arXiv:2407.13699 (2024)

    2024

  17. [25]

    Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI conference on artificial intelligence , ...

    2019

  18. [26]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense Reasoning about Social Interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na...

    2019

  19. [27]

    Andy Seaborne and Eric Prud’hommeaux. 2008. SPARQL query language for RDF. W3C Recommendation, W3C (2008)

    2008

  20. [28]

    Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, Satinder Singh and Shaul M...

    2017

  21. [29]

    Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024. Head- to-Tail: How Knowledgeable are Large Language Models (LLMs)? AKA Will LLMs Replace Knowledge Graphs?. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational...

    2024

  22. [30]

    Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2023. Head- to-tail: How knowledgeable are large language models (llm)? AKA will llms replace knowledge graphs? arXiv preprint arXiv:2308.10168 (2023)

    2023 arXiv

  23. [31]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Com- monsenseQA: A Question Answering Challenge Targeting Commonsense Knowl- edge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2019

  24. [32]

    Armin Toroghi, Griffin Floto, Zhenwei Tang, and Scott Sanner. 2023. Bayesian Knowledge-driven Critiquing with Indirect Evidence. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1838–1842

    2023

  25. [33]

    Armin Toroghi, Willis Guo, Ali Pesaranghader, and Scott Sanner. 2024. Verifiable, Debuggable, and Repairable Commonsense Logical Reasoning via LLM-based Theory Resolution. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 6634–6652

    2024

  26. [34]

    Armin Toroghi, Willis Guo, Mohammad Mahdi Abdollah Pour, and Scott Sanner

  27. [35]

    Armin Toroghi and Scott Sanner. 2024. Bayesian inference with complex knowl- edge graph evidence. In Proceedings of the AAAI Conference on Artificial Intelli- gence, Vol. 38. 20550–20558

    2024

  28. [36]

    Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs. In The Semantic Web - ISWC 2017 - 16th International Semantic Web Conference, Vienna, Austria, October 21-25, 2017, Proceedings, ...

    2017

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processing ...

    2022

  30. [38]

    Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The Value of Semantic Parse Labeling for Knowledge Base Question Answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12,...

    2016 doi

  31. [39]

    Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base ques- tion answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pape...

    2016

  32. [40]

    Weiguo Zheng, Hong Cheng, Lei Zou, Jeffrey Xu Yu, and Kangfei Zhao. 2017. Natural language question/answering: Let users talk with the knowledge graph. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 217–226

    2017

  33. [41]

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. 2024. Evaluation of openai o1: Opportunities and challenges of agi. arXiv preprint arXiv:2409.18486 (2024)

    2024

  34. [42]

    Xin Zhou, Kisub Kim, Bowen Xu, Jiakun Liu, DongGyun Han, and David Lo

  35. [47]

    arXiv preprint arXiv:2309.03567 (2023)

    The devil is in the tails: How long-tailed code distributions impact large language models. arXiv preprint arXiv:2309.03567 (2023)

    2023 arXiv

  36. [2008]

    In Proceedings of the 2008 ACM SIGMOD international conference on Management of data

    Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data. 1247–1250

    2008

  37. [2023]

    In Interna- tional Conference on Machine Learning

    Large language models struggle to learn long-tail knowledge. In Interna- tional Conference on Machine Learning . PMLR, 15696–15707

  38. [2024]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Right for Right Reasons: Large Language Models for Verifiable Com- monsense Knowledge Graph Question Answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . 6601–6633

    2024

  39. [4451]

    doi:10.1609/AAAI.V31I1.11164

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.