{"id":"516886dc-82e3-4533-941f-eb2e7f71c55a","arxiv_id":"2504.14462","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CoLoTa rewrites StrategyQA and CREAK questions to replace famous entities with obscure Wikidata entities and shows that LLMs, including OpenAI-o1, drop sharply in accuracy and reason less reliably on the long-tail versions.","lead":"This paper introduces CoLoTa, a dataset of 3,300 commonsense reasoning questions and claims about obscure, long-tail entities from Wikidata. It shows that strong language models answer these queries far less accurately than the popular-entity questions they were rewritten from, and that they often hallucinate while doing so.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Original-vs-CoLoTa gap in Tables 5-6 does not isolate long-tail entity popularity, because Section 3.2 also rewrites questions, removes implicit assumptions, and adds 'commonsense reasoning indirection'; the claimed causal effect is therefore not supported by the current paired comparison.","rationale":"The reader's weakest assumption identified the same load-bearing concern: the Original versus CoLoTa comparison varies entity popularity together with query rewriting, naturalness, and added commonsense indirection. My stress-test of the full text confirms this is the central threat to the paper's causal claim. The condition needed for the central claim to hold is that the performance gap is attributable to long-tail entity popularity; Section 3.2 explicitly admits that other modifications were made, and Table 2 shows semantic changes beyond entity substitution. This does not destroy the value of CoLoTa as a hard benchmark, but it does prevent full acceptance of the stronger conclusion about long-tail knowledge being the cause of the drop. A matched popular-entity control would settle the question. Since the reader's verdict already reflects this concern, I recommend no change to the verdict.","tokens_in":16808,"tokens_out":5263,"duration_ms":48568,"concrete_test":"Construct a matched control set from CoLoTa: for each query, replace only the long-tail anchor entities with popular entities matched for semantic type (same occupation, position, or time period), preserving the rewritten phrasing and any added indirection exactly. Run the same LLM configurations from Table 5 on this control. If accuracy and FActScore on the popular-entity control remain near CoLoTa levels and far below the original StrategyQA/CREAK numbers, the gap is driven by rewriting or indirection rather than entity popularity; if they recover to near-original levels, the long-tail attribution is supported. A minimal version is to restrict the comparison to the subset of CoLoTa items whose rewrites introduced no indirection and re-estimate the gap on that subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that replacing popular entities with long-tail counterparts causes drops in accuracy, factuality, and reasoning quality, summarized as 'These results demonstrate that LLMs ... struggle with long-tail ones' in Section 4.3. The evidence is the paired Original vs. CoLoTa comparison in Tables 5 and 6. However, the two sets of queries differ on multiple dimensions at once. Section 3.2 (Query rewriting) states that CoLoTa queries were rewritten following Linjordet and Balog's naturalness scheme, had 'incorrect implicit assumptions' removed, and were 'made more challenging by adding commonsense reasoning indirection.' The worked examples in Table 2 confirm this: 'Did Francois Mitterrand ever meet Barak Obama...' becomes 'Could Liau Hiok-hian and Virginia Raggi have met...', changing modality and the reasoning required; 'Could Tom Cruise explain mental auditing?' becomes 'Is it likely for Julia Nickson-Soul to be familiar with mental auditing?', changing the target property and adding a likelihood judgment. The popularity distributions in Figures 4-5 show only that the intended manipulation occurred; they do not show that question difficulty, naturalness, and reasoning complexity were held fixed. Consequently, the observed accuracy/FActScore/reasoning-score gaps are consistent with the hypothesis that CoLoTa queries are simply harder questions, not specifically with the claim that long-tail entity knowledge is the cause. The dataset may still be a useful hard benchmark, but the causal conclusion in Section 4.3 is not internally valid as currently evidenced. A secondary issue is that Table 6's manual FActScore and reasoning scoring lacks inter-annotator agreement and sample size, but the primary problem is the confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoLoTa, a dataset of 3,300 entity-based commonsense reasoning queries (half question answering, half claim verification) formed by rewriting StrategyQA and CREAK queries, replacing popular entities with long-tail Wikidata entities, and annotating each entry with anchor-entity QIDs, a relevant Wikidata sub-graph, an inference rule, and reasoning steps. The authors evaluate five LLMs under zero-shot and few-shot Chain-of-Thought prompting plus two LLM-based KGQA systems on both the original queries and their CoLoTa counterparts. They report consistent drops in accuracy, answer rate, FActScore, and reasoning score for CoLoTa and interpret these as evidence that LLMs are unreliable for commonsense reasoning about long-tail entities. They further propose CoLoTa as a benchmark for both LLM commonsense reasoning and KGQA.","tokens_in":16953,"tokens_out":6476,"duration_ms":56171,"significance":"If the central claim were fully established, the paper would provide a valuable benchmark and a systematic demonstration of a significant LLM failure mode. The dataset construction has real strengths: queries are anchored to Wikidata QIDs, factual support is explicitly verified, inference rules and reasoning steps are supplied, and the popularity distributions confirm the intended long-tail focus. The availability of the data and evaluation code is a plus, as is the breadth of baselines covering both general LLMs and KGQA methods. However, the current evidence does not isolate entity popularity as the cause of the performance drop, because the paired comparison also changes question wording, removes implicit assumptions, and deliberately adds reasoning indirection. The dataset may still be useful as a hard benchmark, but the paper's stronger causal claims about long-tail knowledge are not yet supported.","major_comments":[{"comment":"The central comparison between Original and CoLoTa queries does not isolate entity popularity as the cause of the observed drops. Section 3.2 states that queries were rewritten following Linjordet and Balog's naturalness scheme, that incorrect implicit assumptions were removed, and that \"we also make some queries more challenging by adding commonsense reasoning indirection.\" The examples in Table 2 illustrate changes beyond entity substitution: modality changes (\"Did\" to \"Could\"), property changes (\"explain mental auditing\" to \"familiar with mental auditing\"), and the introduction of intermediate entities (Figure 1). Therefore the accuracy, FActScore, and reasoning-score gaps in Tables 5 and 6 could be due to differences in question difficulty or required reasoning complexity rather than long-tail entity knowledge. This directly affects the paper's conclusion in Section 4.3 that \"LLMs can adeptly perform commonsense reasoning about popular entities ... but struggle with long-tail ones.\" I recommend either adding a control condition that replaces only the entities while preserving all other lexical and logical properties, or substantially tempering the causal interpretation and reframing CoLoTa as a difficulty-controlled benchmark.","section":"Section 3.2; Tables 5 and 6"},{"comment":"The manual evaluation of FActScore and reasoning score lacks the details needed to support the hallucination and reasoning-error claims. The authors state that calculating these metrics requires manual verification of all intermediate steps, but they do not report the number of responses annotated, the annotation protocol, or inter-annotator agreement. Without reliability information, the reported score gaps (e.g., OpenAI-o1 FActScore dropping from 0.98 to 0.58 on the question-answering task with few-shot CoT) cannot be distinguished from annotation noise. In addition, the definition of the reasoning score (V(s_i)=1 iff s_i can be logically deduced from all previous steps) checks only internal logical consistency and does not require the premises to be factually true, so a chain of hallucinated but internally coherent steps could receive full credit. The metric should be clarified, and preferably revised to require both factual support and logical validity.","section":"Section 4.2 and Table 6"},{"comment":"The paper does not describe how the ground-truth answers of the rewritten queries were validated. Since the rewriting removed implicit assumptions and changed the meaning of some queries (for example, \"Could Tom Cruise explain mental auditing?\" becomes \"Is it likely for Julia Nickson-Soul to be familiar with mental auditing?\"), the original answers may not carry over automatically. The authors should report an independent validation of the answer labels for CoLoTa, such as a second annotator checking the inference rule and the resulting answer, or at least an agreement measure. This is important because the dataset is proposed as a benchmark and the correctness of the labels underpins all conclusions.","section":"Section 3.2"}],"minor_comments":[{"comment":"Accuracy and answer-rate differences are reported without confidence intervals or significance tests; given that some drops are small (e.g., Gemini-1.5 Flash 0.20 on several rows), a statistical test would help the reader assess whether the paired differences are reliable.","section":"Table 5"},{"comment":"The formal definition of the inference rule contains undefined notation and the equation appears corrupted in the rendered text (e.g., \"› « |P|Û\" and the placement of operators). Please fix the typesetting and explicitly define all symbols, including the range of the index i and the semantics of the comparison operator.","section":"Section 3.1, Eq. (1)"},{"comment":"Some labels in the KG subgraph appear to be typos or artifacts (e.g., \"Divingsharesborder with\" and \"Toronto sport\"); please verify the triples and correct any erroneous labels.","section":"Figure 1"},{"comment":"There are citation formatting issues, notably the malformed reference \"[32? ]\" in Section 2.2 and the duplicate/reordered entries for Sun et al. (2023 and 2024); please standardize all references.","section":"References"},{"comment":"The choice of k=2 for few-shot Chain-of-Thought prompting is not justified; please discuss whether results are sensitive to the number of examples or provide a small sensitivity analysis.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well within the scope of a SIGIR datasets-and-evaluation track, and the dataset has clear potential value. The main obstacle is the causal claim about long-tail entity popularity, which is not supported by the current paired comparison because of the simultaneous rewriting and difficulty manipulations in Section 3.2. If the authors add a controlled entity-only substitution experiment or reframe the contribution as a hard benchmark without causal claims, the paper could become a solid accept. I would also encourage the editor to ask for full annotation details for the manual metrics in Table 6."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoLoTa is a genuinely useful dataset, but the paper's central causal claim — that long-tail entity popularity is what makes LLMs fail — is not supported by the current experiments. The original and CoLoTa queries differ on entity popularity, yes, but also on wording, naturalness, and added 'commonsense reasoning indirection' (Section 3.2). That is a confound, and the stress-test note gets it right.\n\nWhat the paper does well: the resource itself. 3,300 queries over two tasks, each with Wikidata QIDs, relevant KG subgraphs, explicit inference rules, and decomposed reasoning steps. That is a real contribution. It makes the dataset usable not only for LLM evaluation but also for KGQA work, and the authors are honest about the fact that they do not provide SPARQL annotations. The entity popularity distributions in Figures 4-5 confirm the long-tail manipulation. The accuracy drops in Table 5 are large and consistent across ten LLM configurations, so the phenomenon is real even if the attribution is muddy.\n\nWhere it gets soft: Section 4.3's summary sentence — 'These results demonstrate that LLMs can adeptly perform commonsense reasoning about popular entities... but struggle with long-tail ones' — is stronger than the design allows. You could get the same pattern if the CoLoTa rewrites simply made the questions harder or less familiar in ways unrelated to entity frequency. A controlled comparison that varies only the entity would be the fix. Also, Table 6 reports manual FActScore and reasoning scores without inter-annotator agreement or a stated sample size. That is a minor-to-moderate weakness; it doesn't sink the paper but it should be reported.\n\nThe 'first KGQA dataset with commonsense reasoning' claim is hedged with 'to the best of our knowledge' in the text, which is fine, though a broader prior-art check would make it safer.\n\nWho is this for? Benchmark builders, LLM hallucination researchers, and KGQA people who want a non-factoid challenge set. It deserves a serious referee. I'd recommend accepted with revisions: soften the causal interpretation or add the controlled experiment, and add annotation reliability statistics.","headline":"CoLoTa is a valuable new benchmark, but the paper's causal claim about long-tail entity popularity is confounded by simultaneous query rewrites.","tokens_in":17652,"tokens_out":2375,"would_cite":true,"duration_ms":20068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that large language models are systematically unreliable at commonsense reasoning about obscure, long-tail entities, and introduces CoLoTa, a 3,300-query dataset with Wikidata-backed facts, inference rules, and reasoning…","keywords":["commonsense reasoning","long-tail knowledge","entity-based reasoning","hallucination","LLM evaluation","benchmark dataset","knowledge graph question answering","Wikidata"],"falsifier":"Run a controlled parallel pair where the identical natural-language query is evaluated once with a popular entity and once with an obscure entity that has comparable Wikidata support, holding wording, sentence length, and commonsense steps fixed; if accuracy, FActScore, and reasoning score show no systematic gap across models, the central claim is refuted. Alternatively, if providing the relevant Wikidata triples to the model restores performance to the popular-entity level, the observed gap would be attributable to missing facts rather than to reasoning failure over long-tail knowledge.","tokens_in":16470,"feed_emoji":"🧠","tokens_out":6237,"duration_ms":50838,"temperature":0.7,"pith_summary":"The paper sets out to show that the commonsense reasoning ability of large language models depends heavily on how famous the entities in the question are. To test this, it builds CoLoTa, a 3,300-query dataset that rewrites questions from two existing benchmarks, replacing well-known entities with obscure ones that still have full factual support in the Wikidata knowledge graph. Across the models tested, accuracy, factual precision, and reasoning validity all drop on the long-tail versions, while the models still answer at nearly the same rate. That combination, wrong answers delivered confidently, is what the authors identify as hallucination under commonsense reasoning, and it is the phenomenon the dataset is designed to expose and measure.","feed_headline":"LLMs lose commonsense when questions target obscure people","feed_subtitle":"A new 3,300-query benchmark shows accuracy and reasoning precision collapse on long-tail Wikidata entities.","key_machinery":"The central mechanism is the parallel query pair: each CoLoTa entry takes an original StrategyQA question or CREAK claim and rewrites it around a long-tail Wikidata entity, so the factual information needed to answer is still present in the graph but likely absent from model training. Each entry also carries an explicit commonsense inference rule, an ordered decomposition into reasoning steps, and the relevant Wikidata sub-graph with entity QIDs, turning the task into one that can be scored for factual support (FActScore) and reasoning validity separately. Entity substitution is done by SPARQL queries over Wikidata that find candidate entities with similar properties and much smaller triple counts, which operationalizes 'long-tail' as low Wikidata popularity.","core_discovery":"CoLoTa demonstrates that state-of-the-art LLMs are systematically less reliable when commonsense reasoning must be grounded in obscure rather than popular entities. For every model and prompting strategy tested, accuracy on the rewritten long-tail queries falls well below accuracy on the original popular-entity queries, with drops from 0.15 to 0.42 depending on model and task. The drop is not mainly a refusal to answer: answer rates stay high for the strongest models, while FActScore and a novel reasoning score fall sharply, indicating that correct-sounding answers rest on unsupported facts and invalid inference steps. The same degradation appears in LLM-based KGQA pipelines, which suggests that existing KGQA methods do not combine graph facts with commonsense inference.","pith_inferences":["A controlled variant of CoLoTa that changes only entity names while holding wording identical would isolate how much of the drop is due to popularity rather than the accompanying rewrites.","The parallel structure could be extended to other reasoning types, such as physical or social commonsense, to test whether the long-tail effect generalizes beyond the skills sampled from StrategyQA and CREAK.","The dataset's QIDs and reasoning steps make it a plausible testbed for retrieval-augmented systems; one could predict that supplying the relevant subgraph reduces hallucinations but may not fix invalid inference steps."],"forward_implications":["Real deployments that ask models about rare people, places, or products should expect confident fabricated facts and faulty commonsense steps, not calibrated refusals.","Benchmarking LLM commonsense on famous entities overstates ability; evaluation should include long-tail counterparts as a standard practice.","Because every CoLoTa query is backed by a Wikidata subgraph, the benchmark can test whether supplying those triples to a model closes the accuracy gap.","The results imply KGQA methods need a commonsense reasoning component beyond semantic parsing, since graph retrieval alone does not answer these queries.","The reasoning-score drop implies improving factual recall alone will not fix long-tail commonsense; the inference itself needs attention."],"supporting_citations":[{"why":"Supplies the original StrategyQA questions, their popularity-focused baselines, and the strategy decompositions the authors prioritize when selecting queries.","marker":"[7]"},{"why":"Supplies the original CREAK claims used as the source for the claim verification half of CoLoTa.","marker":"[21]"},{"why":"Provides prior evidence that LLMs struggle with long-tail facts, motivating the gap this paper extends to commonsense reasoning.","marker":"[30]"},{"why":"Defines FActScore, the metric used to measure whether atomic facts in model responses are supported by Wikidata.","marker":"[19]"},{"why":"Provides the naturalness rewriting scheme that CoLoTa follows when rephrasing queries, including grammar and form dimensions.","marker":"[16]"},{"why":"Serves as a KGQA baseline that uses an LLM to propose claims and then retrofits them on the knowledge graph.","marker":"[10]"},{"why":"Serves as a KGQA baseline that uses few-shot in-context learning to convert natural language queries to SPARQL.","marker":"[15]"}],"fun_headline_variants":["LLMs stumble on commonsense about obscure entities","New benchmark: LLMs fail commonsense on long-tail knowledge","Obscure entities break LLM commonsense reasoning","CoLoTa: LLM reasoning collapses on rare entities","Even o1 trips on commonsense about unknown people"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's claim depends on the assumption that the performance gap between original and CoLoTa queries is caused by entity popularity, even though the rewritten queries were also made more natural and sometimes added extra reasoning steps; if those wording and indirection changes drive the gap, the long-tail story weakens.","fun_headline_variants_meta":{"raw":{"variants":["LLMs stumble on commonsense about obscure entities","New benchmark: LLMs fail commonsense on long-tail knowledge","Obscure entities break LLM commonsense reasoning","CoLoTa: LLM reasoning collapses on rare entities","Even o1 trips on commonsense about unknown people"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1391,"prompt_tokens":955,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":357}},"tokens_in":571,"tokens_out":436,"duration_ms":4477,"temperature":1.0,"reasoning_tokens":357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:48:19.987117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled parallel pair where the identical natural-language query is evaluated once with a popular entity and once with an obscure entity that has comparable Wikidata support, holding wording, sentence length, and commonsense steps fixed; if accuracy, FActScore, and reasoning score show no systematic gap across models, the central claim is refuted. Alternatively, if providing the relevant Wikidata triples to the model restores performance to the popular-entity level, the observed gap would be attributable to missing facts rather than to reasoning failure over long-tail knowledge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FActScore, the metric used to measure whether atomic facts in model responses are supported by Wikidata."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the naturalness rewriting scheme that CoLoTa follows when rephrasing queries, including grammar and form dimensions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as a KGQA baseline that uses an LLM to propose claims and then retrofits them on the knowledge graph."}],"review_version":1}