{"id":"e05bd82e-2ff7-44d7-9157-e6dea5411478","arxiv_id":"2411.12449","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Neon builds a timestamped knowledge graph of entity-event tuples extracted from news, and augmenting LLM prompts with these tuples improves temporal entity-centric question answering.","lead":"This paper describes Neon, a system that reads news articles, extracts short timestamped summaries of entity interactions (for example 'Doja Cat deleted her album announcement from Instagram'), and feeds those summaries to an LLM to answer time-sensitive questions about people and companies. The authors report that these compact event tuples make chatbot answers more helpful and faithful, sometimes rivaling a live web search engine.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported QA gains rest on an LLM judge that the paper itself finds correlates only weakly with human ratings (relevance 0.12), so the central comparison is not established.","rationale":"The reader's weakest_assumption focuses on whether the LLM-written openIE tuples are accurate, complete, and faithful to the underlying news articles. That is a genuine concern: Section 2.1 constructs the graph by prompting an LLM to generate tuples, and no precision or recall numbers are reported anywhere. However, I think the more load-bearing issue is upstream of tuple quality: even if every tuple were correct, the evaluation protocol used to support the main claim is not validated. The paper's own human-assessment results show very weak agreement between the automated judge and human ratings, especially for relevance (0.12), and the qualitative analysis in Section 5 shows the judge violates the rating scale and is confounded by response length. These are internal observations, not outside disagreements, so they directly undermine Tables 4 and 5. I therefore partially agree with the reader: the tuple-quality gap is real, but the unvalidated LLM judge is the condition that must hold for any of the experimental conclusions to be meaningful. Because the reader already assigned CONDITIONAL with specific requests for human evaluation and significance testing, my recommendation does not change the verdict; it sharpens the reason why those conditions are necessary. The proposed concrete test--method-level human factuality and preference evaluation with external verification--would settle whether the reported Neon advantage reflects genuine QA quality or a miscalibrated automatic metric.","tokens_in":17986,"tokens_out":4978,"duration_ms":54303,"concrete_test":"Run a method-level human evaluation on a stratified sample of at least 300 temporal queries from the paper's dataset: for each query, give annotators the reformulated query, the retrieved supporting passages, and the response from Neon(M2), NewsRAG, and WebRAG under temporal retrieval, and ask for (a) a factuality check against the cited original news articles with external search allowed, and (b) a pairwise preference between Neon(M2) and WebRAG. If human pairwise preference and factual accuracy do not place Neon(M2) at or above WebRAG, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is not tuple accuracy alone; it is that the 3-point Likert scores from GPT-4o used in Tables 4 and 5 measure the construct the paper claims. Section 4's Human Assessment reports Pearson correlations between human and few-shot automatic evaluations of only 0.24 (helpfulness), 0.12 (relevance), and 0.19 (faithfulness) over 100 responses. For relevance, the attribute that temporal retrieval is specifically designed to improve, the correlation is essentially zero. Section 5 further documents that the judge emits ratings of 5 and 9 on a 3-point scale, and Table A3 shows that ratings track response length. Consequently, the headline result that Neon(M2) 'achieves the highest overall average score across metrics' (Table 5) and the abstract's 'substantial improvements' are not supported as evidence about QA quality: the apparent advantage could be an artifact of the judge preferring longer, structured, tuple-flavored answers rather than more correct or temporally relevant ones. This is independent of whether the Neon tuples themselves are accurate; even perfect tuples would not rescue the claim if the outcome measure is miscalibrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Neon, a framework for constructing an entity-centric, timestamped knowledge graph from news streams by prompting LLMs to produce openIE-style interaction tuples (subject, object, relation verbalization). Two graph variants are proposed: Neon(M1), which extracts interactions around a single subject entity, and Neon(M2), which extracts interactions for explicitly paired subject-object entities using TF-IDF-selected co-occurring pairs. The authors evaluate the framework on temporal entity-centric question answering using a dataset of roughly 3,000 real Bing queries across 50 entities and a four-month news corpus. They compare retrieval-augmented generation using Neon tuples against retrieval from news chunks (NewsRAG) and live Bing snippets (WebRAG), with responses generated by GPT-4o and scored by GPT-4o on 3-point Likert scales for helpfulness, relevance, and faithfulness. The main reported result is that, under few-shot evaluation prompts, Neon(M2) achieves the highest overall average score across the three metrics in the temporal retrieval setting, rivaling a live web-scale search engine. The paper also reports a human assessment on 100 responses showing weak positive correlations between human and automatic scores, and it documents evaluation artifacts such as off-scale ratings and a correlation between response length and scores.","tokens_in":18186,"tokens_out":4090,"duration_ms":43819,"significance":"If the central claim held, Neon would make a useful empirical contribution: a compact, structured representation of timestamped entity interactions that can be injected into LLM prompts to improve temporal entity-centric QA, with performance competitive against live web retrieval. The work has genuine strengths: it uses real user queries from search logs, includes both zero-shot and few-shot evaluation settings, reports a human annotation study, and candidly discloses known artifacts of LLM-based evaluation. However, the paper's own evaluation evidence is the major weakness. The correlation between the automated judge and human scores is weak for helpfulness and faithfulness, and essentially zero for relevance (0.12), which is the attribute the method is specifically intended to improve. The paper also never measures the precision, recall, or temporal accuracy of the Neon tuples that constitute the whole system.","major_comments":[{"comment":"The paper's headline result is based entirely on the few-shot GPT-4o judge scores used in Tables 4 and 5, yet the Human Assessment subsection reports Pearson correlations between human ratings and these automatic scores of only 0.24 (helpfulness), 0.12 (relevance), and 0.19 (faithfulness) over 100 responses. Relevance is the attribute that temporal retrieval is specifically designed to improve, and its correlation is effectively zero. Consequently, the claim in Table 5 that 'Neon(M2) achieves the highest overall average score across metrics' and the abstract's 'substantial improvements' are not established as claims about QA quality; the apparent advantage could be an artifact of the judge preferring longer, more structured responses. The authors must either report method-by-method comparisons using human scores (or a recalibrated judge validated against humans on the full evaluation set) and provide significance tests, or substantially soften the central claims.","section":"Section 4, Human Assessment"},{"comment":"The Neon graph is the core contribution, but its construction is never evaluated. Section 2.1 describes prompting GPT-4o to generate lexicalized NEI tuples and assigning each tuple the timestamp of the source chunk, yet the paper reports no precision, recall, or temporal-accuracy figures for these tuples. If a nontrivial fraction of tuples are hallucinated, ungrammatical, or misattributed to the wrong date, the faithfulness scores in Tables 4 and 5 only measure whether the response uses the provided passages, not whether those passages are true or temporally aligned. The authors should sample Neon tuples and have annotators judge (a) whether the interaction is entailed by the source chunk, (b) whether it is semantically valid, and (c) whether the timestamp matches the event described; the extraction quality should be reported for both Neon(M1) and Neon(M2).","section":"Section 2.1, Graph Construction"},{"comment":"Tables 4 and 5 report means on a 3-point Likert scale without variance, confidence intervals, or significance tests. The key differences are small—for example, the temporal few-shot average is 1.55 for Neon(M2) versus 1.48 for NewsRAG and 1.52 for WebRAG (generic). These margins are on the same order as the noise implied by the weak human-automation correlations. Without per-query paired tests or bootstrap confidence intervals, the claim that Neon 'rivals' or 'substantially improves' over baselines is not statistically grounded. At minimum, the paper should report standard errors or confidence intervals and a paired significance test for the headline comparisons.","section":"Section 4, Tables 4 and 5"},{"comment":"There is a potential circularity in the evaluation loop: GPT-4o is used to generate the Neon tuples (Section 2.1), to generate the QA responses (Section 3, Evaluation), and to rate those responses on the three Likert attributes (Section 3, Evaluation). The paper does not control for judge-model preferences for outputs that resemble its own style of tuple-flavored, structured text. The response-length analysis in Section 5 and Table A3 suggests exactly such a confounding bias, since longer responses receive higher scores for helpfulness and relevance. Using a different model (e.g., a different family or a fine-tuned evaluator) for the automatic assessment, or reporting human evaluation on the full set, would materially strengthen the claim that the observed gains reflect temporal relevance rather than formatting preferences.","section":"Sections 2.1, 3, and 4"}],"minor_comments":[{"comment":"The formal definition G = (E, T, R) does not include the lexicalized interaction descriptions I that are introduced in the same paragraph; clarify whether I is part of the graph or a notational shorthand for the verbalization function.","section":"Section 2.1, formal definition"},{"comment":"The prompt templates for Relevance and Faithfulness contain a copy-paste error: they instruct the model to provide 'reasoning for the helpfulness rating' instead of reasoning for the relevance/faithfulness rating.","section":"Table A4, Prompt templates"},{"comment":"The temporal retrieval strategy uses an exact timestamp match on t_d = t_q with a ±r day fallback, but the paper does not specify how 'timestamp' is normalized across different time zones or source publication times; a brief clarification would help reproducibility.","section":"Section 2.2, Temporal Retrieval"},{"comment":"The entity linking system NEMO is cited only via a 2014 ERD paper and is described as proprietary; for a research audience, at least a high-level description of its accuracy or recall on the news domain would help calibrate the trust one can place in the entity markup that drives both Neon variants.","section":"Section 3, News Articles"},{"comment":"The three quality attributes are defined in the text, but their mutual correlations are not examined despite the paper noting that helpfulness and relevance correlate >0.8 in both human and automatic scores; this threatens the independence of the averaged metric and should be discussed.","section":"Section 3, Evaluation"},{"comment":"The paper does not release code, data, or the set of 3,000 queries (likely due to privacy), but it also does not state a clear availability statement; readers are left without a way to reproduce or build on the results.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a strong and timely idea, but the experimental evidence as presented does not support the abstract-level claims. The authors are unusually honest about the flaws of LLM-based evaluation, but the reported correlations (especially relevance 0.12) and the absence of any tuple-quality measurement make the central result unverified. This is fixable within scope: re-run the evaluation with human judgments or a validated judge, report statistical uncertainty, and add an extraction-quality study. If the authors deliver those, the paper could be acceptable. I would not reject outright because the framework is well-motivated and the shortcomings are explicitly, if incompletely, acknowledged. One scope concern worth raising with the authors: the use of proprietary Bing logs and NEMO means the exact dataset is not public, which limits reproducibility, but this is common in industrial collaborations and should not by itself block publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: Neon is a sensible system for temporal entity-centric QA—extract timestamped openIE-style tuples from news into a knowledge graph, retrieve them with temporal constraints, and feed them into an LLM prompt—plus a new 3,000-query Bing dataset. The engineering is coherent and the paper is honest about its own evaluation limits. But the quantitative claim that it rivals live web search is not supported by the numbers.\n\nThe few-shot LLM judge it relies on correlates with human scores at only 0.24 (helpfulness), 0.12 (relevance), and 0.19 (faithfulness). Relevance is exactly the attribute temporal retrieval is meant to improve, and that correlation is essentially zero. Worse, Table A3 shows longer answers get higher ratings, and Section 5 admits ratings of 5 and 9 on a 3-point scale. So the small margins in Tables 4 and 5 (e.g., Neon(M2) at 1.55 vs. NewsRAG at 1.48) could easily be an artifact of the judge preferring longer, structured, tuple-flavored text. That is a load-bearing flaw for the main claim.\n\nWhat's genuinely new: the entity-first markup to get openIE extraction without canonicalization, the M1/M2 variants, and the real Bing query collection with spiking dates. The qualitative examples are instructive, and the authors earn credit for reporting the weak correlations, the off-scale ratings, and the length effect. They don't hide these issues.\n\nThe other soft spots are secondary. Tuple precision/recall is never measured, which is a real gap, but it matters less than the judge miscalibration—even perfect tuples wouldn't rescue the claim if the outcome measure is broken. No error bars or significance tests, and the margins are tiny. The WebRAG comparison is a bit apples-and-oranges, but that's minor.\n\nWho is this for? Anyone working on RAG, temporal QA, or LLM-based evaluation. It's a useful case study in how a broken judge can drive conclusions. The dataset and system are potentially useful if released. Should it be peer-reviewed? Yes—the system is real, the dataset is new, and the evaluation problems are fixable with human evaluation at the method level, extraction quality numbers, and significance tests. It deserves a serious referee, not a desk rejection.","headline":"Neon is a well-built temporal QA system with a new real-query dataset, but its headline evaluation relies on an LLM judge that correlates weakly with humans, so the claimed edge over web search is not established.","tokens_in":18786,"tokens_out":3057,"would_cite":true,"duration_ms":28400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Timestamped entity-interaction tuples from news improve temporal QA and rival live web search.","keywords":["temporal question answering","information extraction","large language models","knowledge graph","retrieval-augmented generation","entity-centric query","news streams","open information extraction"],"falsifier":"Have two annotators verify a random sample of 200 Neon(M2) tuples against the original news chunks and article timestamps, computing precision and timestamp-accuracy rates; if a substantial share of tuples are unsupported or misdated, the faithfulness gains reported in Tables 4 and 5 would not reflect genuine grounding.","tokens_in":17742,"feed_emoji":"📰","tokens_out":6840,"duration_ms":59860,"temperature":0.7,"pith_summary":"The paper tries to establish that a timestamped graph of short entity-interaction tuples, extracted from news streams by an LLM, can make question answering about recent events more accurate and grounded than feeding the same model raw news passages. The authors argue that compact, entity-centric propositions avoid the 'lost in the middle' degradation of long contexts and give the LLM exactly the dated facts it needs. If correct, this would offer a lightweight alternative to live web search for entity-centric temporal queries, with results that rival a commercial search engine in few-shot evaluation.","feed_headline":"Timestamped news tuples rival live web search for entity queries","feed_subtitle":"Compact, dated entity-interaction extracts beat raw news snippets, and can match a full web search engine.","key_machinery":"The load-bearing mechanism is the LLM-generated lexicalized entity-interaction tuple $(t, \\text{NEI}(s,o))$ — a single natural-language sentence describing an event or activity between entities, stamped with the article's timestamp. The paper calls these 'news entity-interactions' and stores them in a temporal datastore indexed by dense embeddings. Retrieval either matches the query's timestamp exactly (with a $\\pm r$-day back-off) or uses semantic similarity over the reformulated, time-stamped query. Because each tuple is one sentence, the prompt context stays compact, avoiding the lost-in-the-middle and information-overload problems that hurt longer news chunks.","core_discovery":"On its own terms, the paper's central discovery is that replacing raw news chunks with LLM-generated openIE-style tuples, each stamped with the article's date, makes an LLM answer temporal entity-centric queries more faithfully and, with few-shot evaluation prompts, gives the highest average score across helpfulness, relevance, and faithfulness, ahead of live web search. This is achieved by two graph construction variants: Neon(M1), which extracts interactions of a single subject with marked entities, and Neon(M2), which targets explicit subject-object pairs selected by co-occurrence and TF-IDF, producing a larger, more detailed graph. The paper reports that under temporal retrieval, Neon(M2) outperforms WebRAG on the overall average in the few-shot setting, and that both Neon variants beat news-chunk RAG on helpfulness and faithfulness.","pith_inferences":["The paper never measures tuple precision or recall, so the faithfulness scores may partly reflect the LLM's confidence in its own paraphrases rather than verified grounding in the articles.","Given the weak correlation between human and automated ratings (0.12–0.24), the absolute Likert scores should be read cautiously even if the relative ordering of methods still holds.","The same compact-tuple recipe could transfer to other fast-moving text streams, such as financial filings or sports feeds, with timestamps taken from the stream items.","Replacing exact-date matching with a learned time-relevance scoring function could improve retrieval on sparsely covered dates beyond the paper's $\\pm r$-day back-off."],"forward_implications":["A precomputed entity-interaction graph can serve temporal QA without an external search engine, using a retriever-agnostic dense index.","Injecting the query date explicitly during reformulation is enough to activate effective temporal retrieval.","Doubling the supporting passages from five to ten lowers faithfulness for every method, with the drop more acute for longer passages.","A hybrid system that combines Neon's faithfulness with web search's breadth could improve all three metrics, as the paper leaves to future work.","The larger Neon(M2) graph, about four times the size of Neon(M1), yields the highest overall average score under few-shot temporal retrieval."],"supporting_citations":[{"why":"Supplies the entity linking tool NEMO that annotates named entities and co-references in news articles, the first step of Neon graph construction.","marker":"[9]"},{"why":"Documents the lost-in-the-middle effect that motivates using compact single-sentence tuples instead of long news chunks in LLM prompts.","marker":"[28]"},{"why":"Provides the unsupervised dense retrieval method used to index Neon tuples and retrieve by semantic similarity.","marker":"[17]"},{"why":"Supplies the contrastive text embeddings used to vectorize both queries and tuples for the temporal datastore.","marker":"[36]"},{"why":"Establishes the LLM-as-evaluator approach that the paper adapts for 3-point Likert scoring of helpfulness, relevance, and faithfulness.","marker":"[29]"},{"why":"The GPT-4 model family used as the backbone for both tuple extraction/evaluation and temporal QA generation (GPT-4o in experiments).","marker":"[39]"},{"why":"Defines the retrieval-augmented generation paradigm that Neon extends to temporal entity-centric QA.","marker":"[26]"}],"fun_headline_variants":["Dated openIE tuples beat raw news chunks for entity QA","NEON's timestamped entity graph outranks news-chunk RAG","Fresh news tuples with dates improve LLM answer faithfulness","Entity-interaction extracts with timestamps rival web search","Timestamped news tuples lift QA over live web results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The LLM-written tuples in the Neon graph accurately reflect the news articles they were extracted from, with correct timestamps, even though the paper never measures tuple precision or recall.","fun_headline_variants_meta":{"raw":{"variants":["Dated openIE tuples beat raw news chunks for entity QA","NEON's timestamped entity graph outranks news-chunk RAG","Fresh news tuples with dates improve LLM answer faithfulness","Entity-interaction extracts with timestamps rival web search","Timestamped news tuples lift QA over live web results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1493,"prompt_tokens":924,"completion_tokens":569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":486}},"tokens_in":540,"tokens_out":569,"duration_ms":5761,"temperature":1.0,"reasoning_tokens":486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:29:48.729260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two annotators verify a random sample of 200 Neon(M2) tuples against the original news chunks and article timestamps, computing precision and timestamp-accuracy rates; if a substantial share of tuples are unsupported or misdated, the faithfulness gains reported in Tables 4 and 5 would not reflect genuine grounding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the entity linking tool NEMO that annotates named entities and co-references in news articles, the first step of Neon graph construction."},{"cited_title":"Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang","cited_arxiv_id":null,"evidence_quote":"Documents the lost-in-the-middle effect that motivates using compact single-sentence tuples instead of long news chunks in LLM prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the unsupervised dense retrieval method used to index Neon tuples and retrieve by semantic similarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-evaluator approach that the paper adapts for 3-point Likert scoring of helpfulness, relevance, and faithfulness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the retrieval-augmented generation paradigm that Neon extends to temporal entity-centric QA."}],"review_version":1}