{"id":"5437ca49-7611-4101-a5e3-9a19ff33bab6","arxiv_id":"2501.03040","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ChronoSense evaluates LLMs on all 13 Allen interval relations and temporal arithmetic, finding weak, inconsistent performance and signs of memorization across seven models.","lead":"ChronoSense is a new benchmark that tests large language models on all 13 Allen interval relations and three temporal arithmetic tasks using year-level event data from Wikidata. Across seven models, accuracy is often near chance, and the authors argue that models may rely on memorization rather than genuine temporal reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Year-level labels do not entail the gold Allen relation for items with shared boundary years, so several task accuracies and the memorization comparison are not cleanly interpretable.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I find. The paper's contribution—a 13-relation, multi-prompt, multi-model benchmark—is real, and the broad 'before/after easier than endpoint-sensitive relations' pattern is suggestive. But the year-granularity label ambiguity does not just add noise; it makes the gold answer undefined for a substantial class of items. A model can be marked wrong because it answers according to the true exact-interval relation, which the context does not uniquely determine. This directly threatens both the per-relation accuracy claims and the memorization comparison. Since the reader already conditioned acceptance on fixing this issue, I keep the verdict unchanged rather than escalating. I do not see an independent, more damaging objection: the synthetic arithmetic tasks are unambiguous, and the instruction-following failures are honestly flagged in the tables.","tokens_in":15941,"tokens_out":8110,"duration_ms":81356,"concrete_test":"Entailment audit of the released test set: for each Allen item, enumerate (or densely sample) all exact start/end dates consistent with the given start and end years; compute the exact Allen relation for every instantiation and determine whether the gold label is true for all compatible instantiations, false for all, or mixed. Report mixed fractions overall and per relation. If any of the 13 tasks has more than ~5% mixed items, those accuracy scores should be recomputed on the subset of entailed items only; if the subset is too small for stable estimates, the benchmark must be regenerated at day/month granularity or relabeled as approximate year-interval comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central comparison—LLMs fail at Allen interval relations—requires each True/False item to have a determinate Allen-relation answer. Section 3 fixes granularity to years, and A.1.1 acknowledges ambiguity only when choosing negative candidates. Positive labels are generated by comparing start/end years as if they determined the Allen relation, but they do not. Two events both stated as 'between year 1900 and year 1900' can be exactly Equals, Before, After, Meets, or several other Allen relations depending on the days involved. The same underdetermination affects Equals, Starts, Finishes, Meets, Met-By, Contains, During, Started-By, Finished-By, Overlaps, and Overlapped-By whenever any boundary year coincides. Thus the gold 'True' may be true for some exact dates consistent with the context and false for others; accuracy on those tasks measures agreement with an ill-posed question, not Allen reasoning. The A.1.1 negative-sample exclusions do not repair this: they restrict only the pool of false hypotheses, and the positive labels are still not entailed. Because several of the reported 'hard' relations (notably Equals, Starts, Finishes) are exactly the underdetermined ones, the headline low accuracies and the abstract-vs-real memorization gap are not cleanly interpretable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ChronoSense introduces a benchmark for evaluating LLMs on Allen's 13 interval relations and three temporal arithmetic tasks, using real event intervals from Wikidata plus abstract versions, across 0-shot, few-shot, and chain-of-thought prompting. The authors evaluate seven LLMs and report generally low accuracy on Allen relations, uneven performance across symmetric relations, and a drop in accuracy when real event names are replaced by abstract labels, which they interpret as evidence that models rely on memorization. The dataset and code are released.","tokens_in":16152,"tokens_out":2364,"duration_ms":24753,"significance":"If the measurement were clean, the benchmark would fill a genuine gap: most prior temporal-reasoning benchmarks cover only a subset of Allen relations, whereas ChronoSense covers all 13 and releases data and code, which is a useful resource for the community. The paper also provides a broad comparison across seven open and closed models and multiple prompting regimes. However, the load-bearing measurement is compromised by the year-granularity problem: for many items the provided context does not determine the gold Allen relation, so the accuracy numbers and the memorization comparison are not cleanly interpretable. The paper's own qualitative example (Appendix A.5, Example #1) shows a model correctly noting this ambiguity, which underscores that the issue is real rather than hypothetical.","major_comments":[{"comment":"The core technical issue is that year-level start/end values do not determine the true Allen relation for any pair in which a boundary year coincides. For example, two events both stated as occurring 'between year 1900 and year 1900' may be Equals, Before, After, Meets, Overlaps, or other relations depending on the actual days; similar underdetermination affects Starts, Finishes, Started-By, Finished-By, Contains, During, Overlaps, Overlapped-By, Meets, and Met-By whenever a start or end year is shared. The construction in Section 3 ('We set the time granularity to years') and the positive-label generation in A.1.1 therefore assign a unique gold relation where none is entailed by the context. The negative-sample exclusions in A.1.1 address only the pool of false hypotheses; they do not repair positive labels. Because the reported accuracies for these relations (Tables 2, 4-8) include items with indeterminate labels, the headline low accuracies for relations such as Equals, Starts, and Finishes are not a valid measure of Allen-relation reasoning. The fix is to filter items to those where the year-level interval uniquely entails the relation, or to use finer-grained dates.","section":"Section 3 and A.1.1"},{"comment":"The memorization conclusion rests on comparing real-event accuracy (Table 4) with abstract-event accuracy (Table 8). This comparison is confounded: the abstract setting removes event names, but it also changes the distribution of the items (e.g., by removing any possibility of using real-world priors about event durations) and is only reported in the 0-shot setting, whereas the real-event results are reported across all settings. The observed drop could reflect the absence of useful topical priors or a format mismatch rather than memorization specifically. The authors should either provide a matched comparison that controls for these factors (e.g., using plausible but incorrect event names, or measuring memorization by probing the model's factual recall separately) or temper the claim in the abstract and conclusion that models 'may rely on memorization.'","section":"General Findings, item 6 and Table 8"},{"comment":"The paper's own qualitative example illustrates the label-validity problem: GPT-4o-mini answers 'True' to a Finishes question while stating that the exact dates are not specified, and the paper labels this a failure. In fact, given only the years, the model's expressed uncertainty is justified, and the gold label itself is not entailed. This example should either be removed or reframed; as presented, it supports the referee's concern rather than the paper's interpretation.","section":"A.5, Example #1"}],"minor_comments":[{"comment":"The 'Contains' prompt variant 1 reads 'Does Event A fully enclose Event A...' but should refer to Event B; this typo appears in two places in the appendix table and could affect reproducibility if the templates are used verbatim.","section":"Table 10"},{"comment":"The relation name is written inconsistently as 'Met-by' in Table 3 and 'Met-By' in Tables 2, 4-8 and in the text; please standardize the capitalization.","section":"Throughout"},{"comment":"Several typographical issues: 'Next-Occurence' should be 'Next-Occurrence', and the table header repeats 'End-Timepoint (0-shot)' style inconsistently with the text; also, in Table 11 the spacing around '0 .95' and similar entries appears as a LaTeX artifact that should be cleaned.","section":"Table 9 and A.2"},{"comment":"The sentence 'we do not observe a relation that has consistently high values' is unclear; the intended meaning appears to be that no relation has consistently low variance across prompt variants, or that no prompt variant consistently outperforms others. Please rephrase.","section":"A.3"},{"comment":"The claim that 'random prediction would lead to 0.50 accuracy' is only true if the model always answers True or False; the paper's own asterisk convention shows that several models produce many unclear answers, so the random baseline should be stated conditionally or per-model.","section":"Section 4, General Findings item 1"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful contribution in terms of dataset breadth and public release, but the central measurement is undermined by the year-granularity label-entailment problem. This is not a mere presentation issue: it affects the interpretation of the main accuracy tables and the memorization claim. The fix is straightforward in principle (filter to entailed items or use finer granularity), so I do not see this as a reject, but it requires a substantive revision and re-analysis of the affected numbers. I would also encourage the authors to address the confound in the abstract-versus-real comparison and to consider whether the 'unclear answer' handling should be treated as a separate outcome rather than as incorrect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ChronoSense is a benchmark that does something no prior dataset has done: it covers all 13 Allen interval relations for LLM evaluation, with both real and abstract event names, plus arithmetic tasks. The related-work survey is honest about what existing benchmarks cover, the dataset construction is transparent, and the evaluation spans seven models and multiple prompting strategies. That part is solid and worth knowing about.\n\nThe soft spot is serious. The dataset sets time granularity to years, and the gold label for each item is computed by comparing start and end years. But an Allen relation is a claim about exact intervals. When two events share a boundary year, the stated context does not entail the label. For example, two events both \"between 1900 and 1900\" can be Equals, Before, After, Meets, or other relations depending on days. The paper's own appendix (A.1.1) acknowledges this ambiguity when sampling negative candidates, excluding relations that would be underdetermined. But it never applies the same reasoning to positive labels. So accuracy on Equals, Starts, Finishes, Meets, Met-By, and the overlap variants measures agreement with an ill-posed question, not Allen reasoning. That undercuts the headline claims about those relations being especially hard, and the abstract-vs-real memorization gap is confounded as well.\n\nThe broad conclusion that LLMs struggle with temporal reasoning is probably right, and the Before/After results, where year-level intervals do determine the relation, are meaningful. But the benchmark as released does not support the per-relation analysis for a majority of the 13 relations.\n\nOther, smaller issues: the starred results flag instruction-following failures that are averaged into the headline numbers, and there are no error bars in the main tables (the appendix gives prompt-variant std devs, which is good). The paper also has a few typos in the prompt alternatives (\"Does 'Event A' fully enclose 'Event A'\").\n\nI would send this to a serious reviewer. The benchmark is a real contribution to the temporal reasoning evaluation space, and the fix is tractable: filter to event pairs whose Allen relation is entailed by year intervals, or move to finer granularity, or explicitly relabel the task as approximate interval comparison. With that fix, it could become a standard diagnostic.","headline":"A genuinely new benchmark, but the year-granularity labeling makes most of the per-relation results uninterpretable.","tokens_in":16704,"tokens_out":2380,"would_cite":false,"duration_ms":23149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current LLMs lack robust temporal understanding of Allen interval relations, as shown by accuracies near or below 60 percent and a drop when real event names are replaced by abstract labels, which the authors…","keywords":["temporal reasoning","Allen interval algebra","large language models","benchmark","Wikidata","memorization","chain-of-thought prompting","temporal arithmetic"],"falsifier":"Take only the event pairs where the two intervals are unambiguous at year granularity, for example Before pairs with a gap of several years, and compare model accuracy on those against accuracy on boundary-sharing pairs such as identical start years. If models perform near chance on the unambiguous pairs too, the paper's claim of poor temporal reasoning is confirmed; if they score high there but low on ambiguous pairs, the benchmark's labels, not the models' reasoning, are the problem.","tokens_in":15692,"feed_emoji":"⏳","tokens_out":6829,"duration_ms":57419,"temperature":0.7,"pith_summary":"The paper presents ChronoSense, a benchmark that tests large language models on all thirteen Allen interval relations between pairs of events, plus three temporal arithmetic tasks. It evaluates seven widely used LLMs in zero-shot, few-shot, chain-of-thought, and abstract-label settings. The central finding is that models perform poorly and inconsistently on Allen relations, with several scoring below 0.60 accuracy, and that replacing real event names with abstract labels lowers accuracy for most models, which the authors take as evidence of memorization rather than reasoning. A sympathetic reader would care because temporal understanding underpins applications like historical analysis, legal AI, and medical timelines, and the paper provides a diagnostic tool for measuring it.","feed_headline":"Most LLMs score below 60% on Allen time-interval relations","feed_subtitle":"New ChronoSense benchmark tests all 13 Allen relations; abstract labels drop scores, hinting at memorization.","key_machinery":"The central object is Allen's interval algebra, the set of 13 mutually exclusive relations that can hold between two time intervals (Equals, Before, After, Overlaps, Overlapped-by, Contains, During, Starts, Started-by, Finishes, Finished-by, Meets, Met-by). The benchmark verbalizes each relation as a natural-language Hypothesis, presents pairs of event intervals from Wikidata, and asks the model to judge the statement True or False. The argument's engine is the controlled comparison between named events and abstract events ('Event A'/'Event B'): the accuracy gap between those two conditions is what carries the memorization claim.","core_discovery":"On its own terms, the paper claims that current large language models do not robustly reason about the time intervals of events. Using the 13 mutually exclusive Allen relations, ChronoSense asks models to answer True or False about whether a described relation holds between two events whose start and end years are given. The results show that one model, GPT-4o, reaches 0.87 zero-shot accuracy, while most other models remain near or below 0.60, and that the models treat symmetrical relations (e.g., Before vs After, Meets vs Met-by) asymmetrically. When the real event names are replaced with abstract labels, accuracy drops for most models, and the paper interprets this gap as a sign that the models lean on memorized event knowledge rather than computing the relation from the stated intervals. The authors conclude that temporal understanding in LLMs needs improvement and offer ChronoSense as a benchmark for future research.","pith_inferences":["If the memorization finding holds, then temporal QA benchmarks built on encyclopedic events should always include abstract or synthetic controls to separate recall from reasoning.","A natural next experiment is to generate synthetic event pairs with random names and day-level timestamps; if the abstract-accuracy gap persists, it would confirm the effect is not an artifact of Wikidata naming.","The year-granularity ambiguity for boundary-sharing relations means some reported accuracies may be conservative lower bounds; re-annotating with finer dates or excluding ambiguous pairs could change model rankings.","The novel Intermediate Timepoint task, which asks whether an event was happening at a given year, could be extended to continuous intervals and used as a probe for interval-membership reasoning separate from arithmetic."],"forward_implications":["The benchmark gives researchers a way to test all 13 Allen relations instead of the subsets covered by earlier temporal reasoning datasets.","Chain-of-thought prompting lifts temporal arithmetic accuracy substantially for nearly all tested models, suggesting step-by-step reasoning is effective for calculation-style temporal questions.","The named-versus-abstract accuracy gap implies that evaluations using only real-world events may overstate genuine temporal reasoning ability.","Models' asymmetric performance on symmetric relation pairs such as Before/After and Meets/Met-by shows that relation labels carry lexical biases that should be controlled in future benchmarks."],"supporting_citations":[{"why":"Defines the 13 interval relations that constitute ChronoSense's task taxonomy.","marker":"Allen, 1989"},{"why":"Supplies the real-world event intervals extracted via Wikidata SPARQL for named-event questions.","marker":"Vrandečić and Krötzsch, 2014"},{"why":"Source of the 'Let's think step by step' instruction used in the chain-of-thought condition.","marker":"Kojima et al., 2022"},{"why":"Establishes chain-of-thought prompting, the reasoning-augmentation method the CoT setting builds on.","marker":"Wei et al., 2022"},{"why":"Provides the (Context, Hypothesis, Correctness) format used for each test instance.","marker":"Yang et al., 2023"},{"why":"Previous temporal reasoning benchmark that covers only subsets, motivating ChronoSense's coverage of all 13 relations.","marker":"Chu et al., 2023"}],"fun_headline_variants":["Most LLMs under 60% on Allen time-interval tests","ChronoSense: LLMs struggle with Allen interval relations","New benchmark reveals LLMs weak on time interval reasoning","LLMs rely on memorization for time questions, ChronoSense shows","ChronoSense: Most LLMs fail Allen time interval tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset assumes that the start and end years given for each event are sufficient to determine the correct Allen relation, even though relations like Equals, Starts, and Finishes depend on exact dates, not just years.","fun_headline_variants_meta":{"raw":{"variants":["Most LLMs under 60% on Allen time-interval tests","ChronoSense: LLMs struggle with Allen interval relations","New benchmark reveals LLMs weak on time interval reasoning","LLMs rely on memorization for time questions, ChronoSense shows","ChronoSense: Most LLMs fail Allen time interval tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000964,"raw_usage":{"total_tokens":4102,"prompt_tokens":942,"completion_tokens":3160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":3072}},"tokens_in":558,"tokens_out":3160,"duration_ms":21578,"temperature":1.0,"reasoning_tokens":3072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:57:56.931535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take only the event pairs where the two intervals are unambiguous at year granularity, for example Before pairs with a gap of several years, and compare model accuracy on those against accuracy on boundary-sharing pairs such as identical start years. If models perform near chance on the unambiguous pairs too, the paper's claim of poor temporal reasoning is confirmed; if they score high there but low on ambiguous pairs, the benchmark's labels, not the models' reasoning, are the problem.","supporting_citations":[],"review_version":1}