{"id":"c43210b8-1569-4e79-9496-5bed42fdc252","arxiv_id":"2509.03995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RTQA recursively decomposes complex temporal questions into sub-questions, solves them bottom-up with LLMs and retrieved TKG facts, and aggregates multiple answers, improving Hits@1 on MultiTQ and TimelineKGQA.","lead":"RTQA is a training-free framework that helps large language models answer complex temporal questions by recursively breaking them into sub-questions, solving each with retrieved knowledge-graph facts, then combining multiple answer paths. It reports accuracy gains on two temporal QA benchmarks, particularly for questions with compound constraints such as 'before' and 'last'.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may stem from RTQA's stronger reasoning LLM (DeepSeek-v3) rather than the framework; matched-LLM baseline comparison is missing.","rationale":"The reader's weakest_assumption focused on retrieval dependence, which is an acknowledged limitation and common to all retrieval-based methods; it does not by itself invalidate the comparison against baselines if they share the same retrieval constraints. The more load-bearing concern is the uncontrolled LLM backbone in the MultiTQ comparison—the primary evidence for 'state-of-the-art' performance. Because RTQA uses DeepSeek-v3 for reasoning while TimeR4's LLM is unspecified, the 0.037 overall gain could be due to the model rather than the framework. The paper's own 'w/o decomposer' ablation (0.709) is below TimeR4 (0.728), so the framework's contribution is not clearly isolated. The missing matched-LLM experiment is a concrete, testable gap. If the concern lands, the central claim is weakened but not necessarily false; the framework still has positive ablations, code release, and multi-LLM generalization, so a CONDITIONAL verdict remains appropriate. I therefore do not change the reader's verdict, but I identify a different primary concern than the reader did, hence partial agreement.","tokens_in":17901,"tokens_out":7835,"duration_ms":71536,"concrete_test":"Re-run the MultiTQ comparison with RTQA and TimeR4 (and, if possible, ARI) using the exact same reasoning LLM (e.g., DeepSeek-v3 or GPT-4o-mini) and the same retrieval setup (BGE-M3, top-50). If the Hits@1 gap over TimeR4 narrows to less than the reported 0.037 or disappears, the stated advantage is an artifact of the LLM choice. Additionally, report the LLM used for each published baseline; if TimeR4's original LLM is known, run RTQA with that LLM and compare. This single controlled experiment would settle whether recursive decomposition and aggregation, rather than a stronger backbone, drive the claimed improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RTQA 'outperforms all baselines across nearly all metrics' rests on comparisons whose fairness is not established. In Section 5.1, RTQA uses DeepSeek-v3 for answer reasoning on MultiTQ, while the TimeR4 baseline results are taken directly from (Qian et al., 2024) without reporting which LLM TimeR4 used. If TimeR4 used a weaker backbone (e.g., GPT-3.5), the 0.037 overall Hits@1 margin (0.765 vs. 0.728) could reflect the reasoning model's capability, not recursive decomposition. The internal ablation 'w/o decomposer' (0.709) actually underperforms TimeR4 (0.728), so the framework's gain is not clearly separable from the LLM choice. Section 5.4 ('Generalizability across different LLMs') only varies RTQA's solver LLM and never re-runs baselines with the same models, so it cannot control for this confound. The TimelineKGQA comparison similarly uses only a single RAG baseline, which is extremely weak on Complex questions (Hits@1 = 0.009), making the 'SOTA' claim there unsupported. Without a matched-LLM, matched-retriever comparison, the headline result is not trustworthy as evidence of framework superiority.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RTQA, a training-free framework for temporal knowledge graph question answering (TKGQA) that recursively decomposes complex temporal questions into sub-questions, solves them bottom-up with an LLM grounded in retrieved TKG facts, and aggregates answers from multiple paths to mitigate error propagation. The method is evaluated on MultiTQ and TimelineKGQA, reporting Hits@1 improvements over existing baselines, especially in 'Multiple' and 'Complex' question categories. The main claimed contribution is a plug-and-play recursive decomposition and aggregation scheme that improves complex temporal reasoning without retraining. The paper includes ablations, generalizability tests across LLMs, efficiency measurements, and a case study.","tokens_in":18172,"tokens_out":3033,"duration_ms":29293,"significance":"If the empirical claims hold, RTQA offers a practical, training-free approach to a challenging problem, with clear modular design and released code. The decomposition tree and multi-path aggregation are intuitive, and the ablation studies (e.g., w/o decomposer, w/o multi-answer) provide useful evidence about which components matter. The paper also honestly acknowledges its dependence on the retriever and LLM capability. However, the headline claim of 'outperforming state-of-the-art methods' rests on comparisons whose fairness is not yet established, and the statistical support is thin. The framework itself is plausible and likely valuable, but the current evidence does not fully separate the framework's contribution from the choice of reasoning LLM and baseline configurations.","major_comments":[{"comment":"The central claim that RTQA outperforms TimeR4 is not supported by a matched-LLM comparison. RTQA uses deepseek-v3 for answer reasoning, while TimeR4's results are taken from (Qian et al., 2024) without reporting the LLM backbone used there. Since the ablation 'w/o decomposer' (0.709) is below TimeR4 (0.728), the 0.037 overall Hits@1 margin could partly reflect the stronger reasoning model rather than the recursive framework. Please re-run TimeR4 (and preferably other LLM-based baselines) with the same reasoning LLM, retriever, and context limit as RTQA, and report these matched results.","section":"§5.1, Table 1"},{"comment":"The 'Generalizability across different LLMs' experiment varies only RTQA's solver LLM; baselines are not re-run with the same models. Thus it cannot control for the LLM confound raised above. The figure also reports results on a randomly sampled 1,000 questions with no variance or confidence intervals, so it is unclear whether differences among LLMs are statistically meaningful. Please include matched-LLM baseline runs on the same sample, and report repeated-run variance or at least a significance test.","section":"§5.4, Figure 4"},{"comment":"On TimelineKGQA, RTQA is compared only against a single RAG baseline, which is extremely weak on Complex questions (Hits@1 = 0.009). The abstract and conclusion claim SOTA performance on this benchmark, but with only one, apparently minimal, baseline this claim is unsupported. Please compare against stronger or additional baselines (e.g., other LLM-based TKGQA methods adapted to this benchmark, or a stronger RAG pipeline with the same retriever/LLM as RTQA). Without this, the 1400% 'improvement' on Complex questions cannot be interpreted as a fair comparison.","section":"§5.2, Table 2"},{"comment":"The context-limit analysis reports Hits@1 = 77.8% at n=50, but Table 1 reports overall Hits@1 = 76.5% for RTQA on MultiTQ. The paper does not explain this discrepancy (e.g., whether Table 6 is on a subset, uses a different evaluation setting, or is a typo). Also, the relationship between Hits@1 and Recall@n is not discussed: Hits@1 can exceed Recall@n only if the metric definitions differ, and the reader needs this clarified. This is load-bearing because n=50 is the main hyperparameter setting for the reported performance.","section":"§5.3, Table 6"},{"comment":"The multi-granular time analysis reports large gains over baselines (e.g., RTQA 0.916 vs. MultiQA 0.445 for Equal/Day), but the baseline numbers are taken from prior work without any indication of whether the same evaluation protocol and LLM settings were used. The lack of any statistical significance testing or repeated-run variance is especially problematic here, given the gains are claimed to be 'significant' in the abstract. Please provide at least standard errors or confidence intervals for the main comparisons, or temper the significance language.","section":"§5.4, Table 4"}],"minor_comments":[{"comment":"Typos: 'anather' in Figure 2; 'Sourth' in Figure 3; 'Timer4' in the reference for TimeR4; inconsistent spacing 'MULTI TQ' vs 'MultiTQ'.","section":"Throughout"},{"comment":"The 'Total' row for MultiTQ appears to have a typo: '587,979' is likely '58,979' for the dev set. Please check.","section":"Table 7"},{"comment":"The decomposition prompt examples are described as 'carefully selected from the validation set' with 'manually crafted' sub-question decompositions. This is a standard and acceptable practice, but it should be explicitly noted that the decomposition templates and few-shot examples are tuned on the validation set, and the paper should discuss potential selection bias.","section":"§4.2"},{"comment":"The implementation details indicate temperature 0 for deterministic outputs, but LLM APIs are not fully deterministic; the paper should state whether the reported numbers are from a single run and whether any sampling variance was observed.","section":"§5.1"},{"comment":"The error analysis mentions five error categories but does not quantify their frequencies. A small table or distribution would make the analysis more useful.","section":"§5.5, Error Analysis"}],"recommendation":"major_revision","confidential_remarks":"The core idea is solid and the paper is generally well-written, but the empirical validation does not yet support the strong SOTA claims. The missing matched-LLM baseline is the most serious issue, as it directly affects the interpretation of the main results. I would encourage the editor to request a revision that either adds the missing comparisons or substantially softens the claims. The paper fits the journal's scope and has potential, but the comparison fairness must be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. RTQA is a clean, training-free pipeline: decompose temporal questions into sub-trees, solve bottom-up with an LLM over retrieved TKG facts, and aggregate independent answer paths. The framework is sensible and the ablations show each piece earns its keep. But the empirical headline—\"outperforms SOTA\"—is not yet established, because the main MultiTQ comparison pits RTQA with DeepSeek-v3 against TimeR4 numbers taken from its paper, and the paper never says what LLM TimeR4 used. If that was a weaker model, the 0.037 overall Hits@1 margin is easily explained by the backbone, not the recursion.\n\nWhat is genuinely new: applying recursive decomposition with placeholder propagation to temporal constraints (before/after, first/last) and combining it with multi-path aggregation. The w/o decomposer ablation (0.214 vs 0.424 on Multiple questions) shows the decomposition is doing real work, and the w/o multi-answer ablation (0.341 vs 0.424 on Multiple) supports the fault-tolerance story. The w/o fact retrieval collapse (0.070) is dramatic and honestly reported. The paper also ships code and detailed prompts, and the limitation section explicitly flags retrieval dependence.\n\nThe soft spots are real, though. First, the LLM confound: Section 5.1 says RTQA uses deepseek-v3 for reasoning on MultiTQ while all TimeR4 results are lifted from (Qian et al., 2024); no LLM is specified for TimeR4. The generalizability experiment varies only the solver LLM and never re-runs any baseline with the same model. So the central superiority claim is unsubstantiated. Second, the TimelineKGQA comparison is only against a RAG baseline; the abstract's \"SOTA\" language overreaches there. Third, there are no error bars or significance tests; with 54K test questions, the 0.765 vs 0.728 gap might be stable, but we don't know. Fourth, Table 6 has Hits@1 exceeding Recall@n at n=50—probably fine since they are different metrics (QA accuracy vs retrieval recall), but it is presented without explanation.\n\nWho this is for: anyone working on temporal KGQA or LLM-based decomposition will find the prompt designs and ablation insights useful. It deserves a serious referee, but with a request for a matched-LLM, matched-retriever baseline comparison and error bars. I would accept it for review, not desk reject, and ask the authors to close the confound. That would turn a promising framework into a convincing one.","headline":"A training-free recursive decomposition framework for TKGQA that shows promising ablations, but the headline SOTA gains rest on a possibly mismatched LLM baseline—needs a matched-LLM comparison before believing the numbers.","tokens_in":18662,"tokens_out":2408,"would_cite":false,"duration_ms":21798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free recursive decomposition-and-aggregation framework raises Hits@1 on complex temporal knowledge graph QA, with the biggest gains on Multiple and Complex question categories.","keywords":["temporal knowledge graph question answering","large language models","question decomposition","recursive reasoning","answer aggregation","retrieval-augmented generation","temporal constraints","MultiTQ and TimelineKGQA"],"falsifier":"Take the subset of MultiTQ “Multiple” questions whose gold supporting fact is not among the top-50 retrieved facts. If RTQA answers a non-negligible fraction correctly, the recursive framework can reason past retrieval misses; if it answers near zero, the framework’s reported gains are bounded by retrieval recall, not by decomposition or aggregation.","tokens_in":17789,"feed_emoji":"🧩","tokens_out":7906,"duration_ms":66357,"temperature":0.7,"pith_summary":"The paper sets out to show that complex temporal questions—those with implicit temporal references, compound constraints such as “before” and “last,” and multi-hop dependencies—can be answered accurately without training a model. The proposed RTQA framework recursively decomposes each question into sub-questions, solves them bottom-up with an LLM grounded in retrieved temporal facts, and aggregates multiple answer paths to keep one wrong sub-answer from poisoning the final result. If true, this matters because prior TKGQA methods either handle only simple questions or decompose in one round, and the reported gains appear exactly where those methods fail: MultiTQ “Multiple” Hits@1 rises from 0.335 to 0.424, and TimelineKGQA “Complex” Hits@1 rises from 0.009 to 0.135. A reader should care because the approach is training-free and plug-and-play, so any capable LLM can be dropped into the solver stage.","feed_headline":"Recursive question-splitting beats trained TKGQA baselines","feed_subtitle":"A training-free framework decomposes temporal queries and aggregates answers; gains are largest on Multiple and Complex.","key_machinery":"The load-bearing object is the temporal question decomposition tree. A root question is split into ordered sub-questions; placeholders such as #1 refer to answers of earlier siblings, so implicit constraints become explicit. The Recursive Solver traverses the tree bottom-up in post-order, replacing placeholders with concrete timestamps or entities as it goes. At each non-leaf node, the Answer Aggregator fuses two candidates—IR_answer from direct retrieval-plus-reasoning and child_answer from the sub-answers—using a strict fallback rule, which is what prevents error propagation.","core_discovery":"RTQA’s central claim is that complex temporal questions can be answered accurately without training by turning each question into a decomposition tree, solving the tree from the leaves upward with an LLM grounded in retrieved temporal-knowledge-graph facts, and at each non-leaf node fusing two independently derived candidates: IR_answer, obtained by direct retrieval plus LLM reasoning, and child_answer, obtained by summarizing the sub-answers. The key idea is that implicit constraints such as “before Kuwait” or “last” become explicit sub-questions whose answers fill placeholders in parent questions, while the aggregator’s fallback rule stops a single wrong sub-answer from propagating to the","pith_inferences":["The same recursive decomposition-plus-fallback aggregation pattern should transfer to static multi-hop KGQA and to temporal QA over text, since those settings share the same failure mode of implicit constraints and propagating sub-errors.","Because Recall@50 is only 71.78%, retrieval, not LLM reasoning, is likely the current capacity bottleneck; a better retriever or a retrieval-verification loop should yield larger gains than stronger reasoning prompts.","The decomposer and solver are decoupled (the paper fixes the decomposer while swapping solver LLMs), so the framework invites a cheap ablation: test whether a stronger decomposer with a weaker solver outperforms the reverse, which would point future tuning at decomposition quality rather than reasoning size."],"forward_implications":["RTQA raises MultiTQ Hits@1 overall from 0.728 (TimeR4) to 0.765, with the Multiple-question category jumping from 0.335 to 0.424.","On TimelineKGQA, RTQA raises Complex-question Hits@1 from 0.009 to 0.135 and Medium from 0.092 to 0.218, while Simple drops slightly from 0.704 to 0.608.","Removing the decomposer cuts Multiple-question Hits@1 by about half (0.424 to 0.214), showing recursive decomposition is the main driver on complex questions.","Removing the answer aggregator reduces MultiTQ Hits@1 by 19.6% on Multiple questions, evidence that multi-path aggregation absorbs single-subquestion errors.","Removing fact retrieval collapses overall Hits@1 to 0.070, so external TKG grounding is a hard requirement for the framework."],"supporting_citations":[{"why":"Provides the MultiTQ benchmark, the multi-granular temporal QA dataset RTQA is evaluated on, and the MultiQA baseline and multi-granular time results it improves on.","marker":"(Chen et al., 2023)"},{"why":"Provides the TimelineKGQA benchmark and the RAG baseline whose Complex Hits@1 of 0.009 RTQA raises to 0.135.","marker":"(Sun et al., 2025)"},{"why":"TimeR4 is the strongest LLM-based baseline on MultiTQ and the source of the retrieval setup (top-50 dense facts with FAISS) RTQA extends; RTQA reports raising its Multiple Hits@1 from 0.335 to 0.424.","marker":"(Qian et al., 2024)"},{"why":"Supplies the BGE-M3 dense retriever that converts TKG quadruples and questions into embeddings and selects the top-K facts on which the whole reasoning chain depends.","marker":"(Chen et al., 2024a)"},{"why":"Decomposed prompting is the decomposition strategy RTQA adapts for temporal questions.","marker":"(Khot et al., 2023)"},{"why":"Defines the 13 temporal interval relations that ground the formal notion of temporal constraints used in the decomposition.","marker":"(Allen, 1984)"},{"why":"Supplies the tree-of-thought decomposition guidelines RTQA adapts for temporal question trees.","marker":"(Cao et al., 2023)"}],"fun_headline_variants":["Training-free recursive solver outperforms TKGQA on complex queries","Recursive question splitting improves temporal KG QA, no training","RTQA: decompose questions recursively, aggregate answers, beat SOTA","No-training LLM approach boosts complex temporal knowledge QA"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the retrieval step returns the temporal facts that support each question; if the correct fact is missing from the top-50 retrieved context, the LLM has no evidence to reason from and the framework cannot recover (ablation drops Hits@1 from 0.765 to 0.070).","fun_headline_variants_meta":{"raw":{"variants":["Training-free recursive solver outperforms TKGQA on complex queries","Recursive question splitting improves temporal KG QA, no training","RTQA: decompose questions recursively, aggregate answers, beat SOTA","No-training LLM approach boosts complex temporal knowledge QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1665,"prompt_tokens":679,"completion_tokens":986,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":926}},"tokens_in":423,"tokens_out":986,"duration_ms":9491,"temperature":1.0,"reasoning_tokens":926,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:27:48.459139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the subset of MultiTQ “Multiple” questions whose gold supporting fact is not among the top-50 retrieved facts. If RTQA answers a non-negligible fraction correctly, the recursive framework can reason past retrieval misses; if it answers near zero, the framework’s reported gains are bounded by retrieval recall, not by decomposition or aggregation.","supporting_citations":[{"cited_title":"TimelineKGQA: A Comprehensive Question-Answer Pair Generator for Temporal Knowledge Graphs","cited_arxiv_id":"2501.04343","evidence_quote":"Provides the TimelineKGQA benchmark and the RAG baseline whose Complex Hits@1 of 0.009 RTQA raises to 0.135."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TimeR4 is the strongest LLM-based baseline on MultiTQ and the source of the retrieval setup (top-50 dense facts with FAISS) RTQA extends; RTQA reports raising its Multiple Hits@1 from 0.335 to 0.424."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Decomposed prompting is the decomposition strategy RTQA adapts for temporal questions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the tree-of-thought decomposition guidelines RTQA adapts for temporal question trees."}],"review_version":1}