{"id":"d7246637-cf12-46c4-b103-54655b21c539","arxiv_id":"2501.04343","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TimelineKGQA is a universal, open-source generator that creates categorized temporal question-answer pairs from any TKG, released with two generated datasets and a retrieval benchmark.","lead":"This paper introduces TimelineKGQA, an open-source tool that generates question-answer pairs from any temporal knowledge graph, using a framework that sorts questions into Simple, Medium, and Complex categories. It also produces two benchmark datasets and a retrieval baseline showing that harder categories are harder to answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'any TKG / all dimensions' claim lacks a coverage guarantee: sparse or disconnected TKGs cannot yield Medium/Complex questions, and non-temporal KGs get assigned time ranges without a stated derivation.","rationale":"TimelineKGQA is a concrete, open-source contribution: the code is public, the categorization framework is clearly described, and the generated datasets are substantial. I did not find internal inconsistency in the template arithmetic. The reader's CONDITIONAL verdict is appropriate. However, the single most load-bearing gap is the universality claim. The paper's own summary commits to 'any TKG' and 'all these complexity dimensions,' but the generator's Medium and Complex branches depend on sampling two or three context facts with specific structural properties (shared entities, temporal overlap, orderability). No assumption, minimum size, or connectivity condition is stated, and no experiment covers a degenerate or sparse TKG. Module 1's promise to convert any knowledge graph into a TKG by adding time ranges is also unsubstantiated: absent a temporal source, the generator would have to assign timestamps without a stated derivation, which would sever generated questions from ground truth. These are not merely 'outside current consensus' objections; they are missing preconditions for the headline claim. The concrete test—running the shipped code on a minimal adversarial KG—would settle whether the claim should be weakened. The LLM paraphrase risk identified by the reader is real, but it is secondary relative to the missing coverage guarantee, because even deterministic templates cannot cover arbitrary inputs.","tokens_in":6648,"tokens_out":5468,"duration_ms":56708,"concrete_test":"Run TimelineKGQA on a deliberately minimal adversarial TKG: three facts involving distinct entity pairs with disjoint time intervals, so no pair satisfies an Allen relation and no entity has two facts. Also run it on a non-temporal KG with no timestamp attributes. Check whether the output contains nonzero Medium/Complex questions and whether any generated time ranges are derivable from input metadata. If Medium/Complex counts are zero or timestamps are invented, the 'any TKG / all dimensions' claim is falsified and should be weakened to 'TKGs with sufficient connected temporal structure.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 Module 2 samples |C| = 2 or 3 context facts to produce Medium/Complex questions, and Figure 2 shows that these branches require shared entities, Allen relations, union/intersection, duration, or ranking among the sampled facts. No property of the input TKG is stated that guarantees such configurations exist. For a TKG with isolated facts or with no temporally overlapping fact pairs, the generator will emit only Simple questions, so it cannot 'generate TKGQA datasets incorporating all these complexity dimensions from any given TKG.' Module 1's promise to accept 'any knowledge graph' by augmenting every fact with a time range is similarly unsupported: no rule is given for deriving t_start and t_end when the source is non-temporal, so such ranges would be ungrounded. The paper evaluates only ICEWS and CronQuestion, both temporal and dense, so the universality claim is not tested on sparse, disconnected, or non-temporal inputs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TimelineKGQA, a Python package that generates temporal knowledge graph question-answer (QA) pairs from a given TKG. It introduces a categorization framework with four dimensions: context complexity (based on the number of context facts |C|), answer focus (factual vs. temporal), temporal relations (Allen relations, set operations, duration, ranking), and temporal capabilities (TCR, TPR, TSO, TAO). The generator has four modules: graph unification, fact sampling, template-based QA generation with three complexity levels, and LLM paraphrasing. The authors evaluate the generated datasets (from ICEWS and CronQuestion) with a retrieval-augmented generation baseline, reporting that retrieval performance degrades from Simple to Complex questions, which they interpret as evidence that their complexity categorization reflects question difficulty.","tokens_in":6920,"tokens_out":3117,"duration_ms":29725,"significance":"If the claims hold, TimelineKGQA would be a valuable resource for the TKGQA community: it is an open-source tool, it attempts a systematic multi-dimensional categorization of temporal questions, and it supports answer types beyond entities and timestamps (yes/no, ranges, durations). The paper also includes a useful reclassification of CronQuestions that highlights gaps in existing datasets. However, the empirical validation of the difficulty gradient is compromised by a circular metric, and the universality claim (\"any TKG\") is not backed by coverage guarantees or tests on sparse or non-temporal graphs. The lack of verification for LLM paraphrases further weakens confidence in the correctness of the generated QA pairs. The framework itself is a reasonable starting point, but the paper needs substantial additional evidence and clarification before its central claims can be accepted.","major_comments":[{"comment":"The claim that the categorization framework \"effectively reflects question difficulty\" (Section 6) is circular. The complexity categories are defined by |C| (Section 4: Simple if |C|=1, Medium if |C|=2, Complex if |C|=3), and the evaluation metric requires all |C| context facts to be retrieved within the top |C|*K positions. Consequently, a question with larger |C| is harder to satisfy under this metric by construction, regardless of the semantic complexity of the question. The drop in Hits@1 from 0.660 (Simple) to 0.011 (Complex) is therefore an artifact of the definition of the metric and the categories, not independent evidence of a meaningful difficulty gradient. To support the claim, the authors should either evaluate with a metric that does not scale with |C| (e.g., require only one relevant fact to be retrieved, or directly measure answer correctness), or compare against a baseline where |C| is not the defining feature of the categories.","section":"Section 6, Table 5"},{"comment":"The abstract and Section 1 claim that TimelineKGQA can \"generate TKGQA datasets incorporating all these complexity dimensions from any given TKG.\" This universality claim is unsupported. Medium and Complex questions require two or three context facts that satisfy specific structural conditions: for Medium.Factual, an Allen relation between the facts; for Medium.Temporal, union/intersection, Allen relations, or duration; for Complex, ranking or multi-fact coordination. No property of the input TKG is stated that guarantees such configurations exist. For a sparse or disconnected TKG where facts do not share entities or overlap temporally, the sampler would only produce Simple questions, so the generator cannot fulfill the \"all dimensions\" promise. The authors should either restrict the claim to TKGs with sufficient density and connectivity, or provide a formal characterization of the conditions under which every complexity dimension is represented.","section":"Section 5, Module 2 and Figure 2"},{"comment":"Module 1 states that the generator accepts \"any knowledge graph\" and converts it into a TKG by augmenting each fact with a time range (t_start, t_end). However, the manuscript provides no method or heuristic for deriving these time ranges when the source KG is non-temporal. Arbitrary or ungrounded time ranges would make the generated temporal questions and their answers meaningless. The paper only evaluates on ICEWS and CronQuestion, which are already temporal and dense, so the non-temporal case is not tested. The authors must specify and justify the augmentation procedure, or explicitly restrict the tool's scope to TKGs that already carry temporal annotations.","section":"Section 5, Module 1"},{"comment":"The LLM paraphrase step applies to every generated question without any verification that the paraphrase preserves the original meaning and the answer. A paraphrase that shifts a temporal constraint or an entity reference would silently produce an invalid QA pair, corrupting the generated dataset. No human evaluation, round-trip consistency check, or automated answer validation is reported. Given that the entire dataset quality rests on this step, the authors need to either add a verification mechanism (e.g., checking that the paraphrased question, when run through the original template's answer logic, yields the same answer) or provide a sample-based human evaluation demonstrating that meaning is preserved.","section":"Section 5, Module 4"}],"minor_comments":[{"comment":"The discussion of \"suspicious works\" and \"potential academic integrity concerns\" regarding references [8] and [9] is accusatory and outside the technical scope of the paper. It should be removed or rephrased as a neutral observation about benchmark saturation.","section":"Section 2"},{"comment":"The notation defines T as the set of timestamps, but facts are described with t_start and t_end. Please clarify whether timestamps are time points or intervals, and whether the generator handles both consistently.","section":"Section 3"},{"comment":"Figure 1 contains typographical errors that distract from the content (e.g., \"Staline\", \"Rossevelt\", \"Secrety\", and \"Genernal\" if present). These should be corrected.","section":"Figure 1"},{"comment":"The modified Hits@K metric (all |C| context facts must be retrieved within the top |C|*K positions) is described only in a sentence; please provide a formal definition with equations to avoid ambiguity.","section":"Section 6"},{"comment":"The counts for \"Relation Duration\" and \"Relation Ranking\" are much smaller than other categories; please verify that these numbers are consistent with the generation logic described in Module 3 and with the totals in Table 3.","section":"Table 4"},{"comment":"The paper would benefit from a short discussion of the limitations of the retrieval-based evaluation: it measures evidence retrieval, not actual QA accuracy, so the reported numbers do not directly assess whether the generated questions are answerable or whether the answers are correct.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is promising, but the empirical validation and the universality claims need substantial rework. The circularity in the evaluation is a serious issue that the authors must address explicitly; simply acknowledging it in a rebuttal may not be enough if the paper is published as is. I would also recommend that the editor ask the authors to release the generated datasets and the code with clear documentation, as these would be the main contributions if the quality concerns are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the generator, not the categorization framework in the abstract. TimelineKGQA takes a temporal KG and emits QA pairs covering duration, ranking, union/intersection, and Allen-relation questions — types that TempQuestions, CronQuestions, and TIQ mostly miss. The code is public, the tool is reusable, and the two generated benchmarks are concrete outputs. People building private-domain temporal QA datasets will get value from this immediately.\n\nWhat the paper does well: the four-module pipeline is sensible, the Table 2 reclassification of CronQuestions is informative (it shows Medium.Temporal is entirely absent there), and the authors are upfront that existing embedding-based models cannot directly handle their time-range inputs. Shipping code and data is real evidence, and the open-source package is a step forward for the subfield.\n\nThe soft spots are proportional. The main one is the circular difficulty evaluation. Section 4 defines Simple/Medium/Complex by |C| = 1, 2, 3 context facts, and Section 6 requires all |C| facts be retrieved in the top |C|*K positions. So Hits@1 dropping from 0.66 to 0.01 as |C| grows is partly baked into the metric. That does not kill the paper, but the claim that difficulty aligns with the categorization needs a non-circular evaluation (e.g., human-judged difficulty or a metric that does not encode |C|).\n\nSecond, there is no verification that generated questions are answerable or that the LLM paraphrase preserves the original answer. A paraphrase that shifts meaning silently corrupts the dataset. A small human audit or a round-trip consistency check would fix this, but as written the benchmark quality is unverified.\n\nThird, the 'any TKG / all dimensions' claim is stronger than the described method. Module 2 samples |C| = 2 or 3 facts with shared entities and overlapping time ranges; a sparse or disconnected TKG cannot produce Medium or Complex questions. And Module 1 says any KG can be augmented with time ranges, but no rule is given for deriving t_start and t_end from a non-temporal source. The evaluated graphs (ICEWS, CronQuestions) are both temporal and dense, so the universality claim is untested.\n\nMinor: the CCS concepts line 'Do Not Use This Code' is a template leftover and should be cleaned before any camera-ready.\n\nVerdict: the paper deserves a serious referee. The tool is useful, the code is out, and the weaknesses are fixable: add a correctness audit, replace the circular metric, and state what KG properties guarantee coverage. I would bring it to a reading group and likely cite it if I worked on temporal KG QA.","headline":"A genuinely useful QA-pair generator with public code and real coverage of under-served question types, but the difficulty evaluation is partly circular and the 'any TKG' claim overreaches; it deserves referee time because the flaws are fixable.","tokens_in":7362,"tokens_out":1720,"would_cite":true,"duration_ms":18908,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generator that maps any temporal knowledge graph to categorized QA benchmarks.","keywords":["temporal knowledge graph","question answering","question-answer pair generation","temporal reasoning","dataset generation","question complexity categorization","TKGQA","knowledge graph QA"],"falsifier":"Audit a random sample of the generated QA pairs by giving human annotators only the intended context facts and asking whether the paraphrased question is answerable from them and whether the recorded answer is the unique correct one; a substantial share of mismatches would refute the claim that the generator reliably produces valid QA pairs.","tokens_in":6459,"feed_emoji":"🕰️","tokens_out":5553,"duration_ms":50080,"temperature":0.7,"pith_summary":"TimelineKGQA is a proposed Python package that turns any temporal knowledge graph into a question-answer benchmark, together with a four-way categorization of temporal questions based on timeline-context relationships. The paper's central claim is that question complexity can be systematically captured by the number of context facts, the factual-versus-temporal focus of the answer, the temporal relations involved, and four temporal capabilities: temporally constrained retrieval, timeline position retrieval, temporal semantic operations, and timeline arithmetic operations. A sympathetic reading is that this gives the field a unified way to generate custom TKGQA datasets, covering answer types from entities and timestamps to time ranges, durations, and yes/no, so that private-domain and harder benchmarks are no longer blocked by data scarcity. The paper demonstrates the generator on two knowledge sources and shows retrieval performance falls steadily from Simple to Medium to Complex questions, aligning difficulty with the proposed categorization.","feed_headline":"New tool turns any temporal knowledge graph into QA benchmarks","feed_subtitle":"Questions are categorized by complexity and answer type, covering timestamps, durations, ranges, and yes/no.","key_machinery":"The load-bearing object is the timeline-context categorization, which treats time as a homogeneous line and classifies questions by |C| (number of context facts), answer focus, temporal relation type, and required temporal capability. The four capabilities are TCR (retrieve facts under a given time constraint), TPR (retrieve the time range of a given fact), TSO (compute a new time range from a temporal signal word), and TAO (apply set operations, Allen's thirteen relations, duration calculation, or ranking to time ranges). This categorization directly drives Generator Module 3's branching logic, so the framework is not just a taxonomy but the control structure of the generator.","core_discovery":"The discovery is that a temporal question's complexity can be decomposed along four independent dimensions, and that every combination of these dimensions can be materialized as concrete QA pairs from any TKG. The generator first unifies the input graph by giving every fact a time range, samples one, two, or three context facts, then constructs questions according to the category path: simple factual or temporal questions, medium questions needing temporal semantic operations or timeline arithmetic operations, and complex questions adding ordinal ranking. Recategorizing CronQuestions under the framework exposes what is missing from existing benchmarks, notably the entire Medium.Temporal category, ranking beyond first/last, duration comparisons, and time range inference. The paper claims this is the first universal generator that incorporates all identified complexity dimensions.","pith_inferences":["A testable extension is to audit the LLM paraphrase step: no verification is reported, so generated datasets may contain questions whose paraphrase changes the answer; measuring this drift rate would show how much of the pipeline's validity rests on the paraphraser.","The generator's sampling prioritizes temporally proximate facts and frequent entities, which likely oversamples mainstream entity pairs and undersamples rare or long-tail facts in generated benchmarks.","The same categorization could serve as an evaluation metric for any TKGQA system, reporting accuracy per capability (TCR, TPR, TSO, TAO) rather than one global score, which would localize which reasoning skill a model lacks.","Because the generator is template-driven before paraphrase, it may produce answer bias toward the sampled facts' ordering; future work could randomize fact order and check answer invariance."],"forward_implications":["Any TKG, including private-domain graphs, can be turned into a TKGQA benchmark without manual annotation, enabling domain-specific fine-tuning.","Generated datasets can include question types that existing benchmarks lack, such as temporal aggregation, duration comparison, time range inference, and ordinal ranking.","Because answer types include time ranges, durations, and yes/no, retrieval systems must move beyond timestamp-or-entity outputs.","Applying the categorization to existing datasets reveals coverage gaps, so the framework doubles as an audit tool for benchmark completeness.","If difficulty aligns with context count, the generator can produce controllable difficulty curricula by mixing Simple, Medium, and Complex questions."],"supporting_citations":[{"why":"Supplies CronQuestions, the largest existing TKGQA dataset, used here both as a source KG for one generated benchmark and as the target of the recategorization that reveals missing categories.","marker":"[7]"},{"why":"Introduces TempQuestions, the early benchmark whose ordinal and implicit categories motivate the complexity dimensions.","marker":"[2]"},{"why":"Complex-CronQuestions provides a prior effort on complex temporal questions with first/last ordering that the framework extends to ranking and duration operations.","marker":"[5]"},{"why":"TIQ defines implicit time constraints, used in the comparison table to show what existing datasets cover or omit.","marker":"[4]"},{"why":"Reports very high Hits@1 on CronQuestions, cited as evidence that existing datasets no longer differentiate TKGQA methods.","marker":"[6]"}],"fun_headline_variants":["Turn any temporal knowledge graph into QA pairs","Universal TKG QA generator with complexity categories","Generate QA benchmarks from any temporal graph","TimelineKGQA: categorize and generate TKG questions","Four-axis TKG question generator for any graph"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes the sampled context facts are sufficient to answer each generated question and that LLM paraphrasing preserves both the intended meaning and the recorded answer, and neither assumption is verified in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Turn any temporal knowledge graph into QA pairs","Universal TKG QA generator with complexity categories","Generate QA benchmarks from any temporal graph","TimelineKGQA: categorize and generate TKG questions","Four-axis TKG question generator for any graph"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1673,"prompt_tokens":759,"completion_tokens":914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":375,"completion_tokens_details":{"reasoning_tokens":845}},"tokens_in":375,"tokens_out":914,"duration_ms":8950,"temperature":1.0,"reasoning_tokens":845,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:35:33.543277+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of the generated QA pairs by giving human annotators only the intended context facts and asking whether the paraphrased question is answerable from them and whether the recorded answer is the unique correct one; a substantial share of mismatches would refute the claim that the generator reliably produces valid QA pairs.","supporting_citations":[],"review_version":1}