{"id":"4a3c4ba5-44d1-4efd-a7f1-d1aea7dd4c66","arxiv_id":"2505.07637","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Chronocept is a benchmark that models the temporal validity of statements as skew-normal probability curves over logarithmic time, with baseline regressors predicting the curve parameters.","lead":"This paper introduces a new benchmark dataset that asks AI models to predict how long a statement stays true, outputting a continuous curve instead of a yes or no answer. It is built from synthetic sentences annotated by students, and simple neural models beat a fine-tuned BERT on the task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The released 'probability distributions' are not normalized densities: Eq. (4) is mathematically invalid, and Appendix D admits the targets are max-normalized 'proportional validity curves,' so the central claim is internally inconsistent.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test found a more specific internal inconsistency in the paper's central claim. The reader identified the same mathematical error in their rationale ('the formal definition contains a mathematical error'), but their weakest_assumption concerned external validity of the annotations. I focus on the normalization issue because it is decisive: the paper defines temporal validity as a continuous probability distribution (§3.1), yet the annotation instructions (§4.2, Appendix A) and curve-fitting appendix (Appendix D) explicitly produce a max-normalized 'proportional validity curve' that is 'no longer a true probability distribution.' Eq. (4) is mathematically invalid: the probability of validity throughout an interval is not the integral of the marginal validity probability. This is not a matter of consensus; it is a contradiction between the headline claim and the released artifact. The dataset may still be a useful resource for predicting subjective validity profiles, but the claim of being the first benchmark to model validity as a probability distribution requires either renormalized targets or a weaker claim. The external-validity concern remains relevant: even a properly normalized curve needs real-world anchoring to measure temporal truth, but that is a second-order design question. Since the fix is straightforward and the resource is otherwise plausible, I do not move the verdict.","tokens_in":23868,"tokens_out":5392,"duration_ms":52479,"concrete_test":"Download the released Chronocept dataset and baseline code; for each sample, use the target (ξ,ω,α) to evaluate the skew-normal density on the logarithmic time axis and compute its integral (e.g., scipy.integrate.quad). If the labels are probability densities, every integral should be 1; Appendix D predicts integrals equal to max(f) (or non-unity). Additionally, re-derive Eq. (4) from Eq. (3) for a two-timepoint example: take a fact valid on [a,b] with p_i(t) constant c<1 on that interval; the LHS is a number like P(T≥b), not c·(b−a), and the RHS grows with interval length. These two checks settle whether the central claim holds in the released artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that Chronocept is \"the first benchmark to model temporal validity as a continuous probability distribution over time.\" The formal definition in §3.1 does not support this. Eq. (3) defines p_i(t)=P(validity_i(t)=1), a marginal probability in [0,1], but Eq. (4) then asserts P(∀t∈[a,b], validity_i(t)=1)=∫_a^b p_i(t)dt. For a continuous-time binary process, the probability of being valid at every instant in an interval is not the integral of the marginal probability (which is at best the expected measure of valid times and can exceed 1). If p_i is instead a probability density, Eq. (3)'s notation is wrong. The annotation and post-processing make the situation worse: §4.2/Appendix A instruct annotators to plot proportions relative to the maximum and \"need not worry about making an ideal probability distribution with AUC=1\"; Appendix D then fits the skew-normal, normalizes AUC=1, and rescales by the maximum, explicitly stating the result \"no longer a true probability distribution\" and calling it a \"proportional validity curve.\" Therefore the released (ξ,ω,α) targets do not constitute a probability distribution at all; they are shape parameters of a max-normalized curve. The central novelty claim is thus internally inconsistent with the dataset's actual content. This is a correctness risk, not a stylistic choice: the benchmark can be repaired by releasing AUC-normalized densities or by reframing the claim as modeling \"validity profiles,\" but as written the contribution does not do what it says.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Chronocept, a benchmark that claims to model temporal validity of textual information as a continuous probability distribution over time. The resource consists of two datasets (Benchmark I: 1,254 atomic facts; Benchmark II: 524 multi-sentence passages) generated by GPT-o1, with eight human annotators providing temporal segmentation, axis categorization, and skew-normal curve parameters (location, scale, skewness) on a logarithmic time axis. The paper reports inter-annotator agreement (ICC 0.843/0.893), evaluates several regressors on the task of predicting curve parameters, and presents ablations showing that axis structure matters. It claims to be the first benchmark of this kind and discusses applications in RAG, fact-checking, and proactive agents.","tokens_in":24171,"tokens_out":4135,"duration_ms":38675,"significance":"If the resource is sound, Chronocept addresses a real gap: existing temporal reasoning benchmarks mostly use binary or discrete validity labels, while Chronocept offers continuous validity profiles with a transparent, publicly released annotation protocol, disclosed generation prompts, and reproducible code. The baseline experiments and ablations provide useful initial evidence that simple regressors can predict curve parameters and that axis-level structure helps. The main weakness is that the released targets are not normalized probability densities, so the central claim of modeling a 'continuous probability distribution over time' is not currently supported by the data. The paper is therefore valuable as a dataset proposal, but its central framing needs substantial revision or the data needs to be re-released.","major_comments":[{"comment":"Equation (4) asserts that the probability of validity throughout an interval equals the integral of the marginal probability p_i(t). For a binary process, the integral of the marginal probability is the expected measure of time during which the process is valid; it is not the probability that validity holds at every instant and can exceed 1. If p_i is intended to be a probability density rather than a marginal probability, then the notation in Eq. (3) is inconsistent with the density interpretation in Eq. (5). This is the formal foundation of the paper's novelty claim, so it must be corrected, for example by defining p_i as a density of a latent validity process or by explicitly abandoning the probability interpretation.","section":"Section 3.1, Eq. (4)"},{"comment":"The annotation and post-processing pipeline does not produce probability densities. Appendix A instructs annotators not to worry about AUC=1 and to plot proportions relative to the temporal point of highest probability; Appendix D then fits a skew-normal with a free scaling factor, enforces AUC=1 by trapezoidal normalization, and rescales by the maximum, explicitly stating that the result is 'no longer a true probability distribution' and calling it a 'proportional validity curve.' The released (xi, omega, alpha) targets therefore encode max-normalized shapes, not probability distributions. This directly contradicts the abstract, Section 3.1, and Section 6. The authors should either release AUC-normalized densities (with appropriate handling of long tails) or reframe Chronocept as modeling temporal validity profiles and adjust all 'probability distribution' claims accordingly.","section":"Section 4.2 and Appendix D"},{"comment":"The reported BERT MSE values of 145.8611 on Benchmark I and 68.1507 on Benchmark II are implausible given that targets are stated to be Z-score normalized (variance 1 per parameter): such MSE values would imply an R2 far below -100, yet the table reports R2 of -0.0090 and -0.1122. Either the MSE values, the R2 values, or the normalization description is erroneous. In addition, no standard errors or multiple-seed results are reported for any baseline, so the small margins between models (e.g., FFNN MSE 0.8763 vs. Bi-LSTM 0.9203 on Benchmark I) cannot be assessed for statistical reliability.","section":"Table 6, BERT row"}],"minor_comments":[{"comment":"The claim of 'substantial performance degradation' from the axis ablations is overstated. Excluding axes increases Bi-LSTM MSE from 0.9203 to 0.9625 (4.59%), and the R2 improvement of 112% is an increase from 0.0375 to 0.0797, both values close to zero. The effect appears real but modest, and the text should be toned down or supported with significance tests or confidence intervals.","section":"Section 5.4 and Tables 7, 12, 13"},{"comment":"The choice of the skew-normal distribution is validated by fitting six hand-authored scenarios (Table 10) that were designed to embody the same qualitative patterns the model is supposed to capture. This is a mild model-selection circularity; the paper should note this limitation or validate the distribution choice on independent data.","section":"Appendix D"},{"comment":"The logarithm base 1.1 is a free parameter that affects the scale of xi and omega and hence the numeric results, but the paper does not assess the sensitivity of the benchmark or baseline results to this choice. A short sensitivity analysis or at least a discussion of the impact of the base would strengthen the contribution.","section":"Appendix C"},{"comment":"The final ICC values (0.843 and 0.893) are below the warm-up threshold of ICC > 0.90 mentioned in Section 4.3. The paper should explain whether the threshold was applied only during warm-up and, if so, why the final benchmark-level agreement is lower. Also, Table 3 axis counts sum to more than the sample totals; this should be stated explicitly in the caption.","section":"Section 4.4 and Table 1"},{"comment":"There are numerous typographical and formatting issues, such as inconsistent spacing in 'T emporal' and 'T ext' in Appendix A, the phrase 'axis -removal' in Section 5.4, and the unlabeled co-occurrence matrices in Appendix B. A careful proofread is needed before publication.","section":"Appendix A and throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful dataset contribution, but the mismatch between the headline claim and the actual released normalization is substantial. I would be willing to see a revised version that either re-releases AUC-normalized densities or reframes the contribution as 'temporal validity profiles' rather than probability distributions. The BERT MSE anomaly in Table 6 must be resolved, and the authors should provide error bars or multiple-seed results. The paper is otherwise well within scope for a CS/CL venue, and the transparent appendices are a genuine strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a resource paper with a genuinely new idea—predicting continuous temporal validity curves instead of binary or discrete labels—but the headline claim is currently false. Equation (4) defines p_i(t) as a marginal probability and then integrates it as a density, which is mathematically invalid, and Appendix D explicitly says the released targets are max-normalized 'proportional validity curves,' not true probability distributions. The central claim is internally inconsistent. It is fixable, but needs fixing before the benchmark is usable as advertised.\n\nWhat is good: the task formulation is new relative to the cited literature; the dataset is public with full prompts, annotation guidelines, and axes; IAA (ICC 0.84/0.89) is decent for subjective annotation; the ablations show axis structure matters; the limitations section is honest.\n\nSoft spots, in order:\n\n1. The math. Eq. (3) sets p_i(t) in [0,1]; Eq. (4) integrates it over an interval. The integral can exceed 1 and is not the probability of staying valid everywhere in [a,b]. Appendix D then admits the fitted curves are rescaled by their maximum and are 'no longer a true probability distribution.' That makes the NLL evaluation and the 'first probability distribution benchmark' claim unsupported. Fix: release AUC-normalized densities, or reframe the contribution as 'validity profiles.'\n\n2. BERT baseline. MSE ~146 on standardized targets versus ~0.9 for everything else is not plausible; it points to a training bug. Fix it or drop it, and rewrite the 'BERT underperforms' discussion.\n\n3. No error bars. With test sets of 129 and 55 samples, the FFNN-vs-Bi-LSTM differences are likely noise. Report variance across seeds or bootstrap intervals.\n\n4. 'Outperforming classification-based approaches' is not demonstrated—there is no classification baseline in the paper. Add one or drop the claim.\n\nAlso, the labels come from eight students annotating GPT-generated sentences; there is no external anchor to real-world validity. That is acceptable for a benchmark, but it means the curves encode annotator intuition, not ground truth about fact dynamics. Say so more plainly.\n\nWho this is for: researchers in temporal reasoning, RAG, or fact-checking who want a continuous label resource. It could be useful after major revision. My recommendation: send to peer review, yes—the idea is worth engaging with—but require these fixes first. The current formalization and one baseline are not publishable.","headline":"A genuinely new benchmark idea is undermined by a core formal error: the released 'probability distributions' are max-normalized shapes, not densities, and Eq. (4) is mathematically invalid.","tokens_in":24694,"tokens_out":4895,"would_cite":false,"duration_ms":47584,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chronocept models facts as validity curves over time, not binary true/false labels, and shows simple regressors can predict the curves' shape from text.","keywords":["temporal validity","skew-normal distribution","benchmark","temporal reasoning","annotation","retrieval-augmented generation","regression","natural language processing"],"falsifier":"Build a small set of real-world statements with objectively measurable validity endpoints (e.g., a stock price being above a threshold, a sports record standing, an API version being supported), measure their true validity over time, and check whether human annotators following the Chronocept protocol produce skew-normal curves that match those measured validity functions. Strong mismatch between annotated curves and measured validity would indicate the benchmark captures annotation habits rather than temporal truth.","tokens_in":23659,"feed_emoji":"⏳","tokens_out":2676,"duration_ms":21151,"temperature":0.7,"pith_summary":"Chronocept is a new benchmark that treats a statement's temporal validity as a continuous probability distribution over time rather than a binary valid/invalid label. The paper argues that AI systems need this richer target to reason about when facts emerge, peak, and fade, and it provides two synthetic datasets — 1,254 atomic facts and 524 multi-sentence passages — annotated with the three parameters of a skew-normal curve: location, scale, and skewness. The authors show that simple models (a feedforward network on facts, a Bi-LSTM on passages) can predict these parameters from text, and that removing or shuffling the semantic axis structure degrades performance. If the benchmark is sound, it gives the field a denser label to train and evaluate temporally aware systems.","feed_headline":"Facts get validity curves, not true/false labels","feed_subtitle":"A new benchmark fits skew-normal curves to how long statements stay true, and simple models learn to predict them.","key_machinery":"The central object is the skew-normal probability density over a base-1.1 logarithmic time axis, parameterized by location ξ, scale ω, and skewness α. Annotators mark 3–5 points on a validity-versus-time graph and a skew-normal curve is automatically fit through those points; the fitted parameters become the prediction target. The second load-bearing component is the multi-axis decomposition (Main, Intention, Opinion, Hypothetical, Generic, Static, Recurrent, Negation), which structures the text so that the model can associate different validity dynamics with different semantic roles rather than collapsing them into one timeline.","core_discovery":"Chronocept introduces the first benchmark that frames temporal validity as a continuous probability density function over elapsed time. Each statement is decomposed into semantic axes and annotated by fitting a skew-normal distribution over logarithmic time, with targets (ξ, ω, α) capturing when validity peaks, how long it lasts, and how asymmetrically it rises and falls. Inter-annotator agreement reaches ICC 0.843 on atomic facts and 0.893 on passages. The paper reports that neural regressors predicting the curve parameters from text outperform classification-style baselines, with a fixed-order axis embedding contributing a 4.6 percent MSE improvement on the LSTM ablation.","pith_inferences":["A testable extension would be to replace the fitted skew-normal target with the raw 3–5 plotted points per sample and train a model to predict the curve non-parametrically, testing whether the low-dimensional parameterization is discarding information that matters for downstream ranking.","The benchmark's synthetic design makes it a useful controlled probe: because the sentences are generated, one could deliberately inject ground-truth validity dynamics into the generator (e.g., 'this fact decays in 2 hours') and measure whether predicted curves recover the injected parameters.","The unimodality restriction means the benchmark cannot yet support temporally recurring facts (seasonal, periodic, or intermittent), so a natural next step is a multi-modal extension using mixtures of skew-normals — but that would also require the annotation protocol to handle secondary peaks."],"forward_implications":["Retrieval-augmented generation could use predicted validity curves to rank documents not just by topical relevance but by whether the contained facts are still within their window of peak validity.","Fact-checking systems could flag borderline claims by checking whether the current time falls on the rising, plateau, or decaying part of the predicted curve, rather than relying on a hard expiry date.","The failure of fine-tuned BERT to beat simpler regressors on this small dataset suggests that efficient embedding-plus-regressor pipelines are a practical default for parameterized temporal targets.","The ablation results imply that explicit axis structure in the input representation is informative, so future models that reason over axes separately rather than concatenating embeddings may improve further."],"supporting_citations":[{"why":"The skew-normal distribution that is the paper's core modeling target.","marker":"(Azzalini, 1986)"},{"why":"Supports using skew-normal distributions for spatiotemporal relevance data.","marker":"(Schmidt et al., 2017)"},{"why":"Source of the multi-axis annotation scheme (MATRES) that Chronocept adapts.","marker":"(Ning et al., 2018)"},{"why":"The nearest prior benchmark (Temporal Validity Change Prediction) that Chronocept extends from discrete classes to continuous distributions.","marker":"(Wenzel and Jatowt, 2024)"},{"why":"The GPT-o1 model used to generate the synthetic dataset.","marker":"(OpenAI, 2024)"},{"why":"BERT, the embedding backbone for the baseline regressors.","marker":"(Devlin et al., 2019)"}],"fun_headline_variants":["Validity curves replace true/false for facts","Benchmark models when facts expire with skew-normal curves","AI learns to predict how long facts stay valid","Chronocept: first benchmark to time-knowledge decay","From true/false to probability curves over time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's labels rest on the assumption that a single-peaked skew-normal curve fitted through 3–5 points plotted by eight student annotators on GPT-generated sentences captures the true temporal validity of those statements, with no external real-world validity data anchoring the curves.","fun_headline_variants_meta":{"raw":{"variants":["Validity curves replace true/false for facts","Benchmark models when facts expire with skew-normal curves","AI learns to predict how long facts stay valid","Chronocept: first benchmark to time-knowledge decay","From true/false to probability curves over time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1544,"prompt_tokens":848,"completion_tokens":696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":622}},"tokens_in":464,"tokens_out":696,"duration_ms":6645,"temperature":1.0,"reasoning_tokens":622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:11:47.526166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a small set of real-world statements with objectively measurable validity endpoints (e.g., a stock price being above a threshold, a sports record standing, an API version being supported), measure their true validity over time, and check whether human annotators following the Chronocept protocol produce skew-normal curves that match those measured validity functions. Strong mismatch between annotated curves and measured validity would indicate the benchmark captures annotation habits rather than temporal truth.","supporting_citations":[],"review_version":1}