{"id":"0e705bd7-3519-4880-8ac1-dfc0af4804d8","arxiv_id":"2607.04088","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Temporalized full-text BM25 is the strongest LongEval-Sci 2026 Task 1 submission, while uncalibrated temporal overlays and citation features remain fragile or unproven.","lead":"Full-text BM25 with temporal reranking was the strongest LongEval-Sci 2026 Task 1 submission, beating title+abstract BM25 and dense baselines across three collection snapshots. The paper also shows that uncalibrated temporal overlays can destroy top-rank quality and that citation features added no official gain.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Official temporal gains may reflect an underspecified submission pipeline rather than the temporal features themselves.","rationale":"The paper is a careful CLEF participant note: full-text BM25 as foundation is well supported by both official and development tables, and the authors already flag calibration failures and missing citation ablations. The load-bearing risk is not that the reported ARP numbers are false, but that the interpretive step from those numbers to 'temporal integration improves longitudinal effectiveness' is under-specified given the catastrophic local sparse overlays. That is exactly the reader's weakest_assumption. A no-op vs calibrated reproduction of the official temporal path is the single check that would settle whether the concern lands. No stronger internal inconsistency is present; significance tests and code packaging would help but are secondary. Verdict remains CONDITIONAL with high confidence in the reported ranking facts and medium confidence in the causal temporal claim.","tokens_in":15123,"tokens_out":570,"duration_ms":6392,"concrete_test":"Reproduce the official custom-lexical-FT-T run from the same FT BM25 first-stage candidates with a no-op temporal overlay (β=0, identity rescoring) and with the paper's stated temporal formula under a small grid of α/β and head depths; if no-op matches plain FT BM25 and only a narrow calibrated setting recovers Table 5's ARP/RC gains, the causal claim holds; if gains appear without temporal features or only under undocumented settings, the claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that temporal integration improves longitudinal effectiveness when applied to full-text BM25 (Table 5: ARP 0.285/0.267/0.180; S3 RC 0.368 vs pivot 0.481). That causal reading is the softest point. The paper's own local temporal overlays on FT BM25 collapse nDCG@10 from 0.3302 to 0.0088 (Table 7), while the official FT BM25+temporal run improves over plain FT BM25. Sections 4.4 and 4.6 describe a heuristic intent-weighted recency/foundation/novelty overlay with query-level normalization, but do not fully specify the official submission's α/β weights, candidate-head depth, intent classifier, half-lives, or whether the submitted path is the same overlay as the failing local one. Without that equivalence, the official win is consistent with better calibration, different candidate handling, or other unreported pipeline differences rather than with temporal evidence as such. Citation adds no official gain (exact match to temporal-only), so it does not rescue the causal story.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This CLEF 2026 Working Notes paper reports official LongEval-Sci Task 1 DCTR results and snapshot-1 development diagnostics for sparse, dense, expansion, reranking, fusion, and temporal/citation systems. The central empirical claim is that full-text BM25 is the strongest foundation and that temporal integration on that foundation yields the best official ARP across three snapshots (0.285/0.267/0.180 nDCG@10) while reducing snapshot-3 relative change from 0.481 (BM25 pivot) to 0.368; citation adds no aggregate gain. Internal diagnostics show FT BM25 strongest on nDCG@10/MAP, RRF strongest on deep recall, and some uncalibrated local temporal overlays collapsing top-rank quality. A separate qualitative weekly update-monitoring policy based on ingestion velocity and stale-coverage drift is also proposed.","tokens_in":15576,"tokens_out":1446,"duration_ms":15907,"significance":"If the official temporal full-text gains are reproducible and attributable to temporal evidence rather than unreported pipeline differences, the paper provides a useful, practice-oriented finding for scientific longitudinal IR: full-text lexical retrieval plus carefully calibrated temporal reranking can improve both absolute effectiveness and robustness under collection growth. Strengths include clear model-name alignment (Table 1), explicit separation of official vs development protocols, honest reporting of failed local overlays (Table 7), and appropriately weak claims about citation. The cascade design (first-stage vs overlay) and RRF deep-recall results are reusable baselines for participants. The update-monitoring section is a secondary systems contribution and is correctly framed as qualitative maintenance analysis rather than a ranking claim.","major_comments":[{"comment":"§5.1 Table 5 vs §5.3 Table 7: the central causal claim that temporal integration improves longitudinal effectiveness on full-text BM25 is under-supported. Official FT BM25+temporal improves ARP over FT BM25 (0.285 vs 0.275 on S1; 0.180 vs 0.153 on S3), but the local FT BM25+temporal overlay collapses nDCG@10 from 0.3302 to 0.0088. Sections 4.4 and 4.6 describe intent-weighted recency/foundation/novelty features and S'=αŜ_base+β(q)T̂+γ(q)Ĉ, yet do not specify the official submission’s α/β, candidate-head depth, half-lives h_f/h_a, intent-classifier rules, or whether the submitted path is the same overlay as the failing local one. Without an equivalence statement, no-op sanity check, and controlled ablation that isolates temporal features under the official pipeline, the official win is consistent with better calibration or other unreported differences rather than temporal evidence as such","section":"§5.1 Table 5; §5.3 Table 7; §4.4–4.6"},{"comment":"§5.1 and §8: key official comparisons (FT BM25+temporal vs FT BM25 vs BM25 pivot on ARP/RC/DRI) are reported without paired significance tests or confidence intervals. For a longitudinal claim that temporal full-text is strongest and more robust (S3 RC 0.368 vs 0.481), at least query-level paired tests on nDCG@10 (or bootstrap CIs) across the three snapshots are needed; the authors themselves list this as future work, but it is load-bearing for the ranking conclusions as currently stated.","section":"§5.1; §8"},{"comment":"§4.5–4.6 and Table 5: citation evidence is reported as matching temporal-only exactly in the official summary, yet there is no citation-only ablation, no citation-coverage/timestamp audit, and no quantification of zero-filled candidates. The conclusion that citation “still requires cleaner ablation and calibration” is fair, but the manuscript still presents FT BM25+temporal+citation as a top official system without establishing that citation features were active or informative. Either demote the citation system to a non-claim control or add coverage statistics and a citation-only run under the same protocol.","section":"§4.5–4.6; Table 5"}],"minor_comments":[{"comment":"Table 1 / model names: submitted identifiers (custom-lexical-FT-T, custom-lexicalft-tc, delete-me) are inconsistent in hyphenation and casing; a single canonical naming scheme in code, tables, and prose would aid reproducibility.","section":"Table 1"},{"comment":"§3.1 RC formula is written as RC_{1→t}=(ARP_1−ARP_t)/ARP_1; ensure the same sign convention is used consistently in Table 9 (where negative RC is improvement) and in prose about “lower RC means less degradation.”","section":"§3.1; Table 9"},{"comment":"§5.4–5.5 monthly protocol is carefully caveated, but Table 8/9 still risk being read as official longitudinal evidence. Consider moving monthly pivot-relative metrics fully to an appendix or labeling every monthly table “internal diagnostic only.”","section":"§5.4–5.5"},{"comment":"§6 update policy thresholds (3.0× velocity, staleness≥0.80, coverage_gap≥0.03) are free parameters with no sensitivity analysis; a short sensitivity note would strengthen the qualitative monitoring claim without overclaiming ranking benefit.","section":"§6.1"},{"comment":"§8 notes that complete software and run artifacts are not yet packaged; for Working Notes reproducibility, releasing run files and evaluation scripts should be prioritized before camera-ready.","section":"§8"},{"comment":"Minor wording: Abstract and §1 say “Submitted and Diagnostic Analysis”; ensure title/abstract consistently reflect both official submissions and diagnostics. Also fix “apublication-window” spacing in §3.","section":"Abstract; §3"}],"recommendation":"major_revision","confidential_remarks":"Fit is appropriate for CLEF Working Notes / longitudinal IR workshop venues; for a top archival IR journal the novelty bar would be higher and the underspecified temporal pipeline would be a harder block. The authors are admirably honest about local overlay failures—this should be preserved, not sanded away. Main editorial risk is over-reading Table 5 as a clean causal win for temporal features."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is the official result: full-text BM25 with temporal integration is their strongest Task 1 submission (ARP 0.285/0.267/0.180 nDCG@10), and it cuts snapshot-3 relative change versus the BM25 pivot. The second thing is that they report their own local temporal overlays on sparse BM25 collapsing nDCG@10 from ~0.33 to ~0.009. That honesty is rare and useful.\n\nWhat is actually new is not the components—BM25, RRF, RM3, MiniLM rerank, recency/foundation features, OpenCitations-style signals—but the shared-task comparison on LongEval-Sci 2026 plus the diagnostic split between official wins and failed local overlays. Full-text lexical retrieval as the foundation is the cleanest practical finding. RRF as a deep-recall generator (R@1000 ≈ 0.97) is well supported. They correctly treat citation as inconclusive (exact match to temporal-only) and the weekly velocity/staleness monitor as qualitative maintenance, not a ranking claim.\n\nThe soft spot is real but bounded. The stress-test concern lands: sections 4.4–4.6 describe a heuristic intent-weighted overlay, yet the official FT BM25+temporal path is not shown to be the same calibrated object as the failing local one. Without α/β, head depth, half-lives, and equivalence checks, “temporal integration improves longitudinal effectiveness” is consistent with better calibration or unreported pipeline differences, not only with temporal evidence as such. Missing significance tests, unaudited citation coverage, and no public runs/code are ordinary workshop gaps, not load-bearing fraud. Free parameters (half-lives, intent weights, update thresholds) are disclosed enough to be tunable, not hidden.\n\nMath and metrics are standard IR evaluation; DCTR labels and ARP/RC/DRI are organizer constructs. Citations are appropriate for longitudinal IR, fusion, temporal ranking, and dynamic indexing. No circular redefinition of the target.\n\nThis is for people building scientific search under collection growth and for LongEval/CLEF participants who want practice guidance plus failure modes. It is not a theory paper. I would send it to peer review as a working-notes systems paper; expect revision on pipeline equivalence and ablations, not desk rejection. Engage if you care about longitudinal scientific retrieval; skip if you only want new methods.","headline":"Honest LongEval-Sci note: full-text BM25 plus temporal is their best official run, but the local overlay collapse undercuts a clean causal claim.","tokens_in":16125,"tokens_out":600,"would_cite":true,"duration_ms":9778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Full-text BM25 with temporal integration is the strongest submitted strategy for scientific search as the collection grows.","keywords":["LongEval-Sci","scientific retrieval","full-text BM25","temporal reranking","citation reranking","reciprocal rank fusion","longitudinal IR","index maintenance"],"falsifier":"A controlled ablation that freezes the first-stage full-text BM25 candidates and score normalization, then turns only the temporal weights on or off under the same official snapshot protocol: if ARP and RC become identical, the temporal-integration claim fails.","tokens_in":16027,"feed_emoji":"📚","tokens_out":678,"duration_ms":6175,"temperature":0.7,"pith_summary":"This paper studies scientific document search under collection change: systems must work well now and stay usable as papers accumulate. On official LongEval-Sci Task 1 DCTR evaluation, full-text BM25 with a temporal overlay (and the same model plus citations) posts the best average nDCG@10 on all three snapshots and cuts late-snapshot degradation relative to the title-abstract BM25 pivot. Snapshot-1 development diagnostics show why that foundation works: full-text BM25 is the strongest single retriever on top-rank metrics, while reciprocal rank fusion of full-text and dense runs maximizes deep recall. The same diagnostics show that poorly calibrated temporal or citation overlays can destroy top-rank quality even when tail recall is preserved. The practical claim is therefore conditional: full-text lexical retrieval is the strongest base, temporal evidence helps when carefully applied to that base, and citation signals still need cleaner ablations. Separately, the authors sketch a weekly maintenance policy that triggers index refreshes from ingestion velocity and stale-coverage drift rather than from ranking scores alone.","feed_headline":"Full-text plus time beats official science-search baselines","feed_subtitle":"Temporal full-text BM25 leads all three LongEval-Sci snapshots and cuts late degradation.","key_machinery":"A retrieval cascade: first-stage full-text BM25 (or RRF over full-text and dense runs), then optional temporal and citation reranking overlays that rescore only the candidate head using publishedDate-based recency/foundation features and OpenCitations-derived citation signals, without building a new index.","core_discovery":"In official three-snapshot DCTR evaluation, FT BM25+temporal and FT BM25+temporal+citation achieve the best ARP on every snapshot (0.285, 0.267, 0.180 nDCG@10) and reduce snapshot-3 relative change from 0.481 for the BM25 pivot to 0.368, while internal diagnostics establish full-text BM25 as the strongest single foundation and show that uncalibrated temporal overlays can collapse top-rank quality.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Temporal full-text BM25 tops all three LongEval-Sci snapshots","FT BM25+temporal leads official DCTR and cuts late drift","Full-text with time beats BM25 pivot on science retrieval","Temporal full-text reduces snapshot-3 relative change to 0.368","Full-text foundation plus temporal beats official baselines"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The official temporal gains are caused by well-behaved temporal evidence rather than unreported differences in calibration or candidate handling, even though the paper's own local temporal overlays on sparse BM25 collapse top-rank quality.","fun_headline_variants_meta":{"raw":{"variants":["Temporal full-text BM25 tops all three LongEval-Sci snapshots","FT BM25+temporal leads official DCTR and cuts late drift","Full-text with time beats BM25 pivot on science retrieval","Temporal full-text reduces snapshot-3 relative change to 0.368","Full-text foundation plus temporal beats official baselines"]},"model":"grok-4.5","effort":"low","cost_usd":0.003034,"raw_usage":{"total_tokens":1171,"prompt_tokens":902,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":30340000,"prompt_tokens_details":{"text_tokens":902,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":197,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":902,"tokens_out":72,"duration_ms":7675,"temperature":1.0,"reasoning_tokens":197,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T21:46:35.836975+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A controlled ablation that freezes the first-stage full-text BM25 candidates and score normalization, then turns only the temporal weights on or off under the same official snapshot protocol: if ARP and RC become identical, the temporal-integration claim fails.","supporting_citations":[],"review_version":1}