{"id":"06a40631-1f18-496f-9f42-8a2d38e6a125","arxiv_id":"2506.07248","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TF-IDF-based sentence selection lets a Marathi BERT classifier match full-article accuracy using about 40 percent of the text.","lead":"This paper tests a TF-IDF sentence ranking method to shorten long Marathi news articles before classification with a BERT-based model. It reports that using around 40 percent of each article keeps accuracy nearly unchanged while cutting inference time by about 43 percent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.33% accuracy drop is not trustworthy because the full-context baseline appears inconsistent with the same pipeline at 100% sentence selection, and no variance or validation-based selection is reported.","rationale":"The reader's weakest assumption focuses on whether the sum of within-article TF-IDF scores captures label-relevant informativeness. That is a plausible concern, but the paper's own experiments partially address it: TF-IDF ranking consistently beats first/random/last selection, which empirically validates the proxy on this dataset. The more urgent threat to the central claim is that the headline 0.33% accuracy drop is computed against a baseline that is not demonstrated to be reproducible under the same experimental protocol. Table 4's 100% rows should coincide with the full-context baseline if the pipeline is consistent, but they do not; this internal inconsistency directly undermines the accuracy-parity claim. The absence of variance estimates compounds this: even if the baseline were consistent, a 0.3-point difference on a 2,550-sample test set could easily be within noise, especially when the configuration was chosen post hoc over many grid points. These issues do not invalidate the method's efficiency advantages or its superiority over simple baselines, but they do mean the specific '0.33 percent drop' headline is not established. The reader's CONDITIONAL verdict already reflects the need for validation-based selection and repeated seeds; my concern adds a specific reproducibility check for the baseline, so the verdict should remain conditional pending that check. Hence UNCHANGED, with the load-bearing concern now being an internal comparability defect rather than the TF-IDF proxy itself.","tokens_in":10059,"tokens_out":3800,"duration_ms":41509,"concrete_test":"Run the full-document baseline and the 40% ranked-normalized condition under an identical MahaBERT-v2 fine-tuning script, preprocessing, and seed budget (e.g., 5 seeds each). First, verify that the 100% sentence-selection condition reproduces the full-context baseline within 0.1 points; if it does not, the baseline in §4.2 is not comparable. Second, report the mean and 95% bootstrap confidence interval for the accuracy difference between the 40% and 100% conditions; if the interval includes zero, the '0.33% drop' is not statistically meaningful, and the paper should reframe its claim as 'no significant degradation observed in this dataset.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 compares reduced-context accuracies to a full-context baseline of 94.706%. Yet Table 4 reports that selecting 100% of sentences (which should equal the original full document under the paper's own pipeline) yields accuracies between 94.35% and 94.78% depending on selection method, with Ranked Normalized reaching 94.78%, above the stated baseline. If the baseline were measured under the same preprocessing, tokenization, and training protocol, the 100% condition should reproduce it within noise; the observed spread of ~0.4 points indicates the baseline and the reduced runs are not directly comparable. The claimed 0.33% drop at 40% context is therefore not a controlled comparison. Additionally, no error bars, repeated seeds, or validation-based selection are reported. The 40% / λ2=0.7 configuration appears selected from a grid (Tables 2 and 4) using test-set accuracy, so the 94.39% result may reflect optimism from post hoc selection. The method's practical efficiency may still hold, but the headline accuracy-parity number is not currently demonstrable from the reported data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a TF-IDF-based sentence ranking method for long document classification, aiming to reduce input length for BERT-style models without modifying the architecture. The method ranks sentences by the sum of within-article TF-IDF scores and selects either a fixed number or a percentage of top-ranked sentences. A variant combines length-normalized TF-IDF scores with sentence length via a hyperparameter λ2. Experiments on the MahaNews LDC dataset (Marathi news articles) with MahaBERT-v2 compare ranked, first, last, and random selection at various counts and percentages. The paper claims near-identical accuracy to the full-context baseline (94.706%) with a 0.33% drop at 40% context, plus a 43% inference-latency reduction.","tokens_in":10257,"tokens_out":7744,"duration_ms":80076,"significance":"If the efficiency and accuracy-parity claims hold, the method offers a simple, model-agnostic, and practical alternative to architectural modifications for long document classification, particularly for low-resource languages like Marathi. The paper provides a broad comparison of selection strategies across multiple context budgets, and the directional ordering ranked > first > random > last is consistently reflected in the reported point estimates. The focus on a non-English dataset is a valuable contribution. However, the headline numerical claim (0.33% drop and 43% latency reduction) is not currently supported by the reported experimental protocol, because the full-context baseline is not reproduced by the same pipeline at 100% selection, the best configuration appears to be chosen post hoc from the test set, and no variance estimates are provided. These issues make the central efficiency claim unverifiable in its present form.","major_comments":[{"comment":"The full-context baseline is not reproduced by the same pipeline at 100% selection. Section 3.1 states the full-document accuracy is 94.706%, but Table 4 reports 100% selection accuracies of 94.11–94.78% across methods, with the Ranked Normalized condition (94.78%) exceeding the stated baseline. The claimed 0.33% drop at 40% context (94.39% vs. 94.706%) is therefore not a controlled comparison; using the matched 100% Ranked Normalized condition as baseline, the drop is 0.39 percentage points, and using other 100% conditions it ranges from 0.07 to 0.66 points. The authors should either use the 100% condition from the same pipeline as the baseline or provide repeated-run statistics demonstrating that 94.706% is within noise of the 100% conditions.","section":"Section 4.2 (Table 4) vs. Section 3.1"},{"comment":"The best result (40% selection, Ranked Normalized, λ2=0.7) is selected after inspecting test-set accuracies over a grid of percentages and λ2 values. The paper does not describe a validation-based configuration-selection procedure, despite a 2,548-sample validation split being mentioned in Section 3.1. Test-set-driven selection over a grid of roughly 50 configurations (10 percentages × 4 λ2 values plus non-normalized variants) makes the reported 94.39% an optimistic, post-hoc maximum. The authors should select the configuration on the validation split and report test accuracy only for that configuration, or otherwise quantify the selection bias, e.g., by reporting all grid results and applying a multiple-comparison correction.","section":"Section 4.2 (Tables 2 and 4)"},{"comment":"No standard deviations, confidence intervals, or repeated-seed experiments are reported. Differences of 0.1–0.5 percentage points (e.g., Table 4: 94.39% vs. 94.51% at 50%, or Table 1: 93.17% vs. 93.01% at 2 sentences) are used to support the ordering ranked > first > random > last and the near-parity claim. Without variance estimates, these differences may be within run-to-run noise. The authors should report means and standard deviations over at least 3–5 seeds, or provide statistical significance tests for the key comparisons, especially for the claimed 0.33% drop relative to the baseline.","section":"Section 4 (Tables 1–4)"},{"comment":"The 43% inference-latency reduction is reported without details of the measurement protocol: hardware, batch size, number of runs, and whether the TF-IDF ranking time is included or excluded. Since the ranking computation is an additional preprocessing step, its cost should be factored into the efficiency comparison. Without this information, the latency claim is not reproducible, and the practical efficiency advantage of the method is not fully demonstrated.","section":"Section 4.3 (Inference Time)"}],"minor_comments":[{"comment":"The summation in the score definition is garbled: \"Score(Si) = X tj∈Si TF-IDF(ti)\" should be \"Score(S_i) = \\sum_{t_j \\in S_i} TF-IDF(t_j)\". The same subscript error appears in the following lines.","section":"Section 3.2 (Eq. 1)"},{"comment":"The formula \"Score = (λ1 · Normalized_TF_IDF) + (λ2 · length)\" does not define \"length\" (number of tokens, characters, or words) nor the exact form of \"Normalized_TF_IDF\" (is it the total TF-IDF sum divided by length, or a per-token average?). Please specify the computation precisely.","section":"Section 3.3 (Score formula)"},{"comment":"The phrase \"only 0.33% below\" should be \"0.33 percentage points below\" to avoid ambiguity between absolute and relative differences.","section":"Section 4.3"},{"comment":"The sentence \"by reducing the context to just 40 to 50 percent ... we are still able to achieve an impressive accuracy of 94.39%\" is imprecise: 94.39% corresponds specifically to 40% with the Ranked Normalized variant and λ2=0.7; at 50% the Ranked variant gives 94.51%. Please state the exact configuration to which the claim refers.","section":"Section 4.2"},{"comment":"\"Exponential attention overhead\" should be \"quadratic attention complexity\" to match the standard description of transformer self-attention.","section":"Section 2.1"},{"comment":"The claim \"Accuracy peaks at λ2 = 0.7\" holds for the 4-sentence case but is not uniform across all sentence counts (e.g., for 2 sentences λ2=0.2 gives 92.82% vs. λ2=0.7 gives 92.73%). Rephrase to avoid overgeneralization.","section":"Table 2 and Section 4.1"},{"comment":"The phrase \"we propose a novel sentence ranking technique\" is overstated since TF-IDF sentence ranking is a well-established approach. Please temper the novelty claim or explicitly differentiate the contribution (e.g., the within-article normalization scheme and the application to Marathi long-document classification).","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The core problem is that the headline result is not demonstrably controlled: the baseline is inconsistent with the 100% selection condition, and the best configuration appears chosen from the test set. This is fixable with additional experiments (validation-set selection, repeated seeds, matched baselines), so the paper is not beyond repair. However, given the current state, an accept would not be defensible. The paper's scope is narrow (one language, one dataset), and its novelty relative to prior TF-IDF extraction methods is modest; if the authors respond well to the major revision, the contribution may be appropriate for a workshop or a specialized venue rather than a top-tier journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clearly-written empirical paper on a genuinely useful problem—cheap long-document classification for Marathi—but the headline result is not as clean as advertised. The method is TF-IDF sentence ranking, which is decades old; the only new bits are the length-weighted variant and the Marathi benchmark. The experiments are easy to follow and the directional finding (ranked beats first/random/last for percentage-based selection) is probably real.\n\nThe main problem is the baseline. The paper cites 94.706% as full-context accuracy, but Table 4 shows that selecting 100% of sentences—which should reproduce the full context—yields 94.35–94.78% depending on selection method, with ranked-normalized beating the baseline. That gap is too big to dismiss as noise, and it means the baseline was not computed under the same pipeline (likely no sentence splitting). So the claimed 0.33% drop at 40% context is not a controlled comparison. You can't claim parity with a baseline you don't actually reproduce.\n\nSecond: the 40% point and λ2=0.7 were selected after looking at test-set accuracy across a grid. No validation split, no repeated seeds, no error bars. The 0.33% number is a fitted quantity, not a robust estimate. A few extra percentage points of accuracy could disappear if you pick the configuration honestly.\n\nAlso, Table 1 shows first-sentence selection beats ranked at one sentence (90.70 vs 90.35), so \"consistently outperforms\" is overclaimed even within the paper.\n\nWhat's good: the paper is honest about its limitations in the methodology section, the scoring formula is clearly defined, and the practical efficiency story (43% latency reduction) is plausible if the accuracy parity holds. The release of code/data would help, but neither is provided.\n\nWho's it for: practitioners in low-resource NLP who need a cheap extraction baseline. It's not a methodological advance. My recommendation: send it to review—the question is legitimate and the paper is coherent—but flag that the baseline comparison and configuration selection need to be fixed before it's publishable. A serious referee would demand a same-pipeline baseline and error bars.","headline":"A plausible but overclaimed empirical study: TF-IDF sentence ranking helps for Marathi long-document classification, but the headline 0.33% accuracy drop is not trustworthy because the baseline and 100%-selection conditions don't match and the best config was chosen on the test set.","tokens_in":688,"tokens_out":732,"would_cite":false,"duration_ms":45469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TF-IDF sentence ranking cuts long-document input in half with almost no accuracy loss.","keywords":["long document classification","sentence ranking","TF-IDF","context reduction","transformer efficiency","Marathi news classification","MahaBERT","Indic languages"],"falsifier":"Use a long-document corpus in which the label-bearing sentence consists entirely of common words while an unrelated sentence contains a rare technical term; if ranked selection consistently picks the rare-term sentence and falls below first-sentence selection, the TF-IDF informativeness assumption is falsified.","tokens_in":9855,"feed_emoji":"⚡","tokens_out":8782,"duration_ms":83406,"temperature":0.7,"pith_summary":"This paper tries to establish that a lightweight TF-IDF sentence ranking can replace most of a long document's text without meaningfully hurting classification accuracy. On the MahaNews long-document dataset of Marathi news articles, keeping only the top 40 percent of ranked sentences yields 94.39 percent accuracy, a 0.33 percent drop from the 94.706 percent full-context baseline. That reduction cuts inference latency by 43 percent with MahaBERT-v2, and ranked selection outperforms first, last, and random selection at small context sizes before all methods converge. The practical point is that input reduction can happen at the data level, before the transformer sees the text, rather than by changing the model architecture.","feed_headline":"Sentence ranking halves long-document input for a 0.33% accuracy drop","feed_subtitle":"Top-ranked sentences match full-document accuracy on Marathi news while cutting inference time by 43 percent.","key_machinery":"The central object is the per-sentence TF-IDF score: each sentence $S_i$ is treated as a mini-document, term frequencies are computed within the sentence, inverse document frequencies are computed against all other sentences in the same article, and the sentence score is $\\mathrm{Score}(S_i)=\\sum_{t_j\\in S_i}\\mathrm{TF\\text{-}IDF}(t_j)$. Top-ranked sentences are then selected either as a fixed count (1 to 5 sentences) or as a percentage of the document (10 to 100 percent). A length-normalized variant blends normalized TF-IDF with sentence length using $\\mathrm{Score}=\\lambda_1\\mathrm{normTF\\text{-}IDF}+\\lambda_2\\ell$ with $\\lambda_1=1-\\lambda_2$, letting the ranking trade term uniqueness against sentence length. This scoring carries the entire argument: it converts a long document into a short ranked list before the transformer ever processes the input.","core_discovery":"The central claim is that sentence-level TF-IDF, computed by treating each sentence of an article as its own document and summing the TF-IDF weights of its terms, ranks sentences by label-relevant informativeness well enough that a standard BERT-style classifier needs only a fraction of the document. With fixed-count selection, five top-ranked sentences reach 94.19 percent accuracy, within 0.544 percent of the full-context baseline. With percentage-based selection, 40 percent coverage reaches 94.39 percent, a 0.33 percent drop from full context while inference latency falls by 43 percent. The paper also reports that ranked selection consistently orders as ranked greater than first, random, and last, and that adding sentence length through a weighted normalization term changes results only marginally in fixed-count mode. The conclusion is that a large part of a long document is redundant for classification, and a bag-of-words ranking captures enough of the informative core to preserve accuracy.","pith_inferences":["Because the scoring is a bag-of-words heuristic, sentences that carry the label through pronouns, negation, or high-frequency phrasing may rank low; a testable extension is to rerank with a semantic model and measure whether the 40 percent accuracy point moves.","The 43 percent latency figure covers transformer inference only, not the preprocessing cost of tokenizing and scoring sentences; an end-to-end comparison would show where the practical break-even sits.","The same pipeline should transfer to other languages in the IndicNews collection whenever a sentence tokenizer and a language-specific BERT exist, though the paper evaluates only Marathi.","Percentage-based selection is likely to be more robust than fixed-count selection for documents of very different lengths, since it scales the retained context with document size."],"forward_implications":["A standard transformer can classify long documents accurately from a short ranked excerpt, so data-level context reduction is a viable complement to architectural changes.","At 40 percent coverage the method reaches 94.39 percent accuracy with 43 percent lower inference latency, making it practical where response time matters.","The consistent ordering rank greater than first, random, and last shows that where sentences come from matters more than how many are used, at least at small context sizes.","All selection methods converge as coverage approaches 100 percent, so the ranking advantage is concentrated exactly where the computational savings are largest."],"supporting_citations":[{"why":"Supplies the MahaNews LDC dataset of long Marathi news articles used in all experiments.","marker":"Mittal et al. [2023]"},{"why":"Introduces the IndicNews corpus from which the Marathi long-document subset is drawn.","marker":"Aishwarya et al. [2023]"},{"why":"Provides the MahaBERT-v2 Marathi language model fine-tuned for the classification experiments.","marker":"Joshi [2022]"},{"why":"Establishes TF-IDF as a relevance measure for words in a document, grounding the sentence scoring.","marker":"Qaiser and Ali [2018]"},{"why":"Defines the BERT-style fixed-input-length constraint that motivates reducing the input context.","marker":"Devlin et al. [2018]"},{"why":"Represents prior work on efficient long-document classification with transformers that this data-level method is contrasted with.","marker":"Park et al. [2022]"},{"why":"Provides a comparative study of long-document classification approaches whose overhead motivates input reduction.","marker":"Wagh et al. [2021]"}],"fun_headline_variants":["Sentence ranking halves input size for 0.33% accuracy drop","TF-IDF sentence ranking preserves accuracy with half the input","0.33% accuracy cost for 43% faster inference with sentence ranking","Skip half the document, keep 99.7% accuracy","Sentence ranking: near-perfect accuracy with 43% faster inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes that summing a sentence's within-article TF-IDF scores measures how informative that sentence is for the article's category.","fun_headline_variants_meta":{"raw":{"variants":["Sentence ranking halves input size for 0.33% accuracy drop","TF-IDF sentence ranking preserves accuracy with half the input","0.33% accuracy cost for 43% faster inference with sentence ranking","Skip half the document, keep 99.7% accuracy","Sentence ranking: near-perfect accuracy with 43% faster inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2670,"prompt_tokens":922,"completion_tokens":1748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1658}},"tokens_in":538,"tokens_out":1748,"duration_ms":13786,"temperature":1.0,"reasoning_tokens":1658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:37:40.828931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a long-document corpus in which the label-bearing sentence consists entirely of common words while an unrelated sentence contains a rare technical term; if ranked selection consistently picks the rare-term sentence and falls below first-sentence selection, the TF-IDF informativeness assumption is falsified.","supporting_citations":[{"cited_title":"L3cube-mahanews: News-based short text and long document classification datasets in marathi","cited_arxiv_id":null,"evidence_quote":"Supplies the MahaNews LDC dataset of long Marathi news articles used in all experiments."},{"cited_title":"L3cube-indicnews: News-based short text and long document classification datasets in indic languages","cited_arxiv_id":null,"evidence_quote":"Introduces the IndicNews corpus from which the Marathi long-document subset is drawn."},{"cited_title":"Efficient classification of long documents using transformers","cited_arxiv_id":null,"evidence_quote":"Represents prior work on efficient long-document classification with transformers that this data-level method is contrasted with."},{"cited_title":"Comparative study of long document classification","cited_arxiv_id":null,"evidence_quote":"Provides a comparative study of long-document classification approaches whose overhead motivates input reduction."}],"review_version":1}