{"id":"340d81b3-87d9-43f4-8dfc-7d511e10c5c5","arxiv_id":"2508.21569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new human-annotated Marathi sentence similarity dataset and a fine-tuned SBERT model are released, with in-domain correlations around 0.95.","lead":"This paper introduces MahaSTS, a new human-annotated dataset of 16,860 Marathi sentence pairs labeled for semantic similarity, and a fine-tuned sentence-embedding model that scores them. It provides a benchmark resource for Marathi, one of the most widely spoken Indian languages but poorly served by existing NLP datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.96 correlations are likely inflated because the test pairs were selected and labels anchored by MahaSBERT-STS, the same model family being evaluated; external validation is required.","rationale":"The paper's central claim is that MahaSTS is a human-annotated, real-Marathi STS dataset and that MahaSBERT-STS-v2 outperforms baselines with near-0.96 correlations. The weakest point is the data construction pipeline: pairs are selected by cosine similarity from MahaSBERT-STS, and annotators use those same scores as a reference. This creates a selection-and-anchoring loop between the data and the evaluated model family. Even though the paper is transparent about the procedure, transparency does not remove the validity threat: the test set is not a random sample of natural Marathi similarity judgments, and the labels are not independent of the model being tested. The dataset and model are publicly released, which is a real contribution, and the paper gives enough detail to reproduce the pipeline; those facts support a conditional acceptance rather than rejection. However, the 0.96 numbers should be treated as in-distribution performance until an external, model-blind annotation study confirms generalization. The reader's weakest assumption identifies the same issue, and the conditional verdict remains appropriate; no additional verdict change is needed from this stress-test pass.","tokens_in":6832,"tokens_out":4490,"duration_ms":41920,"concrete_test":"Build an external Marathi STS test set without model-based selection: randomly sample 1,000 sentence pairs from L3Cube-MahaCorpus (same length filtering as §3.1, but no cosine bucketing), have at least three independent Marathi speakers assign 0–5 similarity scores without access to any model output, and compute Pearson and Spearman correlations for MahaSBERT-STS-v2 against these labels. If the correlations fall materially below 0.9600/0.9523 (e.g., by more than 0.05), the headline numbers are a construction artifact and the claims should be restricted to in-distribution performance. A complementary check is to evaluate on an independently annotated Marathi STS benchmark, such as a professionally translated STSb test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1–3.2 describe a closed data-construction loop. Candidate pairs are chosen by MahaSBERT-STS cosine similarity, with buckets split at 0.2 intervals, and human annotators were explicitly given those cosine scores 'as a reference' while labeling. The test split is drawn from this same curated pool. MahaSBERT-STS-v2 is fine-tuned from MahaSBERT, which is the same L3Cube/MahaSBERT family used to select the pairs; MahaSBERT-STS itself is referenced but not defined or evaluated in the paper. Consequently, the test distribution is conditioned on the very model family being scored: pairs that MahaSBERT-STS considers similar are over-represented in high buckets, and labels are anchored to its ranking. Fine-tuning on such data can produce high Pearson/Spearman correlations by fitting the selector's inductive bias rather than human semantic similarity. The reported 0.9600/0.9523 therefore does not establish general Marathi STS ability. Baseline comparisons in Table 3 share the same test set, so the model ranking is also confounded. The Limitations section only discusses sentence length, not this selection/anchoring bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MahaSTS, a Marathi sentence-textual-similarity dataset of 16,860 sentence pairs with continuous 0–5 similarity labels, uniformly distributed across six score buckets. The pairs are drawn from L3Cube-MahaCorpus by using MahaSBERT-STS cosine similarity to select one candidate sentence per query per bucket, with annotators using those cosine scores as a reference. The authors fine-tune MahaSBERT with CosineSimilarityLoss on an 85% train split and report Pearson 0.9600 and Spearman 0.9523 on the 10% test split, beating several multilingual and Marathi baselines. They release the dataset and the fine-tuned model for public use.","tokens_in":7200,"tokens_out":3851,"duration_ms":37687,"significance":"If the reported results are valid, MahaSTS would be a valuable low-resource resource: it is, to my knowledge, the first human-annotated Marathi STS dataset built from naturally occurring text, it is large (16,860 pairs), and it is deliberately balanced across the similarity range. The paper also ships the dataset and model publicly, which is a strength that makes external scrutiny and reuse possible. However, the significance is currently conditional. The evaluation is conducted entirely on a test split of the same dataset whose pair selection and bucket assignment were produced by MahaSBERT-STS, and whose labels were anchored to that model's cosine scores. As a result, the headline correlations and the baseline ranking in Table 3 do not currently establish that the model generalizes to Marathi sentence similarity as judged by humans in general. The core contribution is still plausible, but it needs external validation and more transparent annotation reporting.","major_comments":[{"comment":"The test set is not an independent sample of Marathi similarity judgments. Candidate pairs were selected by retrieving, for each query sentence, candidates whose MahaSBERT-STS cosine similarity fell into fixed intervals, and the annotators were given those cosine scores 'as a reference' during labeling. The 10% test split is drawn from this same curated pool. Consequently, the reported Pearson 0.9600 and Spearman 0.9523, and the relative ordering of baselines in Table 3, are measured on a distribution whose composition and bucket boundaries were generated by the same model family that is being evaluated. This makes the evaluation self-referential, and I do not think the current results establish generalization to real-world Marathi semantic similarity.","section":"Section 3.1–3.2, Table 3"},{"comment":"External validation is needed to support the central claim that MahaSTS 'enables effective training' for STS. I recommend evaluating on at least one held-out resource not produced by the MahaSBERT-STS selection pipeline, for example independently sampled sentence pairs from MahaCorpus, a Marathi translation of an existing STS benchmark, or the MahaParaphrase corpus. Without such a check, the improvement claimed in Section 5 could be largely an artifact of learning the selector's bucket structure rather than of genuinely improved semantic similarity modeling.","section":"Section 5, Results and Discussion"},{"comment":"The dataset's reliability depends on annotation quality, but the paper reports no annotator count, no inter-annotator agreement, and no detailed annotation protocol. It also does not specify how the completely dissimilar pairs in bucket 0 were generated. Since the paper's main contribution is a human-annotated, balanced dataset, these details are load-bearing; at minimum, report IAA on a subset and document the full annotation instructions, including how the reference cosine scores were presented to annotators.","section":"Section 3.2, Dataset Curation"},{"comment":"The Limitations section only discusses sentence length and complexity. It should explicitly acknowledge the selection and anchoring bias: candidate pairs were chosen by MahaSBERT-STS cosine similarity, the buckets were defined by that model's score intervals, and annotators saw those scores as references. This is important for users of the dataset to interpret its coverage and for future work to design less circular evaluation protocols.","section":"Section 6, Limitations"}],"minor_comments":[{"comment":"The construction of bucket 0 is not described beyond saying that completely dissimilar pairs were 'eventually put' into it; please specify the selection procedure for these pairs.","section":"Section 3.2"},{"comment":"The bucket boundaries (0.8, 1], (0.6, 0.8], etc. are defined on the embedding model's cosine scale, yet the final labels are on a 0–5 semantic scale; the relationship between these two scales should be clarified and justified.","section":"Section 3.2"},{"comment":"The description of MahaBERT is confusing: the text says it is a multilingual BERT fine-tuned on Marathi data, while elsewhere it is described as a Marathi monolingual model; please clarify the base architecture and training corpus.","section":"Section 4.2"},{"comment":"The text refers to models 'described in Section 3,' but the model descriptions are actually in Section 4; also, using HuggingFace model IDs in the table makes the table hard to read without the repository links.","section":"Section 5, Table 3"},{"comment":"MahaSBERT-STS is only given as a hyperlink; please provide a formal citation, model version, and a short technical description so the selection process is reproducible.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the self-referential data-construction loop: pairs selected and bucketed by MahaSBERT-STS, labels anchored to its scores, and evaluation performed on the same pool. This is fixable in a revision by adding external evaluation and annotation-quality details. The manuscript is otherwise a reasonable resource paper, and the authors' decision to release data and code is commendable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the dataset is genuinely new and useful—a 16,860-pair, human-annotated Marathi STS corpus with balanced buckets, built from native text rather than translation. That alone earns the paper a serious look. But the headline correlations should not be trusted as evidence of general Marathi STS ability.\n\nThe good: MahaSTS fills a real gap. Marathi had no human-annotated continuous STS resource; MahaParaphrase is binary, and existing multilingual STS benchmarks skip Marathi. The dataset is public, the split design is sensible, and the paper is clearly written. As a data release it does what it claims.\n\nThe soft spot is the evaluation. Section 3.1–3.2 show the data construction loop: pairs are selected by MahaSBERT-STS cosine similarity, bucketed at 0.2 intervals, and annotators were given those cosine scores as a reference while labeling. The test split comes from the same pool, and the fine-tuned model starts from MahaSBERT, the same model family. So the test distribution is conditioned on the selector's idea of similarity, and the labels are anchored to its ranking. A Pearson of 0.96 under those conditions mostly says the model fits its own selector's bias. That's not a demonstration of general Marathi sentence similarity. External validation on a test set built without the same selection model and without cosine-anchored labels is needed.\n\nAlso under-specified: annotator count, qualifications, and inter-annotator agreement. For a human-labeled dataset, that's basic reporting. The Limitations section talks about sentence length but doesn't mention the selection/anchoring loop, which is the bigger limitation.\n\nThe baseline table is still informative as a same-test-set comparison, and the pooling comparison is fine. The arbitrary bucket boundaries are the only real free parameter, and they're disclosed.\n\nWho this is for: anyone building or evaluating Marathi sentence embeddings. The dataset will be used regardless of the evaluation weakness. I'd send it to peer review with a clear request: report annotation details, validate externally or re-annotate a held-out set without cosine references, and add a limitations paragraph on the selection loop. Ship the data; treat the model numbers with caution.","headline":"MahaSTS is a valuable new Marathi STS dataset, but the reported 0.96 correlations are likely inflated by a closed data-construction and evaluation loop.","tokens_in":7590,"tokens_out":2403,"would_cite":true,"duration_ms":20789,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MahaSTS, the first human-annotated Marathi sentence-similarity dataset, and shows that fine-tuning MahaSBERT on it reaches Pearson 0.9600 and Spearman 0.9523, beating multilingual baselines.","keywords":["MahaSTS","Marathi","semantic textual similarity","Sentence-BERT","MahaSBERT","low-resource NLP","human-annotated dataset"],"falsifier":"Sample 1,000 Marathi sentence pairs at random from MahaCorpus without prefiltering by cosine similarity, have fresh annotators score them 0-5 without seeing model outputs, and compute MahaSBERT-STS-v2's Pearson and Spearman correlations against those labels; if the correlations are materially below 0.96 and 0.95, the construction pipeline was the source of the reported agreement.","tokens_in":6660,"feed_emoji":"🗣️","tokens_out":7203,"duration_ms":61869,"temperature":0.7,"pith_summary":"MahaSTS is a new human-annotated dataset for Marathi sentence similarity: 16,860 sentence pairs drawn from a large monolingual corpus, each labeled with a continuous score from 0 to 5 and arranged so that all six score buckets contain exactly 2,810 pairs. The paper's central claim is that fine-tuning the Marathi Sentence-BERT model MahaSBERT on this balanced data yields a similarity model, MahaSBERT-STS-v2, that scores Pearson 0.9600 and Spearman 0.9523 against human labels, outperforming MahaBERT, MuRIL, IndicBERT, and IndicSBERT. This matters because Marathi, despite being widely spoken, had no native STS benchmark; translated benchmarks miss cultural and linguistic nuance. The paper argues that human-curated, balanced supervision is what drives the improvement in low-resource settings.","feed_headline":"Marathi sentence similarity dataset pushes Pearson to 0.96","feed_subtitle":"Fine-tuned MahaSBERT on 16,860 native Marathi pairs beats multilingual baselines with balanced 0-5 labels.","key_machinery":"The central mechanism is the pairing of a balanced dataset-construction pipeline with Sentence-BERT fine-tuning. Sentence-BERT is a siamese BERT architecture that maps each sentence to a fixed embedding and is trained so that embedding cosine similarity matches the human label; here the model is MahaSBERT, a Marathi SBERT already trained on NLI data. The dataset pipeline selects candidate pairs from one million real Marathi sentences using MahaSBERT-STS cosine similarity, places them into six non-overlapping score buckets, and keeps exactly 2,810 pairs per bucket after human annotation. That uniform bucketing is what lets the regression training see the full 0-5 range in equal proportion, which the paper argues reduces label bias and stabilizes learned similarity scores.","core_discovery":"On its own terms, the paper establishes that a Marathi STS dataset built from real sentences rather than machine translation can support effective similarity training. MahaSTS contains 16,860 pairs uniformly spread over six similarity buckets from 0 to 5, and MahaSBERT-STS-v2 — MahaSBERT fine-tuned on the training split with CosineSimilarityLoss, MEAN pooling, two epochs, and a 1e-5 learning rate — attains Pearson 0.9600 and Spearman 0.9523 on the held-out test split. This beats every baseline in the comparison, including the same base model before STS fine-tuning and strong multilingual models. The paper also reports that MEAN pooling outperforms CLS and MAX pooling, and acknowledges a limitation on longer or more complex Marathi sentences.","pith_inferences":["Editorial inference: since candidate pairs were selected and bucketed using MahaSBERT-STS cosine similarity, with those scores also used as annotation reference, an independent sample of naturally occurring Marathi sentence pairs could yield lower correlations than 0.96/0.95; the reported numbers are conditioned on the selection pipeline.","Editorial inference: applying the same bucket-balanced annotation recipe to other low-resource languages would be a direct test of the paper's recipe, and the decisive check is whether such models improve downstream retrieval or paraphrase tasks beyond benchmark correlation.","Editorial inference: the uniform bucket distribution makes the test set much flatter than real similarity distributions, where most random pairs are dissimilar, so Spearman gains may partly reflect stratified sampling; reporting performance on an unstratified sample would clarify real-world ranking quality."],"forward_implications":["Marathi NLP gains a reusable STS benchmark and a ready fine-tuned model, so retrieval, question answering, clustering, and paraphrase detection can be built and evaluated on native Marathi text rather than translated English data.","The paper argues that the balanced six-bucket design reduces label bias and stabilizes regression training, so the resulting model is usable across the full 0-5 range, from unrelated pairs to near-equivalents.","The comparison shows task-specific fine-tuning on human-annotated data beating both general Marathi BERT and multilingual sentence encoders, strengthening the case that annotation quality, not model scale, is the bottleneck in low-resource STS.","The released dataset and model give other researchers a fixed benchmark and baseline, making future Marathi sentence-similarity results directly comparable."],"supporting_citations":[{"why":"Supplies the one-million-sentence MahaCorpus that all dataset pairs are drawn from and the MahaBERT base model used by MahaSBERT.","marker":"Joshi (2022)"},{"why":"Introduces MahaSBERT, the base model that is fine-tuned on MahaSTS, and the sequential NLI-then-STS training rationale.","marker":"Joshi et al. (2023)"},{"why":"Defines the Sentence-BERT siamese architecture and the similarity training objective (CosineSimilarityLoss) used for fine-tuning.","marker":"Reimers and Gurevych (2019)"},{"why":"Provides the BERT pretraining that underlies all evaluated encoders, including MahaBERT and MuRIL.","marker":"Devlin et al. (2019)"},{"why":"Introduces the IndicSBERT multilingual baseline that the MahaSTS-trained model is compared against.","marker":"Deode et al. (2023)"}],"fun_headline_variants":["MahaSTS: 16,860 Marathi pairs yield Pearson 0.96","Fine-tuned MahaSBERT hits 0.96 on Marathi STS","Marathi STS dataset: real sentences beat multilingual models","Balanced 0-5 labels lift Marathi similarity to 0.96"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported correlations assume that sentence pairs selected and bucketed by MahaSBERT-STS cosine similarity, then labeled with the model's scores as a reference, represent how Marathi speakers judge similarity in general; if that selection pipeline makes the test set easy or non-representative, the 0.95 numbers overstate real-world performance.","fun_headline_variants_meta":{"raw":{"variants":["MahaSTS: 16,860 Marathi pairs yield Pearson 0.96","Fine-tuned MahaSBERT hits 0.96 on Marathi STS","Marathi STS dataset: real sentences beat multilingual models","Balanced 0-5 labels lift Marathi similarity to 0.96"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2051,"prompt_tokens":900,"completion_tokens":1151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1069}},"tokens_in":516,"tokens_out":1151,"duration_ms":8312,"temperature":1.0,"reasoning_tokens":1069,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:39:07.735791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample 1,000 Marathi sentence pairs at random from MahaCorpus without prefiltering by cosine similarity, have fresh annotators score them 0-5 without seeing model outputs, and compute MahaSBERT-STS-v2's Pearson and Spearman correlations against those labels; if the correlations are materially below 0.96 and 0.95, the construction pipeline was the source of the reported agreement.","supporting_citations":[],"review_version":2}