Pith. sign in

REVIEW 3 major objections 4 minor 37 references

MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MahaParaphrase introduces 8,000 Marathi sentence pairs labeled paraphrase or non-paraphrase, with fine-tuned MahaBERT reaching 88.7% F1.

desk verdict Useful first Marathi P/NP paraphrase resource, but the gold-standard claim rests on an annotation process with zero reported stats and one checkable example that is plainly wrong. read the letter →

arxiv 2508.17444 v1 pith:ZMOLZX3T submitted 2025-08-24 cs.CL cs.LG

Suramya Jadhav , Abhay Shanbhag , Amogh Thakurdesai , Ridhima Sinare , Ananya Joshi , Raviraj Joshi This is my paper
classification cs.CLcs.LG
keywords MarathiNLPparaphrasedetectioncorpuslow-resourcelanguagesBERTfine-tuningsentenceembeddingsback-translationwordoverlap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a gap in Marathi NLP: no existing corpus for the language contained both paraphrase and non-paraphrase sentence pairs. It introduces MahaParaphrase, 8,000 pairs drawn from a large Marathi news-based corpus, with 4,000 labeled paraphrase (P) and 4,000 non-paraphrase (NP) after manual correction by four native Marathi speakers. The pairs are stratified into five word-overlap buckets, so the corpus can probe how lexical overlap relates to meaning equivalence. The authors fine-tune five BERT-style models and report that MahaBERT, a Marathi-pretrained model, achieves the best F1 at 88.7%. If the labels are as reliable as the curation process intends, this is a reusable benchmark for Marathi paraphrase detection and for studying hard cases in low-resource languages.

What carries the argument

The central mechanism is a two-channel dataset curation pipeline. Channel one computes cosine similarity between MahaSBERT embeddings of sentence pairs from a one-million-sentence Marathi corpus, assigning NP below 0.8 and P between 0.8 and 0.99, with BERTScore used as a semantic-similarity check. Channel two creates synthetic candidates by Marathi-English-Marathi back-translation and keeps only pairs whose cosine similarity falls in the same 0.8–0.99 window. Four native speakers manually corrected the resulting labels. Finally, the corpus is split into five buckets via Jaccard word overlap—intersection over union of word sets—which is what lets the dataset test lexical-overlap sensitivity.

What would settle it

Re-annotate a stratified random sample of the 8,000 pairs (say 200 per bucket) with three new native-speaker annotators working independently, and measure agreement against the published labels; if agreement is low or a large fraction of labels flip, the corpus cannot serve as a stable gold standard.

Watch

Extended reading notes

Core claim

MahaParaphrase is the first Marathi paraphrase-detection corpus with balanced P/NP labels. Its construction combines two channels: mined real sentence pairs from MahaCorpus labeled by MahaSBERT cosine-similarity thresholds, and synthetic back-translated pairs filtered by the same threshold window. Human annotators then corrected every label. The five buckets—ranging from 0–20% to 80–100% word overlap—deliberately include hard cases such as high-overlap non-paraphrases and low-overlap paraphrases. On this corpus, fine-tuned MahaBERT reaches 88.7% F1, outperforming the multilingual baselines tested.

Load-bearing premise

The corpus is only as trustworthy as the automated similarity scores that seeded the labels, because the paper reports no measure of how often or how consistently the four human annotators changed those seed labels.

Editorial extensions

If this is right

  • Marathi gains a supervised benchmark with balanced P and NP labels, so paraphrase-detection models can be trained and compared on the language.
  • The bucket structure makes it possible to evaluate models along a lexical-overlap spectrum, isolating cases where high overlap is misleading and low overlap still conveys the same meaning.
  • MahaBERT's 88.7% F1 establishes a baseline that subsequent Marathi models, including larger or generative ones, can be measured against.
  • The corpus can serve downstream tasks that rely on paraphrase, including data augmentation, question answering, semantic parsing, and style transfer.
  • The real-plus-synthetic curation approach offers a template for building similar P/NP corpora in other low-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A bucket-wise evaluation would likely show that accuracy varies with word overlap, and the hardest region is high-overlap non-paraphrases; the paper does not report per-bucket scores, so that remains an open check.
  • The pipeline's portability to other low-resource languages depends on measuring the human-correction step; without reported annotator agreement or correction counts, a similar study could silently inherit seed-label noise.
  • Because existing Marathi paraphrase resources contain only positive pairs, the NP half of this corpus is what makes discriminative paraphrase detection trainable; combining both could support generation and detection jointly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces L3Cube-MahaParaphrase, a Marathi paraphrase detection corpus of 8,000 sentence pairs (4,000 paraphrases and 4,000 non-paraphrases), constructed from the MahaCorpus using two approaches: cosine similarity thresholds with MahaSBERT and back-translation with a similarity filter. The pairs are manually verified by four native Marathi speakers and divided into five word-overlap buckets. The authors fine-tune and evaluate several BERT-based models (MuRIL, mBERT, IndicBERT, MahaBERT) and report MahaBERT as the best model with 88.7% F1. They claim this is the first Marathi paraphrase dataset containing both P and NP labels.

Significance. If the dataset is indeed high quality, it fills a real gap in low-resource Indic NLP: no existing Marathi paraphrase corpus includes both paraphrase and non-paraphrase labels, and the five-bucket structure enables analysis of paraphrase difficulty. The corpus is publicly released, which is a concrete contribution. The paper also compares several strong multilingual and Marathi-specific baselines. However, the gold-standard claim rests on a human-correction step that is not quantified, the benchmark evaluation lacks crucial reproducibility details, and the paper contains at least one internal counterexample that undermines confidence in the labels. The resource is potentially useful, but the current evidence is not yet sufficient to support the claimed quality.

major comments (3)
  1. The gold-standard claim is directly undermined by the NP example in the 80–100% bucket of Table 1: the two sentences are literally identical ('मी खरी भविष्यवाणी केली नव्हती काय?'). Identical sentences are trivially paraphrases, so this is either a labeling error or a typo in the illustration. Since this is the paper's own checkable example, it suggests the human-correction protocol described in §3.3 may not have been applied rigorously. No inter-annotator agreement, correction rate, or second-pass audit is reported. Please provide quantitative evidence of annotation quality (e.g., IAA, number/type of corrections) and correct or explain the Table 1 example.
  2. The model evaluation is not reproducible or fully interpretable as reported. The column header is 'Score', while the abstract and conclusion call it 'F1'; the paper never defines the exact metric (binary F1? macro? accuracy?). No train/validation/test split is described, no hyperparameters are given, no seeds are reported, and no error bars or multiple-run statistics are provided. The claim that MahaBERT is the best model (88.7% F1) therefore cannot be verified or compared to future work. Please specify the metric, the split protocol, and report variance across runs.
  3. There is a self-referential loop in the benchmark construction: the initial P/NP labels are produced by MahaSBERT cosine-similarity thresholds (0.8 and 0.99), and the best-performing model is MahaBERT, which is the underlying model of MahaSBERT. Human correction is the only independent check, but its reliability is not established (see above). The thresholds are also arbitrary and not justified or ablated. This does not invalidate the work, but it means the reported 88.7% F1 may partly reflect that the labels align with the same model family that created them. Please add annotation-quality statistics and, if possible, an independent validation sample.
minor comments (4)
  1. The text states the sentences were 'divided into five equally distributed buckets', but Figure 3 shows clearly uneven row counts across buckets. This is likely a wording error; either correct the claim or clarify what 'equally distributed' means (e.g., five buckets of equal range, not equal size).
  2. 'Each row in the dataset contains a sentence along with a paraphrase or non-paraphrase label' should read 'a sentence pair along with a paraphrase or non-paraphrase label'.
  3. The table header 'Score' is ambiguous; use a precise metric name (e.g., F1) consistently with the text.
  4. Several references are truncated with 'and 1 others' (e.g., Hunt et al., Kanerva et al., Khanuja et al.), likely due to the arXiv formatting; please ensure full author lists in the camera-ready version.

Circularity Check

0 steps flagged · score 2.0 of 10

No material circularity: benchmark is standard supervised evaluation on a new corpus; automated labeling is mediated by a claimed human-correction step. Main risk is missing annotation-quality evidence, not by-construction reduction.

full rationale

The derivation chain is: sample 1M MahaCorpus sentences (§3.1); label P/NP by MahaSBERT cosine similarity thresholds (§3.2.1) and by back-translation with the same 0.8/0.99 filter (§3.2.2); manually verify and correct all pairs by four native speakers (§3.3); bucket by word overlap (§3.4); fine-tune and evaluate BERT models (§6). None of these steps is equivalent to its own input by construction. The P/NP labels are not the benchmark conclusion by definition: the human-correction step is an independent—though thinly documented—intervention, and the 88.7 F1 is a train/test benchmark on those labels, not a fitted parameter renamed as a prediction. The paper repeatedly uses L3Cube's own MahaSBERT, MahaBERT, MahaCorpus, and MahaNLP artifacts, but these are public resources and standard baselines; no uniqueness theorem or ansatz is imported via self-citation, and the self-citations are not load-bearing for the logical derivation. What is genuinely missing is annotation-quality evidence: §3.3 gives no inter-annotator agreement, no correction rate, no second pass, and Table 1's 80–100% NP example contains two identical Marathi sentences ('मी खरी भविष्यवाणी केली नव्हती काय?') labelled NP, which is trivially a paraphrase. That is a data-validation red flag, not a circularity. Hence score 2: minor self-citation presence, but no by-construction reduction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central resource is built on several unvalidated domain assumptions: MahaSBERT cosine scores are a good seed for paraphrase labels, Google Translate round-trips yield paraphrase candidates, and word-overlap buckets capture difficulty. There are no externally validated benchmarks or inter-annotator metrics.

free parameters (3)
  • cosine similarity thresholds for P/NP labeling = 0.8 and 0.99
    Used in Sections 3.2.1 and 3.2.2 to label sentence pairs as P or NP before human correction; values are arbitrary and not justified.
  • bucket boundaries for word overlap = 0-20, 20-40, 40-60, 60-80, 80-100 percent
    Section 3.4 defines five buckets by word overlap percentages; these cut points are chosen by hand.
  • sample size from MahaCorpus = 1,000,000
    Random sample size in Section 3.1; no justification for the specific count.
assumptions (3)
  • domain assumption Cosine similarity between MahaSBERT sentence embeddings is a reliable seed signal for paraphrase vs non-paraphrase.
    Sections 3.2.1 and 3.2.2 rely on MahaSBERT scores to generate initial labels without validating against an external gold standard.
  • domain assumption Back-translation via Google Translate preserves meaning well enough that a round-trip pair is a valid paraphrase candidate.
    Section 3.2.2 uses back-translation without evaluating translation quality or its effect on paraphrase validity.
  • standard math The word overlap formula (Eq. 1) is an appropriate measure for bucketing paraphrase intensity.
    Section 3.4 defines buckets, but no validation that word overlap correlates with paraphrase difficulty or semantic closeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models." pith.science (2026). https://pith.science/paper/ZMOLZX3T

@misc{pith2026250817444,
  author       = {Pith},
  title        = {Pith review of: MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMOLZX3T}},
  note         = {Machine review of arXiv:2508.17444}
}
read the original abstract

Paraphrases are a vital tool to assist language understanding tasks such as question answering, style transfer, semantic parsing, and data augmentation tasks. Indic languages are complex in natural language processing (NLP) due to their rich morphological and syntactic variations, diverse scripts, and limited availability of annotated data. In this work, we present the L3Cube-MahaParaphrase Dataset, a high-quality paraphrase corpus for Marathi, a low resource Indic language, consisting of 8,000 sentence pairs, each annotated by human experts as either Paraphrase (P) or Non-paraphrase (NP). We also present the results of standard transformer-based BERT models on these datasets. The dataset and model are publicly shared at https://github.com/l3cube-pune/MarathiNLP

Figures

Figures reproduced from arXiv: 2508.17444 by the authors.

Figure 1
Figure 1. MahaParaphrase Dataset Curation Workflow. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Sentence length distribution for Paraphrase [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Bucket Wise Distribution. The values in brack [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 25 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alham Fikri Aji, Tirana Noor Fatyanosa, Radityo Eko Prasojo, Philip Arthur, Suci Fitriany, Salma Qonitah, Nadhifa Zulfa, Tomi Santoso, and Mahendra Data. 2022. https://arxiv.org/abs/2205.04651 Paracotta: Synthetic multilingual paraphrase corpora from the most diverse translation sample pair . Preprint, arXiv:2205.04651

    work page Pith review arXiv 2022

  4. [4]

    Ajwad Akil, Najrin Sultana, Abhik Bhattacharjee, and Rifat Shahriyar. 2022. Banglaparaphrase: a high-quality bangla paraphrase dataset. arXiv preprint arXiv:2210.05109

    work page Pith review arXiv 2022

  5. [5]

    M Anand Kumar, Shivkaran Singh, B Kavirajan, and KP Soman. 2016. Shared task on detecting paraphrases in indian languages (dpil): An overview. In Forum for Information Retrieval Evaluation, pages 128--140. Springer

    work page 2016

  6. [6]

    Regina Barzilay and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of the 39th annual meeting of the Association for Computational Linguistics, pages 50--57

    work page 2001

  7. [7]

    Rahul Bhagat and Eduard Hovy. 2013. What is a paraphrase? Computational linguistics, 39(3):463--472

    work page 2013

  8. [8]

    Ruisheng Cao, Su Zhu, Chenyu Yang, Chen Liu, Rao Ma, Yanbin Zhao, Lu Chen, and Kai Yu. 2020. https://doi.org/10.18653/v1/2020.acl-main.608 Unsupervised dual paraphrasing for two-stage semantic parsing . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6806--6817, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2020.acl-main.608 2020

Show all 37 references
  1. [9]

    Asang Dani and Shailesh R Sathe. 2024. A review of the marathi natural language processing. arXiv preprint arXiv:2412.15471

    2024 arXiv

  2. [10]

    Samruddhi Deode, Janhavi Gadre, Aditi Kajale, Ananya Joshi, and Raviraj Joshi. 2023. L3cube-indicsbert: A simple approach for learning cross-lingual sentence representations using multilingual bert. In Proceedings of the 37th Pacific Asia Conference on Language, Information an...

    2023

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...

    2019

  4. [12]

    Ashwini Gadag and BM Sagar. 2016. A review on different methods of paraphrasing. In 2016 International conference on electrical, electronics, communication, computer and optimization techniques (ICEECCOT), pages 188--191. IEEE

    2016

  5. [13]

    Veena Gangadharan, Deepa Gupta, L Amritha, and TA Athira. 2020. Paraphrase detection using deep neural network based word embedding techniques. In 2020 4th International Conference on Trends in Electronics and Informatics (ICOEI)(48184), pages 517--521. IEEE

    2020

  6. [14]

    Silin Gao, Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020. https://doi.org/10.18653/v1/2020.acl-main.60 Paraphrase augmented task-oriented dialog generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 639--649, Online. Associ...

    2020 doi

  7. [15]

    Ali Hejazizo. 2021. Combining variational sampling and metropolis--hastings sampling for paraphrase generation

    2021

  8. [16]

    Ethan Hunt, Ritvik Janamsetty, Chanana Kinares, Chanel Koh, Alexis Sanchez, Felix Zhan, Murat Ozdemir, Shabnam Waseem, Osman Yolcu, Binay Dahal, and 1 others. 2019. Machine learning models for paraphrase identification and its applications on plagiarism detection. In 2019 IEEE...

    2019

  9. [17]

    Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi. 2024. On limitations of llm as annotator for low resource languages. arXiv preprint arXiv:2411.17637

    2024 arXiv

  10. [18]

    Ananya Joshi, Aditi Kajale, Janhavi Gadre, Samruddhi Deode, and Raviraj Joshi. 2023. L3cube-mahasbert and hindsbert: Sentence bert models and benchmarking bert sentence representations for hindi and marathi. In Science and Information Conference, pages 1184--1199. Springer

    2023

  11. [19]

    Raviraj Joshi. 2022 a . L3cube-mahacorpus and mahabert: Marathi monolingual corpus, marathi bert language models, and resources. arXiv preprint arXiv:2202.01159

    2022 arXiv

  12. [20]

    Raviraj Joshi. 2022 b . L3cube-mahanlp: Marathi natural language processing datasets, models, and library. arXiv preprint arXiv:2205.14728

    2022 arXiv

  13. [21]

    Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul NC, Avik Bhattacharyya, Mitesh M Khapra, and Pratyush Kumar. 2020. Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages. In Findings of the associa...

    2020

  14. [22]

    Jenna Kanerva, Filip Ginter, Li-Hsin Chang, Iiro Rastas, Valtteri Skantsi, Jemina Kilpel \"a inen, Hanna-Mari Kupari, Aurora Piirto, Jenna Saarni, Maija Sev \'o n, and 1 others. 2024. Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for fi...

    2024

  15. [23]

    Nina Khairova, Anastasiia Shapovalova, Orken Mamyrbayev, Nataliia Sharonova, and Kuralay Mukhsina. 2022. Using bert model to identify sentences paraphrase in the news corpus. In COLINS, pages 38--48

    2022

  16. [24]

    Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, and 1 others. 2021. Muril: Multilingual representations for indian languages. arXiv preprint arXiv:2103.10730

    2021 arXiv

  17. [25]

    Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.55 Reformulating unsupervised style transfer as paraphrase generation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 73...

    2020 doi

  18. [26]

    D Kubal and H Palivela. 2021. Unified model for paraphrase generation and paraphrase identification

    2021

  19. [27]

    Khapra, and Pratyush Kumar

    Aman Kumar, Himani Shrotriya, Prachi Sahu, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Amogh Mishra, Mitesh M. Khapra, and Pratyush Kumar. 2022. https://arxiv.org/abs/2203.05437 Indicnlg suite: Multilingual datasets for diverse nlg tasks in indic languages

    2022 arXiv

  20. [28]

    C., Avik Bhattacharyya, Mitesh M

    Anoop Kunchukuttan, Divyanshu Kakwani, Satish Golla, Gokul N. C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. https://arxiv.org/abs/2005.00085 Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages . Preprint, arXiv:2005.00085

    2020 arXiv

  21. [29]

    Pawan Lahoti, Namita Mittal, and Girdhari Singh. 2022. A survey on nlp resources, tools, and techniques for marathi language processing. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(2):1--34

    2022

  22. [30]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. https://arxiv.org/abs/1909.11942 Albert: A lite bert for self-supervised learning of language representations . Preprint, arXiv:1909.11942

    2020 arXiv

  23. [31]

    Nitin Madnani and Bonnie J Dorr. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Computational Linguistics, 36(3):341--387

    2010

  24. [32]

    Affan Hilmy Natsir, Indriana Hidayah, and Teguh Bharata Adji. 2023. Deep learning in paraphrase generation: A systematic literature review. In 2023 IEEE 7th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), pages 118...

    2023

  25. [33]

    Hemant Palivela. 2021. Optimization of paraphrase generation and identification using language models in natural language processing. International Journal of Information Management Data Insights, 1(2):100025

    2021

  26. [34]

    Quoc Long Phan, Tran Huu Phuoc Doan, Ngoc Hieu Le, Ngoc Bao Duy Tran, and Tuong Nguyen Huynh. 2022. Vietnamese sentence paraphrase identification using sentence-bert and phobert. In International Conference on Intelligence of Things, pages 416--423. Springer

    2022

  27. [35]

    Hassan Shahmohammadi, MirHossein Dezfoulian, and Muharram Mansoorizadeh. 2021. Paraphrase detection using lstm networks and handcrafted features. Multimedia Tools and Applications, 80(4):6479--6492

    2021

  28. [36]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  29. [37]

    Jianing Zhou and Suma Bhat. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.414 Paraphrase generation: A survey of the state of the art . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5075--5086, Online and Punta Cana, Domin...

    2021 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.