Pith. sign in

REVIEW 3 major objections 4 minor 14 references

UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces UrBLiMP, a 5,696-pair Urdu benchmark on which the best LLM scores 94.73% while human validation reached 96.10% agreement.

desk verdict A genuinely useful Urdu minimal-pair benchmark, but the reliability claim rests on human validation of only about a fifth of the pairs; both the sampling gap and the abstract's mislabeled agreement are fixable. read the letter →

arxiv 2508.01006 v1 pith:H2QXIR62 submitted 2025-08-01 cs.CL

Farah Adeeba , Brian Dillon , Hassan Sajjad , Rajesh Bhatt This is my paper · ORCID
classification cs.CL
keywords Urduminimalpairsgrammaticalacceptabilitylargelanguagemodelssyntacticcompetencelow-resourcelanguagesbenchmarkperplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UrBLiMP is a new diagnostic benchmark for Urdu: 5,696 minimal pairs, each consisting of two sentences that differ in one grammatical property with only one acceptable form, covering ten core syntactic phenomena such as ergativity, honorifics, oblique case marking, and subject-verb agreement. The paper argues the benchmark is trustworthy because native speakers agreed with the intended judgments 96.10% of the time, with a Fleiss' kappa of 0.89. Evaluating twenty multilingual LLMs, the best average accuracy is 94.73%, achieved by LLaMA-3-70B, which is statistically comparable to several other top models, and no model wins every phenomenon. The point of the benchmark is to expose which parts of Urdu grammar models have actually learned, and the results show strong performance on local aspect and agreement contrasts but clear weaknesses on long-distance agreement and oblique morphology. This matters for evaluating and improving LLMs in low-resource, morphologically rich languages.

What carries the argument

The central machinery is the minimal-pair forced-choice task. Each of the 5,696 items is a grammatical sentence paired with a minimally different ungrammatical counterpart generated by a rule-based transformation, either from Urdu Treebank patterns or from regular-expression matches over a 735-million-token Urdu corpus; a model is scored correct when it assigns lower (pseudo-)perplexity to the acceptable sentence. Because the two sentences differ in exactly one syntactic property, accuracy on the benchmark isolates knowledge of that property rather than topic or surface frequency.

What would settle it

Take a random sample of the minimal pairs that the 17 annotators never saw, roughly four-fifths of the 5,696 pairs, and have new native Urdu speakers judge them; if their agreement with the benchmark labels is substantially below the reported 96.10%, or if the inter-annotator kappa falls well below 0.89, the reliability claim and the model rankings built on it would not transfer to the full dataset.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a carefully controlled minimal-pair benchmark can separate grammatical competence from general language ability in Urdu, and that current multilingual LLMs have uneven grammatical competence: humans validate the pairs at 96.10% agreement, the best model (LLaMA-3-70B) reaches 94.73% average accuracy, but that score is statistically tied with Gemma-3-27B-PT, Alif-1.0-8B-Instruct, and Gemma-3-12B-PT. The pattern across phenomena matters: aspect agreement is nearly trivial for all models, while long-distance gender agreement, oblique case on adjectives and singular masculine nouns, and several ergative sub-constructions remain hard even for the best models.

Load-bearing premise

The entire benchmark's reliability rests on the assumption that the roughly one-fifth of minimal pairs checked by human annotators represents the quality of the remaining pairs generated by the same rules.

Editorial extensions

If this is right

  • UrBLiMP can be used as a diagnostic to find where any multilingual model's Urdu grammar fails, rather than relying on aggregate task scores.
  • The near-ceiling performance on aspect agreement and the sharp drops on long-distance agreement indicate which constructions pretraining data make learnable for current architectures.
  • The finding that instruction-tuned Gemma variants underperform their pretrained counterparts on syntax suggests that instruction tuning can trade away grammatical generalization in smaller models.
  • Continued pretraining on Urdu improves average syntactic accuracy, arguing for language-specific continual pretraining for low-resource languages.
  • Among the best models, size is not decisive; Gemma-3-12B-PT, Gemma-3-27B-PT, LLaMA-3-70B, and Alif-8B are statistically indistinguishable, so performance saturates near the 12B scale for this type of probe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because only about a fifth of the pairs were human-checked, the stated 96.10% reliability is a claim about the generation rules as much as the data; a random re-validation of unvalidated pairs would test that directly.
  • Inference: the same template-and-regex construction pipeline could be applied to other Indo-Aryan languages sharing Urdu's ergative, dative, and oblique morphology, producing directly comparable benchmarks.
  • Inference: the paper's perplexity-based forced-choice test measures recognition, not production; a model could pass UrBLiMP and still generate ungrammatical Urdu, so generation-based follow-ups would give a fuller picture.
  • Inference: the instruction-tuning penalty seen in Gemma models is a testable hypothesis about optimization, not a language-specific fact; evaluating additional Urdu-instruction-tuned checkpoints would show whether the trade-off generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. UrBLiMP introduces a 5,696-pair Urdu benchmark of linguistic minimal pairs covering ten phenomena organized into 19 paradigms, generated from the Urdu Treebank and a roughly 735M-token in-house corpus via pattern-based extraction and rule-based transformations. The paper reports a human validation accuracy of 96.10% with Fleiss' kappa of 0.89, and evaluates twenty multilingual LLMs using sentence-level (pseudo-)perplexity forced choice, finding that LLaMA-3-70B achieves the highest average accuracy (94.73%) while being statistically comparable to several other top models. The central claims are that the benchmark is reliable and that current models vary substantially in Urdu syntactic competence.

Significance. If the reliability claim holds, UrBLiMP fills a real gap: it is a substantially larger and more diverse Urdu minimal-pair resource than the existing Urdu component of MultiBLiMP, with linguistically motivated coverage of ergativity, dative object marking, honorifics, obliqueness, and agreement. The paper's strengths are its treebank/corpus grounding, the breadth of the model evaluation (20 models, multiple families and sizes), the use of standard BLiMP-style methodology, the statistical comparisons among top models, and the stated plan to release the dataset and code. There are no fitted parameters or circular derivations; the main risk to the benchmark's value is the incomplete reporting of the human-validation coverage, which directly affects the reliability claim.

major comments (3)
  1. [Section 2.3, Human Evaluation] The paper does not report how many distinct minimal pairs were human-validated. The stated protocol—17 annotators, 'approximately 190 pairs' each, and 'at least three different evaluators' per pair—caps the number of unique validated pairs at roughly floor(17×190/3) ≈ 1,077, i.e., about 19% of the 5,696 pairs. The statement that each annotator covered 'about 10 pairs from each linguistic paradigm' does not specify how those pairs were sampled from each paradigm's full set, and the appendix Table 9 gives no validation N per paradigm. Since the model accuracies in Table 4 are computed over the full 5,696 pairs, the 96.10% human accuracy and Fleiss' kappa of 0.89 currently support only the validated subset. The authors should disclose the sampling protocol, report unique validated counts per paradigm, and either validate more pairs or qualify the reliability claim accordingly.
  2. [Table 4 and Section 2.3] The 'Human' row in Table 4 is compared directly with model accuracies, but the human numbers necessarily come from the validation subset (about 1,000 pairs) while all model numbers are for the full 5,696 pairs. If the human row is intended as a benchmark ceiling, it should be computed on the same sentence pairs as the models and reported with the number of pairs and a confidence interval. Relatedly, the abstract describes 96.10% as 'inter-annotator agreement,' whereas Section 2.3 defines it as raw human accuracy and reports Fleiss' kappa (0.89) separately; the wording should be corrected.
  3. [Section 2.2.2, Minimal Pairs Generation] The manual review described in this section is applied to extracted source sentences before the transformation step, not explicitly to every final generated pair. Because the transformations are rule- or regex-based, any systematic error in a rule (for example, an incorrect morphological form after replacement) would propagate through all pairs produced by that rule and would not be captured by the small validation sample. The paper should either describe an automated verification step applied to all 5,696 pairs or provide per-rule audit evidence that the unvalidated pairs have the same quality as the validated ones.
minor comments (4)
  1. [Appendix F, Table 10] The table claims to give an example from each of the 19 paradigms but contains only 14 rows; all five Oblique paradigms (Adjective, Plural, Pronoun, Noun.SG.M, Verb) are missing. The N values shown sum to 4,289, not 5,696; please complete the table.
  2. [Section 2.1 / Table 2] The text describes ten phenomena but Table 2 appears to list only nine rows, with Obliqueness absent despite being described in Section 2.1; the table should be checked and completed.
  3. [Section 4.1] The claim that models 'struggle particularly with long-distance agreement' is supported only by a single illustrative example and a comparison figure; no distance-controlled quantitative analysis is presented. Please either temper the claim or provide distance-binned results.
  4. [Tables 4 and 10] There are several typos in table headers and captions: 'order vartion' in Table 4, 'average accross' in the Table 4 caption, 'T uned' in the same caption, and 'this' appearing in the Honorific row of Table 10.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UrBLiMP's benchmark construction and model evaluation are self-contained and externally validated.

full rationale

UrBLiMP is a benchmark-construction and model-evaluation paper rather than a derivation-based study. It contains no fitted parameters, no predicted quantities derived from fitted values, and no uniqueness or existence theorems imported from the authors' prior work. The central reliability claim rests on external native-speaker judgments reported in Section 2.3 and Table 9, and the model-comparison claims rest on perplexity-based accuracies computed over the published benchmark in Table 4. The only overlaps with the authors' own prior work are methodological citations, most notably "The dataset was constructed following the SLING framework" (Section 5), citing Song et al. (2022), which includes a co-author of the present paper; this citation is contextual and not load-bearing for correctness. The human-validation protocol described in Section 2.3 samples a subset of pairs, and the possible non-representativeness of that sample is a robustness or coverage concern, not circularity, because the human judgments are independent of the automated transformation rules used to generate the remaining pairs. The Limitations section explicitly acknowledges corpus-size and phenomenon-coverage constraints, further confirming that the paper's claims are benchmark-specific and externally checkable. No step in the paper reduces, by construction or by self-citation, to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters or new theoretical entities. Its claims rest on the established minimal-pair methodology, the accuracy of the Urdu grammatical descriptions used to write transformation rules, and the representativeness of the validated sample.

assumptions (4)
  • domain assumption Minimal-pair forced choice is a valid probe of grammatical competence.
    Adopted from BLiMP, CLiMP, and SLING; the paper does not independently validate that sentence-level perplexity ranking correlates with acceptability in Urdu beyond the human sample. Invoked in Section 3.2.
  • domain assumption The Urdu linguistic generalizations used to construct the pairs (ergativity split, dative subjects, honorific agreement) are correct.
    Relies on the Urdu linguistics literature, e.g., Butt and Rizvi (2008), and the authors' native-speaker expertise. Invoked throughout Section 2.1.
  • ad hoc to paper The transformation rules and regex extraction yield pairs that differ only on the targeted property.
    The paper states minimal pairs were generated by 'systematically introducing controlled variations' but does not provide the complete rule set or quantitative checks that the pairs are truly minimal. Invoked in Sections 2.2.1 and 2.2.2.
  • domain assumption The in-house 735M-token corpus is a representative sample of written Urdu.
    Used for sentence extraction; no distributional analysis is provided to support representativeness. Invoked in Section 2.2.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu." pith.science (2026). https://pith.science/paper/H2QXIR62

@misc{pith2026250801006,
  author       = {Pith},
  title        = {Pith review of: UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2QXIR62}},
  note         = {Machine review of arXiv:2508.01006}
}
read the original abstract

Multilingual Large Language Models (LLMs) have shown remarkable performance across various languages; however, they often include significantly less data for low-resource languages such as Urdu compared to high-resource languages like English. To assess the linguistic knowledge of LLMs in Urdu, we present the Urdu Benchmark of Linguistic Minimal Pairs (UrBLiMP) i.e. pairs of minimally different sentences that contrast in grammatical acceptability. UrBLiMP comprises 5,696 minimal pairs targeting ten core syntactic phenomena, carefully curated using the Urdu Treebank and diverse Urdu text corpora. A human evaluation of UrBLiMP annotations yielded a 96.10% inter-annotator agreement, confirming the reliability of the dataset. We evaluate twenty multilingual LLMs on UrBLiMP, revealing significant variation in performance across linguistic phenomena. While LLaMA-3-70B achieves the highest average accuracy (94.73%), its performance is statistically comparable to other top models such as Gemma-3-27B-PT. These findings highlight both the potential and the limitations of current multilingual LLMs in capturing fine-grained syntactic knowledge in low-resource languages.

Figures

Figures reproduced from arXiv: 2508.01006 by the authors.

Figure 1
Figure 1. Best-performing model from each architecture [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Comparison of model accuracy across linguistic phenomena, showing that the small model outperforms [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

  1. [2]

    Preprint, arXiv:2501.12948

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    arXiv

  2. [6]

    Preprint, arXiv:2504.02768

    Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs . Preprint, arXiv:2504.02768. Daria Kryvosheieva and Roger Levy

    arXiv

  3. [7]

    Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models

    Controlled evaluation of syntactic knowledge in multilingual language models. Preprint, arXiv:2411.07474. J. Richard Landis and Gary G. Koch

    work page Pith review arXiv

  4. [8]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic

    Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderma...

    work page 2021

  5. [11]

    In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606–4634, Abu Dhabi, United Arab Emirates

    SLING: Sino linguistic evaluation of large language models . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4606–4634, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Munief Hassan Tahir, Sana Shams, Layba Fiaz, Farah Adeeba, and Sarmad Hussain

    work page 2022

  6. [12]

    https: //huggingface.co/large-traversaal/ Alif-1.0-8B-Instruct

    Alif 1.0 8b instruct. https: //huggingface.co/large-traversaal/ Alif-1.0-8B-Instruct . Accessed: 2025-07-24. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman

    work page 2025

  7. [13]

    In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online

    mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. Jeremy Zehr and Florian Schwarz

    work page 2021

  8. [2018]

    CoRR, abs/1810.04805

    BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. Toqeer Ehsan and Sarmad Hussain

    arXiv

Show all 14 references
  1. [2020]

    and the HuggingFace Datasets repository ( Lhoest et al. , 2021). The dataset and accompanying code will be released via a public GitHub repository under a permissive open-source license. B Computational Cost The computational cost of evaluating a language model (LM) on UrBLiMP...

    2021

  2. [2021]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominican Republic

    Surface form competition: Why the highest probability answer isn’t always right . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. J...

    2021

  3. [2022]

    arXiv preprint arXiv:2211.01786

    Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. Taiga Someya and Y ohei Oseki

    arXiv

  4. [2023]

    In Findings of the Association for Computational Linguistics: EACL 2023 , pages 1581–1594, Dubrovnik, Croatia

    JBLiMP: Japanese benchmark of linguistic minimal pairs . In Findings of the Association for Computational Linguistics: EACL 2023 , pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, and Mohit Iyyer

    2023

  5. [2024]

    Preprint, arXiv:2407.21783

    The llama 3 herd of models . Preprint, arXiv:2407.21783. Ari Holtzman, Peter West, Vered Shwartz, Y ejin Choi, and Luke Zettlemoyer

    arXiv

  6. [2025]

    Preprint, arXiv:2506.13487

    T urblimp: A turkish benchmark of linguistic minimal pairs . Preprint, arXiv:2506.13487. Miriam Butt and Jafar Rizvi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.