{"id":"b9b3b611-1ede-4cef-bc17-8d695bd90cf2","arxiv_id":"2505.07857","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"The authors propose LLMPIA, a few-shot Urdu intent detection pipeline combining contrastive retraining and prototype-informed attention, and report high F1 scores on ATIS and Web Queries.","lead":"This paper builds a few-shot intent detection pipeline for Urdu by retraining six language models on unlabeled Urdu text and adding a prototype-based attention classifier. It reports strong scores on two datasets, but the evaluation reused test text during retraining, so the unseen-class claims are doubtful.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contrastive re-training in §4.3 uses test/validation text, so the reported F1 does not measure unseen-class generalization.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: §4.3 explicitly combines train, test, and validation sets during LLMCRL re-training. I agree with the reader that this breaks the few-shot unseen-class evaluation. The fix is conceptually simple but requires rerunning all experiments, so the empirical headline claims cannot be accepted as reported. I also note secondary problems that reinforce the verdict: the abstract's F1 values do not match any single row in Tables 5-8 (e.g., Web Queries 76.23% is a 5-shot pre-trained value, not the claimed 1-shot result), and no code, seeds, or error bars are provided. However, the test-split leakage is the decisive issue because it undermines the validity of every reported few-shot result, not just one number. If the authors rerun with a clean held-out unlabeled corpus, the work could become a useful benchmark for Urdu few-shot intent detection; as it stands, the central claim is unsupported.","tokens_in":28846,"tokens_out":4136,"duration_ms":45420,"concrete_test":"Re-run the best LLMPIA configuration reported in Tables 5-8 (MuRIL-base-17 with cosine similarity, 75% seen / 25% unseen splits, 4-way 1-shot and 5-shot) with LLMCRL retraining restricted to D_train only, or to an external Urdu corpus disjoint from D_val and D_test. Keep the PIA training procedure, hyperparameters, and episode construction otherwise identical. If the F1 scores on D_test remain close to 98.25% (ATIS) and 84.42% (Web Queries), the concern is refuted; if they drop materially, the reported numbers reflect test-split adaptation rather than unseen-class generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLMPIA is the first few-shot Urdu intent detection pipeline that predicts unseen intent classes from seen classes and outperforms existing Urdu predictors. For that claim to hold, the evaluation must measure transfer to classes and instances the model has not seen during any training or re-training step. Section 4.3 states: 'During the re-training of models in the LLMCRL, we combined train, test and validation sets of each dataset.' The LLMCRL objective in Eq. (1) is L_LLMCRL = L_MLM(D_unlabeled; θ) + L_SCL(D_unlabeled; θ), so the evaluation-split utterances are directly used to optimize θ via masked language modelling and self-supervised contrastive learning. The test texts are not just unlabeled domain data: they are the exact queries later used to construct the query and support sets for the 4-way few-shot evaluation. The encoder has therefore been adapted to the lexical and distributional properties of the test split before the 'unseen' classes are predicted. This invalidates the claimed generalization: the F1 scores in Tables 5-8 measure performance after encoder adaptation to the evaluation split, not prediction of unseen intent classes from seen classes. A transductive or semi-supervised setting might be defensible if stated explicitly, but the paper presents the results as inductive few-shot generalization to unseen classes. The load-bearing premise fails, so the headline numbers do not support the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMPIA, a few-shot Urdu intent detection pipeline that combines contrastive re-training of pre-trained language models (LLMCRL) with a prototype-informed attention mechanism (PIA). It evaluates six pre-trained language models and thirteen similarity measures on two Urdu datasets, ATIS and Urdu Web Queries, under 4-way 1-shot and 4-way 5-shot settings, and also compares against existing Urdu intent detection predictors. The abstract claims headline F1 scores of 83.28% and 98.25% on ATIS and 76.23% and 84.42% on Web Queries, plus a 53.55% F1-Score improvement over the state of the art.","tokens_in":29212,"tokens_out":9592,"duration_ms":89692,"significance":"If the evaluation were valid, the systematic comparison of six PLMs and thirteen similarity measures on two Urdu datasets would be a useful resource for low-resource Urdu NLP. The manuscript is transparent about hyperparameter search and early stopping, and it covers a broad configuration space. However, as submitted, the central claims are not supported: the LLMCRL re-training stage uses the test and validation text, so the reported few-shot results do not measure inductive generalization to unseen classes. In addition, the abstract misreports which experimental settings produced the headline numbers, and the claimed 53.55% improvement over the state of the art is not present in the results. These issues require a substantial rework of the experimental evaluation and the reported claims.","major_comments":[{"comment":"Section 4.3 states: 'During the re-training of models in the LLMCRL, we combined train, test and validation sets of each dataset.' Since Eq. (1) defines L_LLMCRL = L_MLM(D_unlabeled; θ) + L_SCL(D_unlabeled; θ), and Section 3 defines D_unlabeled as the text from all categories, the encoder used in Tables 5–8 is optimized on the same test and validation utterances that later form the support and query sets described in Section 3.2. The reported F1 scores therefore reflect transductive adaptation to the evaluation split rather than inductive prediction of unseen classes from seen classes. Because the paper's central claim is about generalization to unseen classes, this leakage invalidates the headline few-shot results.","section":"Section 4.3, Eq. (1), Section 3"},{"comment":"The abstract reports '83.28% and 76.23% F1-Score' under the 4-way 1-shot setting for ATIS and Web Queries, respectively. These values correspond to the MuRIL pre-trained 4-way 5-shot results at 75% seen classes in Table 5 (F1 = 0.8328) and Table 6 (F1 = 0.7623), not to any 1-shot result. The actual MuRIL re-trained 1-shot F1 scores at the 75% split are 0.9581 (ATIS) and 0.8403 (Web Queries). The abstract's pairing of experimental settings with numbers is therefore incorrect and must be corrected.","section":"Abstract vs. Tables 5 and 6"},{"comment":"The abstract claims that LLMPIA 'outperformed state-of-the-art predictor by 53.55% F1-Score' on the Web Queries dataset under same-class train/test settings. This figure does not appear in Table 9 or anywhere in the text. Table 9 shows the best proposed pipeline (roberta-small-Urdu, 3-way 5-shot) achieving F1 = 0.9555 versus U-IntentCapsNet's 0.9084, a relative improvement of about 4.71%, which the text itself states. The 53.55% claim should be either substantiated with a precise definition and calculation or removed.","section":"Abstract vs. Table 9, Section 7"},{"comment":"Because the validation set is included in LLMCRL re-training and is also used for early stopping and hyperparameter selection, the model selection reported in Tables 3–4 and the similarity-metric rankings in Tables 7–8 are affected by the same leakage. Furthermore, the paper reports no variance or confidence intervals across episodes or random splits, so it is not possible to assess whether the differences between models or similarity measures that drive the 'best pipeline' conclusions are statistically meaningful.","section":"Section 6, Tables 3–8"}],"minor_comments":[{"comment":"The abbreviation for the contrastive re-training stage is inconsistent: Section 4.1 uses 'LLMRCL' while Sections 4.3 and elsewhere use 'LLMCRL'; please standardize throughout.","section":"Section 4.1"},{"comment":"The line 'Preprint submitted to Nuclear Physics B' appears to be a template artifact and should be removed for a computer-science journal submission.","section":"Header"},{"comment":"The MuRIL model is labeled 'MURiL-base-100-languages' in Table 7 and 'Muril-base-100-languages' in Table 8, while Table 2 and the text correctly call it 'MuRIL-base-17-languages'; the labels should be made consistent.","section":"Tables 7 and 8"},{"comment":"The text quotes accuracy values for MuRIL (e.g., 0.2375, 0.2731) that do not appear in Table 6; please reconcile the prose with the reported tables.","section":"Section 6.1"},{"comment":"Figure 2 shows identical four-intent examples under both 'Web-Queries Dataset' and 'ATIS Dataset' labels, which is confusing; the figure should clarify which examples come from which dataset.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The data leakage in Section 4.3 is the decisive issue. I would not consider acceptance until all experiments are re-run with LLMCRL trained on train-split text only, the abstract and Section 7 claims are corrected, and the results are recomputed. Given the extent of the required re-run, the editor may reasonably treat this as a reject-and-resubmit rather than a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take: this is the first paper I know of to apply few-shot learning to Urdu intent detection, and the systematic model/similarity comparison is genuinely useful. But the evaluation is not trustworthy: Section 4.3 says the LLMCRL re-training combines train, test, and validation sets, which means the encoder has been adapted to the exact queries later used in the \"unseen\" class evaluation. The headline numbers therefore do not measure inductive few-shot generalization. The stress-test note is correct.\n\nWhat is new and good: the paper assembles Urdu ATIS and Web Queries into N-way K-shot episodes, evaluates six pre-trained models and thirteen similarity measures, and combines contrastive re-training with prototype-informed attention. If you work on low-resource intent detection, the model comparison and hyperparameter details are worth mining. The literature review is honest and correctly identifies that prior Urdu work is traditional or zero-shot, not few-shot.\n\nSoft spots, in proportion: the leakage is load-bearing, not a minor quibble. The paper defines D_train, D_val, D_test as having mutually exclusive classes, then re-trains on all three splits. That contradicts its own setting and invalidates the central claim of predicting unseen classes. Separately, the abstract misreports numbers: the 83.28% and 76.23% values correspond to MuRIL's 4-way 5-shot F1 at 75% seen in Tables 5 and 6, not 4-way 1-shot. The claimed 53.55% improvement over SOTA does not appear in Table 9, which shows about 4.7% over U-IntentCapsNet. No code or error bars are provided. There are also typos and inconsistent model names (MURiL vs. MuRIL, \"Muril-100-languages\" which does not exist). All of these are fixable in revision.\n\nMy recommendation: this is not acceptable as-is, but it deserves a serious referee rather than a desk reject. The task is important, the gap is real, and the leakage is fixable by re-running the experiments without test/validation text in re-training. If the authors redo the evaluation cleanly, the benchmark would be a useful contribution to Urdu NLP. I would not cite the current numbers, but I would read a revised version.","headline":"First few-shot Urdu intent detection benchmark, but the re-training step leaks test and validation text, so the headline F1 numbers do not measure unseen-class generalization.","tokens_in":29726,"tokens_out":2713,"would_cite":false,"duration_ms":28537,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Few-shot Urdu intent detection can label classes a model never saw in training, with a reported 98.25% F1 in the 5-shot ATIS setting.","keywords":["Urdu intent detection","few-shot learning","contrastive learning","prototype-informed attention","unseen intent classes","low-resource language processing","multilingual language models","natural language understanding"],"falsifier":"A direct falsifier is to rerun the full LLMPIA evaluation after replacing the combined-split unlabeled corpus with unlabeled text drawn only from the training split; if the 4-way 1-shot and 4-way 5-shot F1 scores on ATIS and Web Queries drop materially, the original unseen-class numbers depend on exposure to test text, not on generalization. A weaker check is to search the re-training corpus for test queries and measure how much of the test set appears in it verbatim.","tokens_in":28675,"feed_emoji":"💬","tokens_out":12695,"duration_ms":118790,"temperature":0.7,"pith_summary":"Urdu is the world's tenth most-spoken language, but prior Urdu intent detectors could only assign intents they saw during training. This paper claims to introduce LLMPIA, the first few-shot Urdu intent-detection pipeline that can also predict unseen intent classes: it re-trains a pre-trained language model on unlabeled Urdu text with a contrastive objective, then a prototype-informed attention network classifies a query by comparing it to a few labeled examples per intent. On the Urdu ATIS and Web Queries benchmarks the pipeline reports 83.28% and 98.25% F1 (4-way 1-shot and 5-shot) on ATIS, and 76.23% and 84.42% F1 on Web Queries, with a same-class case study reporting a 53.55% F1 gain over the strongest existing Urdu predictor. If the result holds, developers of Urdu conversational systems would have a practical route to adding new intents from one to five examples per class instead of large labeled corpora.","feed_headline":"Urdu intent model labels unseen classes, hits 98% F1","feed_subtitle":"Contrastive re-training plus prototype attention lifts few-shot F1 on two Urdu benchmarks.","key_machinery":"The load-bearing mechanism has two stages. First, LLMRCL (Large Language Models Enhanced Representations with Contrastive Learning) re-trains a pre-trained encoder on unlabeled Urdu text with two objectives: masked language modelling at a 25% mask rate, plus a self-supervised contrastive loss that pulls semantically related sentence pairs together and pushes unrelated pairs apart. Second, Prototype-Informed Attention (PIA) builds on a Siamese architecture, meaning two network branches that share the same weights, to turn the encoded support examples into a class prototype and to compare a query embedding against that prototype. The attention layers compute inner-sentence and inner-class feature interactions, a prototype-informed layer refines the prototype, and an adaptive layer maps prototypes and queries into a common space before cosine similarity with a temperature coefficient scores the match. Two unsupervised contrastive regularization terms are added to the cross-entropy metric loss, so the total objective simultaneously polishes prototypes and prevents few-shot overfitting.","core_discovery":"On the paper's own terms, the central discovery is that a language model's Urdu representations can be sharpened for few-shot intent detection without any labeled data, by re-training it with a self-supervised contrastive loss on unlabeled Urdu text, and that those sharper embeddings plug directly into a prototype-informed attention mechanism that compares queries with class prototypes built from a small support set. The paper reports that this combined pipeline, LLMPIA, predicts unseen intent classes from seen classes: 83.28% F1 in 4-way 1-shot and 98.25% F1 in 4-way 5-shot on the Urdu ATIS dataset, and 76.23% and 84.42% F1 on the Urdu Web Queries dataset. Across all six pre-trained language models and both datasets, the contrastively re-trained versions outperform their pre-trained counterparts. In the unseen-class few-shot regime the best encoder is the South-Asian multilingual MuRIL model, and cosine similarity is the most reliable of the thirteen similarity measures tested. In a Web Queries case study where all classes appear in both training and test sets, the paper reports that its pipeline exceeds the strongest existing Urdu predictor by 53.55% F1.","pith_inferences":["A testable extension the paper does not run: repeat LLMRCL using only training-split unlabeled text and compare the F1 gap; if it shrinks, some of the reported unseen-class gain comes from exposure to test and validation text rather than from generalization.","The recipe is not Urdu-specific: any low-resource language with unlabeled text but few intent labels could use the same contrastive re-training plus prototype-attention steps, though the paper does not test this.","For deployed assistants, the 1-shot numbers imply that one user-confirmed utterance could open a new intent class, making the support set a live product operation rather than a fixed training artifact.","Because the strongest few-shot encoder is multilingual rather than Urdu-only, the same prototype pipeline may benefit from simply swapping in newer multilingual encoders as they appear, without changing the rest of the architecture."],"forward_implications":["In the few-shot unseen-class regime, a multilingual model trained on South Asian languages outperforms the Urdu-only model, so the best encoder choice depends on whether the deployment must handle new intents.","Re-training always helps: every one of the six encoders improves after contrastive re-training on unlabeled Urdu text, making the re-training stage a reliable component rather than a lucky exception.","The 13-metric study shows cosine similarity is the safe default for prototype-based Urdu intent matching, while Hamming distance and KL divergence collapse to near-zero F1 in the same settings.","Adding support examples helps: 4-way 5-shot consistently beats 4-way 1-shot, so few-shot Urdu intent detection still rewards collecting a handful of labeled examples per intent.","If the reported unseen-class performance is genuine, Urdu moves from having no few-shot intent detector to having a method whose reported F1 scores are competitive with those for higher-resource languages."],"supporting_citations":[{"why":"Supplies the contrastive-learning objective that the LLMRCL re-training stage is built on.","marker":"[16]"},{"why":"Provides MuRIL, the multilingual model that becomes the top few-shot encoder in the experiments.","marker":"[32]"},{"why":"Provides the RoBERTa base behind the Urdu-small model that wins the same-class case study.","marker":"[40]"},{"why":"Supplies the Urdu Web Queries benchmark and the strongest existing predictor used as the case-study baseline.","marker":"[58]"},{"why":"Supplies the Urdu ATIS benchmark with its flight-related intent labels.","marker":"[59]"},{"why":"Supplies the prototypical-network idea from which the prototype-informed attention layer is developed.","marker":"[61]"},{"why":"Supplies the zero-shot LLM baselines on Urdu that motivate the few-shot approach.","marker":"[64]"},{"why":"Supplies the multilingual BERT model (BERT-base-104-languages) used as one of the six encoders.","marker":"[70]"}],"fun_headline_variants":["Urdu intent detector hits 98% F1 with contrastive LLM re-training","Few-shot Urdu intent: prototype-informed LLM reaches 98% F1","Contrastive re-training lifts Urdu intent F1 to 98% in few-shot","LLM prototype pipeline for Urdu intents: 98% F1, unseen classes","Unlabeled Urdu text refines LLM intent detection to 98% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Section 4.3 of the paper states that during LLMCRL re-training the authors combined the train, test, and validation sets of each dataset; the few-shot unseen-class results therefore rest on the premise that using test and validation text in this unlabeled re-training does not leak information about the unseen classes.","fun_headline_variants_meta":{"raw":{"variants":["Urdu intent detector hits 98% F1 with contrastive LLM re-training","Few-shot Urdu intent: prototype-informed LLM reaches 98% F1","Contrastive re-training lifts Urdu intent F1 to 98% in few-shot","LLM prototype pipeline for Urdu intents: 98% F1, unseen classes","Unlabeled Urdu text refines LLM intent detection to 98% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1690,"prompt_tokens":1104,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":720,"completion_tokens_details":{"reasoning_tokens":478}},"tokens_in":720,"tokens_out":586,"duration_ms":5172,"temperature":1.0,"reasoning_tokens":478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:15:39.333227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier is to rerun the full LLMPIA evaluation after replacing the combined-split unlabeled corpus with unlabeled text drawn only from the training split; if the 4-way 1-shot and 4-way 5-shot F1 scores on ATIS and Web Queries drop materially, the original unseen-class numbers depend on exposure to test text, not on generalization. A weaker check is to search the re-training corpus for test queries and measure how much of the test set appears in it verbatim.","supporting_citations":[{"cited_title":"Debiased contrastive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive-learning objective that the LLMRCL re-training stage is built on."},{"cited_title":"Improving user intent detection in urdu web queries with capsule net architectures","cited_arxiv_id":null,"evidence_quote":"Supplies the Urdu Web Queries benchmark and the strongest existing predictor used as the case-study baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Urdu ATIS benchmark with its flight-related intent labels."},{"cited_title":"Prototypical networks for few- shot learning, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA","cited_arxiv_id":null,"evidence_quote":"Supplies the prototypical-network idea from which the prototype-informed attention layer is developed."},{"cited_title":"Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks","cited_arxiv_id":"2405.15453","evidence_quote":"Supplies the zero-shot LLM baselines on Urdu that motivate the few-shot approach."}],"review_version":1}