{"id":"f86e08f0-fe99-42cb-b561-3ab368b7c493","arxiv_id":"2505.12273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dialect-guided prompt with tokenizer extension and a regression head improves reference-less machine translation evaluation for Sylheti-English, with best Spearman gain +0.1083 over an annotation-guided baseline.","lead":"Researchers built a new Sylheti-English benchmark with human translation-quality scores, then tested a reference-less evaluation pipeline that adds a dialect glossary, extra tokenizer words, and a small learned score head on top of frozen large language models. It raised correlation with human scores for several models by up to 0.1083, but one model scored worse and the gap between training and test data is not described.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on unstated train/test separation; reported gains may be in-sample artifacts.","rationale":"The reader's weakest assumption correctly identifies the missing train/test split as the central threat. I agree that this is the single most load-bearing concern: the entire reported advantage of the proposed pipeline depends on whether the regression head, tokenizer augmentation, and DG glossary were fitted to the same sentences used for the correlation tables. The paper provides no sentence-level split, no cross-validation, and no statement that the tokenizer and glossary were built without access to evaluation sentences. Without that information, the +0.1083 Spearman gain could be an optimistic in-sample result, which would collapse the claim that the pipeline better aligns with human DA judgments. I also note the Gemma result as a secondary, direct contradiction of the word 'consistently,' but I do not treat it as the primary load-bearing issue because it weakens the rhetoric rather than the underlying method. Since the reader already recommended a conditional verdict and the concern is a resolvable reporting/validation gap, my stress-test does not move the verdict; it strengthens the condition under which acceptance should be granted.","tokens_in":5946,"tokens_out":8381,"duration_ms":87875,"concrete_test":"Using the released repository, identify or create a documented split: train the regression head on the disclosed 1,200 samples, construct the tokenizer vocabulary and DG glossary using only those training sentences (or an external Sylheti corpus), and recompute Table 2 correlations on the remaining 300 held-out sentences. Run 5-fold cross-validation with the same separation and report mean and standard deviation. If the mean Spearman gain of DG over AG falls below +0.1083, or the sign reverses for any model, the out-of-sample claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive weakness is that the paper never establishes that the reported correlations are out-of-sample. Section 3.4 states the regression head is fine-tuned on 1,200 samples, but Section 4.1 reports Pearson/Spearman correlations without saying whether Tables 1-3 are computed on a held-out 300, on the training pool, or on all 1,500 sentences. Sections 3.2 and 3.3 similarly describe the DG glossary and dialect-aware tokenizer as built from 'Sylheti sentences,' with no disclosure that those resources exclude the evaluation sentences. If the regression head, tokenizer vocabulary, and glossary were derived from the same sentences used for evaluation, the +0.1083 Spearman gain and every ablation increment in Table 3 are in-sample artifacts rather than evidence for a reference-less evaluation pipeline. This is load-bearing because the central claim is a comparative claim about alignment with human DA scores; without a demonstrated held-out evaluation, the comparison is unverified. In addition, the Gemma rows in Tables 1 and 2 contradict the abstract's 'consistently outperforms,' since AG beats DG for Gemma on both correlations, so the headline claim is also overstated as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a reference-less machine translation (MT) evaluation pipeline for Sylheti–English, a low-resource, dialect-rich language pair. The authors extend the ONUBAD corpus to 1,500 Sylheti–English sentence pairs, obtain NLLB-200 translations, and collect Direct Assessment (DA) scores from two native speakers. The proposed pipeline has three components: a Dialect Guided (DG) prompting strategy that injects a Sylheti–English glossary into the prompt, a dialect-aware tokenizer extension that adds unique Sylheti tokens, and a regression head trained on frozen LLM embeddings to predict scalar DA scores. The system is compared with Annotation Guided (AG) prompting and a combined DAG variant across Llama-2-7B, Llama-2-13B, OpenChat, and Gemma, using Pearson and Spearman correlations as evaluation metrics. The abstract claims consistent outperformance with a maximum Spearman gain of +0.1083. The paper does not disclose the train/test split used for the regression head or the provenance of the glossary and tokenizer vocabulary relative to the evaluation set.","tokens_in":6131,"tokens_out":5989,"duration_ms":50832,"significance":"If the reported numbers are out-of-sample, the paper offers a genuinely useful contribution to low-resource MT evaluation: a native-annotated benchmark for a severely underrepresented dialect, a concrete method for incorporating dialect-specific lexical knowledge into LLM prompts, and evidence that a lightweight regression head on frozen LLM features improves DA alignment. The public release of the dataset and code is a strength. The significance is nevertheless conditional on resolving a central methodological ambiguity: every comparative claim depends on whether the regression head, tokenizer vocabulary, and glossary were derived from the evaluation sentences. As written, the evidence for the headline claim is incomplete, and the Gemma results contradict the 'consistently outperforms' wording.","major_comments":[{"comment":"Section 3.4 states that the regression head was fine-tuned on 1,200 samples, but Section 4.1 reports correlations without disclosing whether Tables 1 and 2 are computed on a held-out 300-sentence subset, on the training pool, or on all 1,500 sentences. If the reported correlations include the training samples, the +0.1083 Spearman gain and the Table 3 increments are in-sample artifacts, not evidence for a working evaluation pipeline. Please specify the split exactly and, ideally, report cross-validated or separate test-set correlations.","section":"3.4 and 4.1"},{"comment":"The DG prompt glossary and the dialect-aware tokenizer vocabulary are described as built from 'Sylheti sentences,' but the paper never states that these resources were constructed without access to the evaluation sentences. If the glossary or the tokenizer vocabulary was derived from the test sentences, the improvements attributed to the DG prompt and the tokenizer would be partially in-sample. Please clarify the data provenance of both resources relative to the evaluation set.","section":"3.2 and 3.3"},{"comment":"The abstract's claim that the proposed pipeline 'consistently outperforms' existing methods is contradicted by the Gemma rows: in Table 1, AG achieves 0.3247 vs DG 0.3079, and in Table 2, AG achieves 0.3566 vs DG 0.3488. The Section 5 discussion acknowledges the Gemma exception, so the abstract and any summary of the results should be revised to reflect gains on three of four models rather than consistent outperformance.","section":"Abstract, Tables 1 and 2"},{"comment":"The ablation study in Table 3 does not report variance, number of runs, or any significance test, so the incremental benefit of each component cannot be distinguished from noise. In addition, the '× × ×' baseline reports Pearson = -0.1446 for OpenChat 'without any modifications,' which is inconsistent with the AG OpenChat result of 0.2299 in Table 1 unless AG is treated as a separate modification; this needs clarification. The relative change from -0.1446 to 0.0336 described as 'approximately 123%' is not a meaningful percentage improvement for a correlation coefficient.","section":"4.2"},{"comment":"The experimental comparison contains only the AG baseline; no standard reference-free QE systems such as COMETKiwi or XCOMET are used. The abstract's phrase 'outperforms existing methods' therefore overstates the support provided by the experiments, which at most show a gain over AG prompting under the reported settings.","section":"4.1"},{"comment":"The benchmark's ground truth consists of DA scores from only two native Sylheti speakers, and the paper reports no inter-annotator agreement (e.g., Pearson correlation between annotators or Krippendorff's alpha). Without a reliability estimate for the averaged scores, the quality of the benchmark itself is not established, which weakens the validity of any downstream correlation claim.","section":"3.1"}],"minor_comments":[{"comment":"The text contains the placeholder '[cite]' next to 'SOTA AG'; this must be replaced with a proper citation.","section":"4.1"},{"comment":"References 4 and 14 are the same item; deduplicate them.","section":"References"},{"comment":"Section 3.1 says ONUBAD contains '1,540 words, 130 clauses, and 980 sentences per dialect,' but the paper does not explain how the 980 extracted sentence pairs relate to the 1,500 final pairs beyond adding 520 new ones; clarify the extraction and filtering process.","section":"3.1"},{"comment":"The paper should specify which Gemma checkpoint (e.g., Gemma-7B) was used, since Section 5 attributes behavior to lack of instruction tuning.","section":"3.5 and 5"},{"comment":"Figure 2 is described in the text but not present in the provided version; ensure the final PDF includes the DG prompt figure.","section":"Figure 2"},{"comment":"Use a consistent spelling for 'Llama' throughout (e.g., 'Llama-2' rather than 'LLaMA-2').","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central ambiguity about train/test separation is serious enough that I would want to see the split and, ideally, cross-validated numbers before accepting. The paper also overclaims consistency in the abstract. These are fixable in revision, so I am not recommending rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The genuinely new thing here is the Sylheti-English benchmark: 520 new sentence pairs, NLLB-200 outputs, and native-speaker DA scores layered on the ONUBAD material. That is real, useful data for a dialect that basically has no MT evaluation resources. The dialect-glossary prompt variation and the tokenizer extension are sensible engineering moves, and the authors ship code and data, which makes the work easy to build on. Credit where it is due: the resource itself is a contribution.\n\nThe soft spots are in the evaluation. Section 3.4 says the regression head is trained on 1,200 samples; Section 4.1 reports Pearson and Spearman correlations without ever saying whether those correlations are computed on a held-out subset, on the training pool, or on all 1,500 sentences. If the numbers are in-sample, the +0.1083 Spearman gain and the ablation increments in Table 3 are artifacts, not evidence. This is not a minor omission; it is load-bearing for the central claim. The stress-test note is right to flag it. Also, the abstract says \"consistently outperforms,\" but Gemma does better with AG than with DG in both tables, so the headline claim is overstated as written.\n\nOther weaknesses are more fixable: no standard QE baselines (an XLM-R or COMET-style reference-free system), no inter-annotator agreement reported, only NLLB-200 as the MT source, and no significance tests for the ablation. The absolute correlations are low (under 0.29 for the best DG setting), so the practical value is modest even if the split is clean.\n\nThe authors need to state the split, report held-out numbers, add at least one standard baseline, and temper the language. If the split is proper, the result is a decent incremental contribution for a low-resource dialect. If it is not, the main result collapses. The paper deserved a serious referee, but only with a demand for the train/test disclosure and a rerun on the held-out 300. This is the kind of work a peer reviewer can meaningfully improve.","headline":"Useful new Sylheti-English benchmark, but the central comparison is unverifiable because the paper never discloses whether the reported correlations are out-of-sample.","tokens_in":6627,"tokens_out":1444,"would_cite":false,"duration_ms":15568,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dialect-glossed prompt, an extended tokenizer, and a regression head let reference-less LLM evaluation of Sylheti-English track human quality scores, improving Spearman correlation by up to +0.1083 over the annotation-guided baseline.","keywords":["Machine Translation Evaluation","Large Language Models (LLMs)","Low-Resource Languages","Prompt Engineering","Dialect-Guided Prompting","Sylheti","Direct Assessment","Reference-less Evaluation"],"falsifier":"Use the released data and code to recompute Pearson and Spearman correlations on the subset of the 1,500-pair pool that the regression head never saw during training, and compare the DG gain over AG there; if the gain shrinks or flips sign on that held-out subset, the central improvement claim collapses. A second, cheaper check: compute inter-annotator agreement between the two native speakers — if the pipeline's correlation with averaged scores approaches or exceeds that agreement, the benchmark's ground truth is the limiting factor, not the pipeline.","tokens_in":5743,"feed_emoji":"🗣️","tokens_out":15820,"duration_ms":136851,"temperature":0.7,"pith_summary":"Evaluating machine translation normally needs human reference translations, which are costly and often nonexistent for dialect-rich low-resource languages like Sylheti. This paper tries to establish that general-purpose LLMs, guided properly, can score such translations without any reference at all: give the model a glossary of dialect terms, extend its tokenizer with dialect vocabulary, and map its frozen embeddings to human Direct Assessment scores through a small regression head. On an extended Sylheti-English benchmark of 1,500 sentence pairs with native-speaker scores, the pipeline raises rank correlation with human judgments by up to 0.1083 over the prior annotation-guided prompting baseline across three of four open LLMs. If the claim holds, MT developers for underrepresented dialects gain a cheap, reference-free way to know which translations are good enough to use, removing a bottleneck that keeps such languages out of translation systems.","feed_headline":"Dialect glosses raise LLM translation-quality scores by +0.1083","feed_subtitle":"Glossed dialect terms and a tuned tokenizer align LLM scores with native-speaker ratings, no references needed.","key_machinery":"The carrying mechanism is the dialect-guided (DG) prompt, a three-part instruction that hands the LLM a glossary of Sylheti words with their English glosses alongside the original Sylheti-influenced Bengali source sentence and the machine-translated English sentence to be scored. The glossary is built by cross-referencing each source word against a curated Sylheti-English dictionary, so the model never has to infer dialect vocabulary from a tokenizer that barely knows it. Two supporting mechanisms complete the pipeline: a byte-level BPE tokenizer the authors retrain on Sylheti sentences and merge with the original LLM vocabulary, and a lightweight regression head, trained with mean-squared error on 1,200 samples of frozen LLM embeddings, that converts the model's representations into a scalar DA score. The ablation isolates each piece — tokenizer, head, and prompt — and shows the prompt stage delivering the final and largest jump in correlation.","core_discovery":"The paper's central claim is that dialect awareness, exposed explicitly to the model, is what unlocks reference-less LLM evaluation for dialect-rich languages. Its demonstration: for Llama-2-7B, Llama-2-13B, and OpenChat, the dialect-guided (DG) prompt correlates more strongly with human DA scores than the annotation-guided (AG) prompt of prior work, in both Pearson and Spearman terms — the largest Spearman gain is +0.1083 (0.1496 to 0.2579) on Llama-2-13B, and OpenChat reaches the best absolute correlations of 0.2826 Pearson and 0.2852 Spearman. The exception is Gemma, which scores marginally higher under AG; the paper attributes this to Gemma lacking chat tuning. The ablation on OpenChat attributes the total effect to the stacking of components: the untouched model sits at -0.1446 Pearson correlation, the dialect-aware tokenizer alone brings it to 0.0336, the regression head to 0.2312, and the DG prompt to 0.2734. The paper also contributes the benchmark itself: 520 new Sylheti source sentences, NLLB-200 translations for all 1,500 pairs, and Direct Assessment scores averaged over two native speakers.","pith_inferences":["The same glossary mechanism should transfer to sibling dialect continua — Chittagong and Barisal Bengali, or Arabic and Chinese dialect groups — as long as a dialect-to-English glossary exists; a direct test would build a DG prompt from such a dictionary and measure the Spearman gain over the annotation-guided baseline, which the released code makes straightforward.","The ablation result suggests that what cripples LLM evaluation on low-resource dialects may be surface-form recognition more than reasoning ability: a vocabulary-only intervention flips OpenChat's correlation from -0.1446 to +0.0336, so cheaper fixes than larger models or fine-tuning deserve attention.","Since every DA label in the benchmark anchors to NLLB-200's error profile, the pipeline's ranking fidelity on other MT systems' outputs is an open question; scoring translations from several engines against the same 1,500 sources would test whether the +0.1083 gain is specific to that distribution.","Editorial note on the manuscript itself: Section 4.1 cites the annotation-guided baseline with a bare '[cite]' marker rather than a numbered reference, so the comparison's provenance must be read back from the background section; the numbers in this summary take that baseline to be the AG prompting of the prior work the paper extends."],"forward_implications":["Reference-free evaluation for dialect-rich languages no longer requires full fine-tuning: only a small regression head is trained, on 1,200 samples, while the LLM stays frozen.","Dialect cues in the prompt carry more signal than the official annotation guidelines: DG beats DAG (guidelines plus dialect cues) on every model tested, so prompt design for low-resource evaluation should lead with dialect context.","MT developers for Sylheti can run the released pipeline and obtain human-aligned quality scores for new translations, with no human references needed for scoring.","The tokenizer extension alone moves OpenChat from a negative to a positive correlation with human scores, indicating that lexical coverage is a first-order barrier for low-resource dialects.","Larger chat-tuned models gain most from dialect guidance, so scaling model size and dialect-aware prompting are complementary rather than competing routes."],"supporting_citations":[{"why":"Supplies the annotation-guided (AG) prompting baseline that every pipeline comparison in Section 4 is measured against.","marker":"[8]"},{"why":"Justifies the regression-head design by showing LLM embeddings serve as feature representations for downstream regression tasks.","marker":"[11]"},{"why":"Provides the ONUBAD Sylheti-English parallel corpus that the benchmark extends with 520 new sentence pairs.","marker":"[12]"},{"why":"NLLB-200 generates the machine translations that carry all DA annotation and all evaluation.","marker":"[13]"},{"why":"Provides the Direct Assessment guidelines the two native speakers followed when scoring translations.","marker":"[14]"},{"why":"The byte-level BPE tokenizer architecture retrained on Sylheti sentences to form the dialect-aware vocabulary.","marker":"[15]"},{"why":"Llama-2-7B and Llama-2-13B are two of the four models scored under AG, DG, and DAG prompts.","marker":"[16]"},{"why":"OpenChat yields the best absolute correlations and is the model used for the component-wise ablation.","marker":"[17]"},{"why":"Gemma anchors the claim's boundary: the one model that does not improve under DG, attributed to missing chat tuning.","marker":"[18]"}],"fun_headline_variants":["Dialect guidance boosts LLM eval scores by +0.1083","Reference-less LLM eval improves with dialect prompts","Glossed dialect terms lift LLM translation scoring","Sylheti-English benchmark aids reference-less LLM evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported correlations are computed on data the regression head did not train on: the paper never states the train/test split, reports no inter-annotator agreement for the two native speakers whose averaged DA scores serve as ground truth, and evaluates only NLLB-200 translations, so if the evaluation subset overlaps the 1,200-sample training set, the +0.1083 gain over the annotation-guided baseline reflects the training distribution rather than reference-less evaluation skill.","fun_headline_variants_meta":{"raw":{"variants":["Dialect guidance boosts LLM eval scores by +0.1083","Reference-less LLM eval improves with dialect prompts","Glossed dialect terms lift LLM translation scoring","Sylheti-English benchmark aids reference-less LLM evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3433,"prompt_tokens":1046,"completion_tokens":2387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":2319}},"tokens_in":662,"tokens_out":2387,"duration_ms":19340,"temperature":1.0,"reasoning_tokens":2319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:36:35.311332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use the released data and code to recompute Pearson and Spearman correlations on the subset of the 1,500-pair pool that the regression head never saw during training, and compare the DG gain over AG there; if the gain shrinks or flips sign on that held-out subset, the central improvement claim collapses. A second, cheaper check: compute inter-annotator agreement between the two native speakers — if the pipeline's correlation with averaged scores approaches or exceeds that agreement, the benchmark's ground truth is the limiting factor, not the pipeline.","supporting_citations":[{"cited_title":"When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages","cited_arxiv_id":"2501.04473","evidence_quote":"Supplies the annotation-guided (AG) prompting baseline that every pipeline comparison in Section 4 is measured against."},{"cited_title":"ONUBAD: A comprehensive dataset for automated conversion of Bangla regional dialects into standard Bengali dialect","cited_arxiv_id":null,"evidence_quote":"Provides the ONUBAD Sylheti-English parallel corpus that the benchmark extends with 520 new sentence pairs."},{"cited_title":"Machine translation human evaluation: an investigation of eval- uation based on post -editing and its relation with direct assessment","cited_arxiv_id":null,"evidence_quote":"Provides the Direct Assessment guidelines the two native speakers followed when scoring translations."},{"cited_title":"Neural machine translation with byte - level subwords","cited_arxiv_id":null,"evidence_quote":"The byte-level BPE tokenizer architecture retrained on Sylheti sentences to form the dialect-aware vocabulary."}],"review_version":1}