{"id":"f9b50111-d6e8-466f-b578-5fc3b7e336a5","arxiv_id":"2505.18709","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors report that an LSTM model achieves 89.3% accuracy on a 1,200-sentence Sylheti-to-Modern Bangla translation task, but the evaluation protocol and dataset are not described rigorously enough to support the claim.","lead":"This paper trains three standard neural network models to translate Sylheti, a spoken dialect of Bangla, into standard Bangla using only 1,200 example sentences. It reports LSTM as the best with 89.3% accuracy, but the result is weakened by undefined evaluation metrics, a tiny dataset, and inconsistent numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 89.3% accuracy claim is not well-defined: Section IV applies classification metrics (equations 9-12) to a variable-length sequence-generation task, no decoding or alignment rule is specified, and Table III reports Recall equal to Accuracy for every model, so the headline number cannot be…","rationale":"The reader rejected the paper on the grounds that the private dataset and undefined metrics make the accuracy claim unverifiable. I agree with the rejection, but I would place the load-bearing weakness one step earlier: even if the dataset were clean and released, the manuscript does not define the evaluation metric for sequence generation, so the reported 89.3% cannot be interpreted or reproduced. The strongest evidence for this is that Section IV applies binary classification formulae to a translation task and Table III shows Recall equal to Accuracy for every model, which is not plausible under the stated formulas unless the metrics were simply copied from training accuracy. A single computational check — running the released evaluation code on the released test split and comparing against a standard translation metric — would settle whether the headline number is meaningful. Since the reader's verdict of REJECT remains correct, no change to the verdict is needed.","tokens_in":9543,"tokens_out":6017,"duration_ms":48953,"concrete_test":"Request the authors' evaluation script and test split, then run greedy decoding with the released LSTM checkpoint and recompute the reported metrics using (a) exact sentence-match accuracy and (b) token-level accuracy with an explicit alignment rule. If either number does not reproduce 0.8973, or if the evaluation script defines accuracy differently from the paper's equations, the 89.3% claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires a reproducible evaluation protocol, and the manuscript does not provide one. Section IV defines accuracy, precision, recall, and F1 via TP/TN/FP/FN (equations 9-12), which are classification metrics. The task is sequence-to-sequence translation with variable output lengths (LSTM max length 50; Seq2Seq input lengths 12 and 11). The paper never states how a prediction is counted as correct during evaluation: greedy decoding vs. teacher forcing, exact sentence match vs. token-level match, how padding and out-of-vocabulary tokens are handled, or whether a standard metric such as BLEU is used. The LSTM output layer is described as shape (None, 50, 10000), suggesting per-timestep softmax classification, but that would make 'accuracy' a token-level training statistic, not a translation accuracy. This reading is supported by Table III, where Recall is numerically identical to Accuracy for all three models (0.8973, 0.7625, 0.7069), and by the inconsistency between the abstract's 89.3% and Figure 6's 89.7%. The private dataset compounds the problem, but the undefined metric alone is sufficient to block verification of the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a comparative study of LSTM, Bi-LSTM, and Seq2Seq models for translating Sylheti Bangla to Modern Bangla using a private dataset of 1,200 sentence pairs. The authors describe data collection from Sylhet region speakers, preprocessing steps (punctuation removal, stemming, padding, tokenization), and model training with an 80/20 train/test split. The central claim, stated in the abstract and conclusion, is that LSTM achieves the best performance with 89.3% accuracy. The paper provides classification-style equations for accuracy, precision, recall, and F1, and a results table (Table III) listing scores for the three models.","tokens_in":9802,"tokens_out":2148,"duration_ms":18617,"significance":"If substantiated, a reliable Sylheti-to-Modern Bangla translator would be a useful contribution to low-resource Bangla dialect NLP, and the paper addresses a genuinely under-resourced language pair. However, the manuscript as written does not support its headline claim: the evaluation metric for a sequence-generation task is undefined, the reported numbers are internally inconsistent, and the dataset is neither released nor demonstrated to be reliable. These issues are load-bearing rather than cosmetic. The paper also provides no code, no ablation, and no statistical analysis, so the comparative ranking of the three architectures is not verifiable.","major_comments":[{"comment":"The evaluation metric for the translation task is never defined. The paper applies classification formulas (accuracy, precision, recall, F1) based on TP/TN/FP/FN, but the task is variable-length sequence-to-sequence generation (LSTM max length 50; Seq2Seq input lengths 12 and 11). The paper does not state whether predictions are obtained by greedy decoding or teacher forcing, whether matches are exact sentence matches or token-level matches, how padding tokens are handled, or how out-of-vocabulary words are scored. Without this protocol, the reported 'accuracy' is not a well-defined measure of translation quality, and the headline number of 89.3% is unsupported.","section":"Section IV, Eqs. (9)–(12), and Table III"},{"comment":"The reported best accuracy is inconsistent across the paper: the abstract and conclusion state 89.3%, Section IV and Fig. 6 report 89.7%, and Table III reports 0.8973 (which is closer to 89.73%). These discrepancies, combined with the undefined metric, make it impossible to know which number is intended or how it was computed.","section":"Abstract vs. Section IV and Fig. 6 vs. Table III"},{"comment":"The dataset is not released, and the sample rows in Table I contain apparently corrupted Bengali characters (e.g., 'আমি ত োিোকে শুনক পোচ্ছি নো' and 'েোগ েকেো নো'), which suggests either encoding errors in the manuscript or data of questionable integrity. The paper provides no annotation protocol, no inter-annotator agreement, and no details on how the Sylheti translations were verified. Since the entire reported accuracy is measured against this private dataset, these gaps undermine the validity of any quantitative claim.","section":"Section III.B.1 and Table I"},{"comment":"Table III reports Recall values numerically identical to Accuracy for all three models (0.8973, 0.7625, 0.7069). For any nontrivial sequence-generation task, this equality would be highly unusual and is not explained. The authors do not discuss the relationship between the two metrics, and the equality suggests that the scores may be copied, miscomputed, or derived from an inapplicable per-timestep classification setting rather than from actual translation evaluation.","section":"Section IV and Table III"},{"comment":"The model descriptions are internally inconsistent and prevent reproduction. For example, the LSTM block states output shape (None, 50, 10000) and 2,767,632 parameters, while the Seq2Seq section says the dense output shape is 885 vectors for each token and later mentions a vector size of 1240 in the Bi-LSTM description. The text also gives contradictory numbers for vocabulary sizes and output dimensions. These inconsistencies, together with the lack of a clear decoding strategy, make it impossible to reconstruct the exact models that produced the reported results.","section":"Section III.C"}],"minor_comments":[{"comment":"Several literature-review citations appear misnumbered: reference numbering in the text does not always match the listed references (e.g., [3] is described as sentiment analysis but the reference list entry for [3] is a Bangla-English NMT paper).","section":"Section II"},{"comment":"The stemming step is described as important, but no stemmer is specified or evaluated; the paper later admits there are not good stemmers for Bangla. It should either clarify which stemmer was used or omit the claim.","section":"Section III.B.2"},{"comment":"The phrase 'the outer space layer of our model is 128' is unclear; presumably it refers to LSTM hidden units, but the terminology is nonstandard and should be corrected.","section":"Section III.C"},{"comment":"Figures 7–12 are referenced but not described in enough detail; the axis labels and units are not visible in the text, making the train/validation curves difficult to interpret.","section":"Section IV"},{"comment":"The conclusion repeats the abstract's 89.3% figure, which conflicts with Section IV's 89.7%; the authors should reconcile these values and state the exact number with the evaluation protocol used.","section":"Section V"}],"recommendation":"reject","confidential_remarks":"The core problem is that the headline accuracy claim is not well-defined and the data supporting it are not available, so the paper cannot be verified. The literature review also leans heavily on Bangla classification work rather than translation, and the novelty relative to existing low-resource NMT work is not clearly established. Even with a major revision that defines the metric and releases the dataset, the current numerical results would likely change, and the paper would need substantial additional analysis to support the claim that LSTM outperforms Seq2Seq for this task."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Sylheti-to-Bangla translation paper. The honest summary: the task is worth doing, and the authors collected a small parallel corpus from Sylhet speakers, which is real work. But the central claim—LSTM at 89.3% accuracy—is unverifiable as written, and I think the metric is simply wrong for the job.\n\nWhat's actually new: a new 1,200-sentence Sylheti–Modern Bangla parallel dataset and a first comparison of three standard sequence models on that pair. That's a legitimate, if modest, contribution to a genuinely low-resource language. The preprocessing description is basic but not unreasonable, and the model architectures are standard.\n\nThe soft spots are load-bearing. Section IV defines accuracy, precision, recall, and F1 with classification equations (9–12). The task is variable-length sequence generation. There is no description of how a predicted sentence is compared to the reference—no decoding rule, no mention of BLEU, nothing. The LSTM output is described as shape (None, 50, 10000), which looks like per-timestep softmax classification; if the 'accuracy' is just Keras's categorical accuracy on training batches, it says nothing about translation quality. Table III has recall numerically identical to accuracy for all three models, which is a red flag that the metric is actually something like token-level accuracy on a flattened sequence, not translation accuracy. The numbers also disagree: abstract says 89.3%, Fig. 6 says 89.7%, Table III says 0.8973. The dataset is private, and the few sample rows in Table I look garbled—if that's representative, even the data integrity is uncertain. The literature review is loosely connected and the citation numbering doesn't match the reference list.\n\nIs anything salvageable? The idea and the dataset, if cleaned and released, could support a real paper. But as submitted, the evaluation doesn't support the headline claim. I'd desk-reject this, not because the application is unimportant, but because the paper doesn't give a reviewer anything to verify. If the authors redo the evaluation with a proper text-generation metric, release the data, and clarify the model outputs, it could be a reasonable workshop paper.\n\nMy recommendation: don't send this to peer review in its current form. It's a useful negative example for a methods discussion, but it's not a sound empirical contribution.","headline":"The Sylheti-to-Bangla translation task is real, but the 89.3% accuracy claim is undefined, unreproducible, and likely a training/classification metric, not translation quality.","tokens_in":10410,"tokens_out":3145,"would_cite":false,"duration_ms":37799,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard LSTM, trained on 1,200 sentence pairs, is reported to translate Sylheti to Modern Bangla at 89.3% accuracy.","keywords":["Sylheti translation","Modern Bangla","LSTM","Bi-LSTM","Seq2Seq","low-resource NLP","Bangla dialects","neural machine translation"],"falsifier":"The claim would be settled by releasing the dataset and a fixed test split, then having independent Sylheti speakers evaluate the LSTM's outputs against their own reference translations; if the token-level match is not close to 0.8973, or if the published sample rows show the kind of corrupted Bengali characters seen in Table I, the reported result cannot be reproduced.","tokens_in":9352,"feed_emoji":"🔤","tokens_out":10673,"duration_ms":84470,"temperature":0.7,"pith_summary":"This paper tries to establish that a standard LSTM sequence model, trained on just 1,200 Sylheti–Modern Bangla sentence pairs, can translate the Sylheti dialect into Modern Bangla with a reported accuracy of 89.3 percent. The authors built a preprocessing pipeline, trained LSTM, Bi-LSTM, and Seq2Seq models, and found that the plain LSTM outperformed the other two on their evaluation. If the result holds, it would give low-resource Bangla dialect translation a simple baseline and a practical path to tools that help speakers of one Bangladeshi region understand another. The paper also frames the work as a step toward expanding Bangla NLP to local languages beyond the standard dialect.","feed_headline":"LSTM tops Sylheti-to-Modern Bangla translation at 89.3%","feed_subtitle":"Trained on just 1,200 sentence pairs, the simplest recurrent model edges out Bi-LSTM and Seq2Seq on the paper's tests.","key_machinery":"The central object is the LSTM as a token-level translator: Sylheti tokens are embedded into dense vectors, processed by a single LSTM layer with forget, input, and output gates, and projected to a vocabulary of 10,000 possible output tokens. The paper describes the standard LSTM gate equations and uses 50 epochs with batch size 32 on an 80/20 train-test split of the 1,200 pairs. The comparison architectures are a bidirectional LSTM with an added attention layer and an encoder–decoder Seq2Seq pair; the reported superiority of the plain LSTM is the mechanism the paper points to for why smaller recurrent models can work in this low-resource setting.","core_discovery":"On a dataset of 1,200 Modern Bangla sentences translated into Sylheti by speakers from the Sylhet region, an LSTM with 64-dimensional embeddings, 128 LSTM units, and a 10,000-token output vocabulary achieves the paper's best results: accuracy 0.8973, precision 0.8745, recall 0.8973, and F1 0.8838. The Bi-LSTM with attention reaches 0.7625 accuracy and the Seq2Seq encoder–decoder reaches 0.7069. The authors conclude that for this small parallel corpus, the plain LSTM is the best fit, and they attribute its advantage to a smaller train–validation gap than the other two models.","pith_inferences":["The reported 'accuracy' is a classification-style metric (TP/TN/FP/FN) applied to sequence generation; a standard machine-translation evaluation such as BLEU or human judgment would be needed to compare these numbers with other translation systems.","Because the metric counts tokens, a sentence with one wrong word is only partially counted; sentence-level exact-match accuracy is likely to be noticeably lower than 89.3 percent.","A testable extension would be releasing the 1,200-pair corpus plus a fixed test split, so the same LSTM can be re-run, scored by BLEU, and checked by independent Sylheti speakers.","The paper's future plan to include other regional languages could be helped by this same pipeline, but only if the collection and alignment protocol is documented and the data are shared."],"forward_implications":["A 1,200-pair corpus is enough to make a plain LSTM the best of the three tested models, so future Sylheti translation work can start from this architecture instead of requiring large pretrained models.","The preprocessing pipeline of punctuation removal, stemming, padding, and tokenization can be reused for other Bangladeshi local languages such as Chittagong.","The authors state that scaling up the dataset and trying hybrid models are the natural next steps; their result implies data volume, not architecture, is the main bottleneck.","If the 89.3 percent figure transfers to new sentences, a simple LSTM-based application could give non-Sylheti speakers a usable translation tool for everyday sentences."],"supporting_citations":[],"fun_headline_variants":["Plain LSTM beats Bi-LSTM and Seq2Seq for Sylheti-Bangla","89.3% accuracy: LSTM wins Sylheti-to-Modern Bangla translation","1,200 sentences, LSTM best for Sylheti to Bangla","Simple LSTM tops complex models at 89.3% for Sylheti"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 1,200 Sylheti–Modern Bangla sentence pairs were correctly translated by Sylheti speakers and accurately aligned, since the reported 89.3 percent accuracy is computed entirely against this unreleased dataset.","fun_headline_variants_meta":{"raw":{"variants":["Plain LSTM beats Bi-LSTM and Seq2Seq for Sylheti-Bangla","89.3% accuracy: LSTM wins Sylheti-to-Modern Bangla translation","1,200 sentences, LSTM best for Sylheti to Bangla","Simple LSTM tops complex models at 89.3% for Sylheti"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1251,"prompt_tokens":874,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":286}},"tokens_in":490,"tokens_out":377,"duration_ms":3119,"temperature":1.0,"reasoning_tokens":286,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:27:18.321896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The claim would be settled by releasing the dataset and a fixed test split, then having independent Sylheti speakers evaluate the LSTM's outputs against their own reference translations; if the token-level match is not close to 0.8973, or if the published sample rows show the kind of corrupted Bengali characters seen in Table I, the reported result cannot be reproduced.","supporting_citations":[],"review_version":1}