{"id":"219c2afd-a6d2-4669-aa2c-66ed05cfc558","arxiv_id":"2412.05184","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"QueEn reports BLEU 17.6 for Quechua-English translation, but its internal table shows BLEU 0.235, the method is not reproducible, and the translation direction is inconsistent.","lead":"The paper proposes QueEn, a system that combines retrieval-augmented generation with LoRA fine-tuning for Quechua-English translation, claiming BLEU 17.6 versus 1.5 for standard GPT models. The paper's own tables and text contradict those headline numbers, and the experimental setup is too underspecified to verify the claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proposed RAG+LoRA model is never evaluated: Table 1 has no combined RAG+LoRA row, and the abstract's BLEU 17.6/1.5 does not appear in the table, so the central claim is unsupported.","rationale":"The reader's strongest claim—that the abstract's BLEU 17.6 vs. 1.5 contradicts Table 1—is valid and directly observable. I depart slightly by identifying the missing combined RAG+LoRA evaluation as the single most load-bearing issue: even if the abstract numbers were corrected, the paper still does not test the system it claims to propose. Section 3 promises an integration of LoRA fine-tuning and RAG, but Table 1 separates the components, with no row combining them. The near-zero BLEU scores also raise doubts about reference construction, and the stated evaluation direction ('English-to-Quechua') conflicts with the title ('Quechua-English'), but the absence of the combined model is the decisive technical gap. This does not change the reader's REJECT verdict; it adds a more fundamental reason. No code, data, or training details are provided, so the missing configuration cannot be verified post hoc.","tokens_in":10953,"tokens_out":4564,"duration_ms":45165,"concrete_test":"Run the exact combined configuration: take the same LLaMA base model, apply the LoRA fine-tuning procedure described in Section 3.1, then augment inference with the RAG pipeline of Section 3.2, and evaluate on the same test set and references used for Table 1. If this configuration is not reported, or its BLEU/ROUGE/BERTScore do not exceed the component rows, the central claim is unsupported. As a secondary check, recompute BLEU for every row from raw model outputs to reconcile the abstract values 17.6/1.5 with Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is that the paper's proposed model is never evaluated. Section 3 defines QueEn as a framework that combines LoRA fine-tuning with RAG, but Table 1 reports only LLaMA, GPT, LLaMA+finetune, LLaMA+RAG, and GPT+RAG; no row instantiates the combined RAG+LoRA configuration. Thus the central claim that the integration 'substantially exceeds' baselines is untested. This is compounded by the abstract's BLEU 17.6 vs. 1.5, which matches no entry in Table 1: the best BLEU in the table is 0.235 for GPT+RAG, and plain GPT is 0.096. Even the strongest table result is RAG-only, and all reported BLEU/ROUGE scores are near zero. If the combined configuration was actually run, both the setup and the output must be reported; as written, the experimental section does not connect the claimed result to the proposed method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QueEn, a system for Quechua-English translation that combines Retrieval-Augmented Generation (RAG) with Low-Rank Adaptation (LoRA) fine-tuning. The authors use the Siminchik corpus, generate English references from Spanish translations with ChatGPT-4, and compare GPT-4o and LLaMA variants with and without RAG and fine-tuning. The abstract claims a BLEU score of 17.6 versus 1.5 for standard GPT models, but Table 1 reports BLEU scores between 0.038 and 0.235, and no experimental row instantiates the combined RAG+LoRA configuration described in Section 3. The manuscript therefore does not currently support its central claim.","tokens_in":11201,"tokens_out":3785,"duration_ms":37462,"significance":"Quechua-English translation is an important and underexplored low-resource task, and the combination of RAG with parameter-efficient fine-tuning is a reasonable direction to investigate. If the claimed results were supported, the paper would be a useful contribution to low-resource MT. However, the current manuscript has no machine-checked proofs, no released code or data, and, critically, does not evaluate the proposed combined model. The significance of the contribution is therefore not yet established.","major_comments":[{"comment":"The abstract's central quantitative claim, \"a BLEU score of 17.6 compared to 1.5 for standard GPT models,\" is not supported by Table 1. The table reports BLEU scores of 0.235 for GPT + RAG and 0.096 for plain GPT, with no double-digit BLEU anywhere. The abstract and the experimental table describe different results, and the paper gives no explanation for the discrepancy.","section":"Abstract and Table 1"},{"comment":"The proposed QueEn framework is defined in Section 3 as combining LoRA fine-tuning with RAG, yet Table 1 contains no row for the combined RAG+LoRA configuration. The rows are LLaMA, GPT, LLaMA + finetune, LLaMA + RAG, and GPT + RAG. The central claim that the integration of RAG with fine-tuning \"substantially exceeds\" baselines is therefore never tested by the reported experiments. A row labeled \"ours\" that instantiates the full method must be added, or the paper must clearly state which existing row corresponds to QueEn.","section":"Section 3 and Table 1"},{"comment":"The experimental setup conflates translation directions. The dataset is described as Quechua-Spanish (\"que spa clean\"), and Section 4.1 states that ChatGPT-4 is used to translate Spanish into English, but the following sentence says this \"enables us to evaluate English-to-Quechua translation scenarios.\" The title and abstract promise Quechua-English translation. The paper must specify the actual source and target language in the evaluation, and the metric computation must match that direction.","section":"Section 4.1"},{"comment":"The reference translations are machine-generated: ChatGPT-4 produced the English references from Spanish, and GPT-based models are then scored against those references. This creates a circularity risk because the reference may share systematic stylistic and lexical biases with the models being evaluated. The authors should validate the ChatGPT-4 references against human translations or an independent reference set, or report inter-annotator agreement.","section":"Section 4.1 and Table 1"},{"comment":"Section 4.2 says a \"combination of automated and human evaluation metrics\" was employed, but no human evaluation setup, protocol, or results appear anywhere in the paper. Additionally, the experimental section omits load-bearing details needed for reproducibility: the specific base model versions (e.g., which LLaMA and GPT variants), LoRA rank r and other LoRA hyperparameters, training hyperparameters, retrieval top-K, embedding model used in RAG, and the train/validation/test split sizes. These omissions prevent verification of the reported results.","section":"Section 4.2 and Section 4.3"}],"minor_comments":[{"comment":"There is a typo in Section 4.1: \"e applied ChatGPT-4\" should read \"We applied ChatGPT-4.\"","section":"Section 4.1"},{"comment":"The metric name is rendered as \"BER TScore\" in Section 4.2; it should be \"BERTScore.\"","section":"Section 4.2"},{"comment":"Table 1 uses inconsistent capitalization, e.g., \"LLama\" for LLaMA and \"Rouge Score\" for ROUGE; please standardize the notation.","section":"Table 1"},{"comment":"In Section 2.5, the text refers to \"the QueEn project\" and cites reference [43], but the cited work by Ortega et al. is about neural machine translation with a polysynthetic language, not the QueEn project. This citation should be corrected or removed.","section":"Section 2.5"},{"comment":"Figure 1's caption says the retrieved information is \"processed by a GPT-4 model,\" but the proposed method in Section 3 is described as a general LLM enhanced with LoRA; clarify which model the figure depicts.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The gap between the abstract's BLEU claim and Table 1 is severe enough to suggest the abstract may have been written for a different set of experiments. The absence of any evaluation of the combined RAG+LoRA method makes the core contribution untested as written. I recommend rejection; if the authors can supply the missing experiments and correct the internal inconsistencies, a fresh submission may be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is an application of RAG and LoRA to Quechua-English translation, and the application is sensible for a genuinely underserved language, but the experiments don't test the proposed method. The abstract claims BLEU 17.6 vs 1.5. Table 1 shows GPT+RAG at 0.235 and plain GPT at 0.096. No row combines RAG and LoRA. The headline result is not in the results.\n\nWhat's worth crediting: the paper identifies real challenges in Quechua MT—agglutinative morphology, scarce parallel corpora, dialect variation—and the literature review is thorough. The retrieval setup is reasonable, and the LoRA formulation is standard. A reader wanting a quick orientation to the language's MT difficulties gets something here.\n\nThe soft spots are not minor. First, the proposed QueEn model is never evaluated; 'LLaMA + finetune' appears, but no RAG+LoRA configuration. Second, the abstract's numbers match nothing in the table. Third, the translation direction is inconsistent: the title says Quechua-English, while Section 4.2 describes English-to-Quechua. Fourth, the English references are ChatGPT-4's Spanish-to-English translations, unvalidated machine output serving as gold. Fifth, no code, data, hyperparameters, or retrieval details are provided, so the results are not reproducible. The human evaluation mentioned in Section 4.2 is never described.\n\nThis is an application note without a working evaluation. The central claim is untested and the reported numbers contradict each other. I wouldn't cite it, and a serious editor should desk reject unless the authors redo the experiments first.","headline":"The abstract claims BLEU 17.6 for a RAG+LoRA model, but Table 1 never tests that model and the best BLEU is 0.235; the paper's central result is unsupported.","tokens_in":11724,"tokens_out":2734,"would_cite":false,"duration_ms":25181,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval-augmented, low-rank fine-tuned LLM lifts Quechua-English translation beyond zero-shot baselines, the authors report.","keywords":["Quechua","machine translation","retrieval-augmented generation","parameter-efficient fine-tuning","LoRA","low-resource languages","endangered languages","BLEU"],"falsifier":"Recompute BLEU, ROUGE, and BERTScore on the Siminchik test set using human-translated English references instead of the machine-generated ones; if the RAG + LoRA models no longer beat zero-shot baselines, the central claim fails. Also verify whether the abstract's BLEU of 17.6 can be reproduced from the released test set, since Table 1 reports 0.235 for the best system.","tokens_in":10789,"feed_emoji":"🗣️","tokens_out":10408,"duration_ms":88092,"temperature":0.7,"pith_summary":"This paper argues that combining retrieval-augmented generation (RAG) with Low-Rank Adaptation (LoRA) fine-tuning turns a large language model into a markedly better Quechua-English translator without full retraining. The authors build QueEn, which retrieves dictionary entries and grammar snippets for each query and feeds them into a fine-tuned LLM, and report that RAG-augmented models beat zero-shot GPT-4o and LLaMA 3.1 405B on BLEU, ROUGE, and BERTScore. The method matters because Quechua has only about 14,000 parallel sentence pairs, so any approach that exploits existing dictionaries and grammar guides could extend machine translation to many endangered languages.","feed_headline":"Retrieval and LoRA lift Quechua-English translation","feed_subtitle":"A RAG-augmented, low-rank fine-tuned model beats zero-shot GPT and LLaMA on an endangered-language task.","key_machinery":"The load-bearing mechanism is the retrieval-augmented generation pipeline with a dual index. For each query $q$, the system retrieves exact matches $D_q^{\\text{keyword}} = \\{d \\in D \\mid q \\cap \\operatorname{Keywords}(d) \\neq \\emptyset\\}$ and semantic matches $D_q^{\\text{vector}}$ from the top $K$ documents by cosine similarity, then combines them as $D_q = D_q^{\\text{keyword}} \\cup D_q^{\\text{vector}}$ and builds a prompt $P$ that includes the retrieved lexical and grammatical content. Low-Rank Adaptation (LoRA) is the parameter-efficient fine-tuning component: it approximates the weight update as $\\Delta W = A \\cdot B$ with $A \\in \\mathbb{R}^{d \\times r}$, $B \\in \\mathbb{R}^{r \\times k}$, and $r \\ll \\min(d,k)$, so only the small matrices $A$ and $B$ are trained. This lets a general-purpose LLM access explicit Quechua dictionary and grammar knowledge at inference time while keeping adaptation cheap.","core_discovery":"The central claim is that retrieval and low-rank adaptation are complementary for low-resource translation: retrieval supplies lexical and grammatical knowledge the model lacks, while LoRA adapts the model's behaviour with few trainable parameters. The system indexes a Quechua-English dictionary and grammar guide using both keyword matching and embedding similarity, then constructs a prompt $P = \\operatorname{ConstructPrompt}(q, D_q)$ from the retrieved set and generates a translation $R = \\operatorname{LLM}(P)$. In the paper's experiments, GPT + RAG reaches the highest scores on all three metrics (BLEU 0.235, ROUGE 0.278, BERTScore 0.963), and the authors interpret the gains over plain GPT and LLaMA as evidence that the retrieval mechanism compensates for Quechua's agglutinative morphology and scarce data.","pith_inferences":["The evaluation references are machine-translated from Spanish to English by a GPT model; if those translations carry systematic bias, the reported ranking is relative to an artificial target, and a human reference set could change it.","The abstract's BLEU of 17.6 is an order of magnitude above the best table value (0.235); reconciling this discrepancy is necessary before the headline number can be taken as the method's true performance.","A natural transfer test would apply the same pipeline to another endangered language with a dictionary, such as Aymara or Guaraní, to see whether the gains come from the method or from Quechua-specific properties.","The pipeline could be extended with morpheme-aware tokenization, since Quechua suffixes carry grammatical roles that dictionary retrieval alone may not disambiguate."],"forward_implications":["If the paper's claim holds, the same RAG + LoRA recipe should improve LLM translation for other low-resource languages that have dictionaries and grammar guides but little parallel text.","The dual retrieval mechanism (exact keyword plus semantic embedding) should help the model handle Quechua's agglutinative morphology by surfacing relevant morpheme-level dictionary entries.","Because BERTScore stays high even when BLEU is low, the results suggest retrieval mainly improves lexical selection rather than overall semantic content, pointing to where future effort should focus.","Parameter-efficient fine-tuning keeps adaptation feasible for communities with limited computational resources, which is a concrete step toward language preservation tools."],"supporting_citations":[{"why":"Supplies the Low-Rank Adaptation method used for parameter-efficient fine-tuning.","marker":"[49]"},{"why":"Defines retrieval-augmented generation, the core retrieval mechanism the paper adapts for translation.","marker":"[40]"},{"why":"Provides the Siminchik Quechua corpus that supplies the evaluation data.","marker":"[50]"},{"why":"Hosts the IWSLT2023 low-resource speech translation track from which the que-spa-clean dataset is taken.","marker":"[43]"},{"why":"The LINGOLLM baseline that also uses linguistic descriptions in prompts, which the paper compares against conceptually.","marker":"[46]"},{"why":"Defines the ROUGE metric used for n-gram overlap scoring.","marker":"[51]"},{"why":"Defines the BLEU metric used to score translation quality.","marker":"[52]"},{"why":"Defines the BERTScore metric used to measure semantic similarity.","marker":"[53]"}],"fun_headline_variants":["RAG + LoRA boost Quechua-English translation","Low-resource translation gets a RAG boost","Quechua-English model blends retrieval and fine-tuning","Retrieval-augmented LoRA beats GPT on Quechua","Endangered language translation via RAG and LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that ChatGPT-4's Spanish-to-English translations, generated in Section 4.1, are accurate enough to serve as ground-truth references for Quechua-English translation; if those references are wrong, the reported scores and rankings are not trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["RAG + LoRA boost Quechua-English translation","Low-resource translation gets a RAG boost","Quechua-English model blends retrieval and fine-tuning","Retrieval-augmented LoRA beats GPT on Quechua","Endangered language translation via RAG and LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2591,"prompt_tokens":882,"completion_tokens":1709,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1629}},"tokens_in":498,"tokens_out":1709,"duration_ms":12081,"temperature":1.0,"reasoning_tokens":1629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:48:56.918588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute BLEU, ROUGE, and BERTScore on the Siminchik test set using human-translated English references instead of the machine-generated ones; if the RAG + LoRA models no longer beat zero-shot baselines, the central claim fails. Also verify whether the abstract's BLEU of 17.6 can be reproduced from the released test set, since Table 1 reports 0.235 for the best system.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the core retrieval mechanism the paper adapts for translation."},{"cited_title":"Siminchik: A Speech Corpus for the Quechua Language","cited_arxiv_id":null,"evidence_quote":"Provides the Siminchik Quechua corpus that supplies the evaluation data."},{"cited_title":"Neural machine translation with a polysynthetic low resource language","cited_arxiv_id":null,"evidence_quote":"Hosts the IWSLT2023 low-resource speech translation track from which the que-spa-clean dataset is taken."},{"cited_title":"Hire a linguist!: Learning endangered languages in LLMs with in-context linguistic descriptions","cited_arxiv_id":null,"evidence_quote":"The LINGOLLM baseline that also uses linguistic descriptions in prompts, which the paper compares against conceptually."},{"cited_title":"Recall-oriented understudy for gisting evaluation (rouge)","cited_arxiv_id":null,"evidence_quote":"Defines the ROUGE metric used for n-gram overlap scoring."},{"cited_title":"Bleu: a method for automatic evaluation of machine translation","cited_arxiv_id":null,"evidence_quote":"Defines the BLEU metric used to score translation quality."}],"review_version":1}