{"id":"45b89a87-97b0-4d1c-97a2-c16f488713f5","arxiv_id":"2507.20111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A LoRA-fine-tuned 8B language model with backtranslation and a dual-agent pipeline generates Old English text, with reported English-to-Old English BLEU rising from 25.94 to 65.41.","lead":"The authors built a machine-learning pipeline that generates new Old English sentences using a fine-tuned large language model and a two-agent writer-and-translator setup. They report large gains in translation quality scores and argue the same approach could expand digital corpora for other endangered languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may be inflated by test-set leakage: the paper never shows that evaluation sentences were excluded from domain adaptation, backtranslation, or synthetic fine-tuning data.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing issue: the evaluation depends on a clean separation between training and test data, and on independent expert scoring. My stress-test converges on the same point, but I would sharpen it as a concrete data-contamination risk. The paper's own methods section describes backtranslation using 'previously unseen fragments from the monolingual Old English corpus' and then fine-tuning on synthetic parallel pairs, yet it never states that these fragments were disjoint from the test split. Given the small size of the DOEC (about 3 million words), sentence-level or document-level overlap is a realistic possibility, and even partial overlap could explain a large share of the reported BLEU jump. The expert evaluation is similarly underspecified: PARCOROE, the likely evaluation resource, is edited by a co-author, and no evaluator names, qualifications, or inter-annotator agreement are reported. These are not allegations of misconduct; they are missing controls that are standard for low-resource MT evaluation. Because the concern is real but not yet confirmed, the appropriate verdict remains CONDITIONAL: the authors should release the split, deduplicate against all training stages, and disclose evaluator independence. My read therefore does not change the reader's verdict.","tokens_in":11291,"tokens_out":5043,"duration_ms":56333,"concrete_test":"Ask the authors to release the exact train/validation/test split and the code from the cited GitHub repository, then recompute Table 5 after removing from all training stages (DAPT corpus, backtranslation source fragments, and synthetic parallel pairs) any sentence that overlaps with a test sentence at the 5-gram level or via exact sentence matching. If BLEU/CHRF/METEOR drop by more than a few points under this deduplicated evaluation, or if overlap is substantial, the reported 65.41 BLEU is inflated. For the human evaluation, request evaluator names and affiliations or a third-party scoring; if the same research group scored its own outputs, Table 7 cannot confirm the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that OldEnglishRefined generates high-quality Old English, rests on the automated scores in Table 5 (BLEU 65.41, CHRF 57.82, METEOR 57.40) and the expert ratings in Table 7. These numbers are only interpretable if the evaluation sentences were held out from every training stage. The paper does not establish this. In 'Data Preparation' the DOEC and PARCOROE data are 'split into training, validation, and testing subsets, with monolingual Old English texts separately prepared for data augmentation tasks such as back-translation', but no document-level or n-gram-level deduplication is reported. Phase 2 ('Task Specialization') then uses 'previously unseen fragments from the monolingual Old English corpus' to generate synthetic parallel pairs via backtranslation, and fine-tunes OldEnglishRefined on those pairs plus human-annotated examples. If those monolingual fragments or the PARCOROE examples overlap with test documents, the improvement from BLEU 25.94 to 65.41 could reflect memorization rather than generalization. The paper also does not identify the test set (DOEC? PARCOROE?) or the evaluators' affiliations; PARCOROE [3] is edited by co-author J. Martín Arista, so the human scores are not demonstrably independent. Because every headline result depends on a clean split, this is the most load-bearing concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a framework for generating Old English text using a multi-stage pipeline: domain-adaptive continual pretraining with LoRA, backtranslation-based data augmentation, and a dual-agent architecture (FragmentGen generating English content and OldEnglishTranslator translating to Old English). The authors report large automated-metric gains (EN→ANG BLEU rising from ~26 to 65.41 in Table 5) and high expert ratings (e.g., 9.0 for inflection, 9.1 for lexical choice in Table 7). The central claim is that the resulting OldEnglishRefined model produces fluent, grammatically accurate Old English suitable for corpus expansion, and that the framework is a replicable blueprint for other low-resource languages.","tokens_in":11553,"tokens_out":3912,"duration_ms":37194,"significance":"If the reported results are valid, the framework is a valuable practical contribution to low-resource language processing and digital humanities. The paper's strengths include a reproducible code repository, a clearly structured pipeline (LoRA, backtranslation, dual-agent prompting), a wide range of qualitative examples, and an explicit discussion of limitations. The reported gains are substantial and the human-evaluation scores are high. However, the significance is currently capped by unresolved evaluation threats: possible test-set leakage across training stages, inconsistent baseline scores across tables, and a human evaluation whose independence is not established. These issues must be resolved before the headline numbers can be trusted.","major_comments":[{"comment":"The paper does not establish that the evaluation test set is disjoint from all data used in training. In 'Data Preparation', the DOEC and PARCOROE data are split into training, validation, and testing subsets, but no document-level or n-gram-level deduplication is reported. In 'Phase 2: Task Specialization', the model is fine-tuned on synthetic parallel pairs generated via backtranslation from 'previously unseen fragments from the monolingual Old English corpus', and in 'Synthetic Data Generation', the FragmentGen agent receives randomly sampled fragments from the DOEC as contextual anchors. If any of these fragments overlap with test sentences, the jump from BLEU 25.94 to 65.41 (Table 5) could reflect memorization rather than generalization. Please report explicitly whether test sentences were excluded from the monolingual backtranslation source, the few-shot anchors, and the domain-adaptation corpus, and provide overlap statistics (e.g., maximum n-gram overlap, document-level deduplication).","section":"Data Preparation; Phase 2: Task Specialization"},{"comment":"The baseline numbers are internally inconsistent. Table 2 reports Llama3.1 8B EN→ANG scores of BLEU 29.13, CHRF 25.51, METEOR 18.95, while Table 5 reports the same baseline (listed as 'Llama') with BLEU 25.94, CHRF 22.97, METEOR 17.43. Additionally, Table 4 shows OldEnglishBase at 3 epochs achieving EN→ANG BLEU 60.73, whereas Table 5 lists OldEnglishBase at BLEU 59.99. These discrepancies make the exact improvement magnitudes ambiguous and should be reconciled, with a clear statement of which configuration (checkpoint, epoch, evaluation set) each table reports.","section":"Tables 2 and 5"},{"comment":"The human evaluation lacks the transparency needed to support the reported high scores. The text mentions 'expert reviewers' and 'specialist linguists', but does not report the number of raters, their affiliations, whether they were independent of the authors, inter-rater agreement, or the number of generated texts rated per criterion. This matters particularly because the PARCOROE corpus [3] used for evaluation is edited by co-author J. Martín Arista, so the evaluation is not demonstrably independent. Please provide full details of the evaluation protocol and, ideally, an external evaluation by linguists not involved in the project.","section":"Expert Linguistic Evaluation; Table 7"},{"comment":"Automated scores are reported without error bars, significance tests, or characteristics of the test set (e.g., number of sentences, source corpus, genre distribution). Given the 2–4 point differences that are used to support claims about backtranslation gains and model specialization, the absence of variance estimates makes it difficult to assess whether these differences are meaningful. Please add test-set statistics and, if possible, bootstrap confidence intervals or multiple evaluation runs.","section":"Results and Discussion; Tables 4 and 5"}],"minor_comments":[{"comment":"The base model is referred to inconsistently as 'Llama-8b' and 'Llama3.1 8B'; please use a single, precise designation (with the parameter count, e.g., Llama-3.1-8B).","section":"Throughout"},{"comment":"The header contains 'Modelo' (Spanish) instead of 'Model'.","section":"Table 2"},{"comment":"The source sentence contains 'except.the’angild’to others, aud fortress-work', where 'aud' appears to be a typo for 'and'.","section":"Table 6"},{"comment":"The figure caption uses 'OE-Base' and 'OE-Refined', while the text uses 'OldEnglishBase' and 'OldEnglishRefined'; please align the terminology.","section":"Figure 3"},{"comment":"The phrase 'BLEU scores increasing from 26 to over 65' is consistent with Table 5 but not with Table 2's baseline of 29.13; clarify which baseline is referenced.","section":"Abstract"},{"comment":"There are minor typos such as 'demostrated' (should be 'demonstrated') and inconsistent capitalization of 'Contemporary English' / 'Modern English'.","section":"Results and Discussion"}],"recommendation":"major_revision","confidential_remarks":"The central result hinges entirely on the integrity of the evaluation protocol. In revision, the authors should be asked to provide the exact data splits, preprocessing code, and deduplication checks, as well as the full details of the human evaluation (rater count, independence, inter-rater agreement). The internal inconsistency between Tables 2 and 5 also needs a clear explanation. If the leakage concern cannot be resolved, the reported gains may be largely artifacts, which would move this toward rejection; but at present, the issues are potentially fixable within the revision scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2507.20111. First, it is a real engineering effort: the authors fine-tune Llama for English–Old English translation using domain-adaptive pretraining, LoRA, and backtranslation, and they report large jumps (BLEU from ~26 to 65 for EN→ANG) plus expert ratings around 9/10 for inflection, word order, and lexical choice. Second, the evaluation hygiene is not yet established, and that is the load-bearing issue.\n\nWhat is genuinely useful: the staged pipeline itself, with four well-chosen DAPT tasks, synthetic parallel pairs from backtranslation, and a dual-agent generation scheme (GPT-4o-mini for English prompts, then their own translator into Old English). The qualitative examples, especially the error table and the sample outputs in Table 6, show the model's progression and its remaining weaknesses. The limitations section is unusually candid: semantic coherence, anachronisms, V2 avoidance, and co-reference drift are all acknowledged. They also cite the right prior work and provide a GitHub link, though no code or data appears in the manuscript itself.\n\nThe soft spots are real but not necessarily fatal. The most serious is the lack of any demonstrated separation between the test set and the data used for domain adaptation, backtranslation, or the synthetic fine-tuning corpus. The paper says the data was split into train/validation/test, but it never shows document-level or n-gram deduplication, and the Phase 2 backtranslation uses \"previously unseen fragments from the monolingual Old English corpus\"—which may still overlap with test documents. If that overlap exists, the BLEU jump from 25.94 to 65.41 could be memorization, not generalization. Second, the numbers are internally inconsistent: Table 2 gives Llama3.1 8B EN→ANG BLEU 29.13, while Table 5 gives 25.94 for the same direction and model, and the base model is named both Llama-8b and Llama3.1 8B. Third, the expert evaluators are not disclosed, and the PARCOROE benchmark [3] is edited by co-author Martín Arista, so the human scores are not demonstrably independent. Fourth, there are no error bars, rater counts, test-set sizes, or hyperparameter values (LoRA rank, alpha, learning rate, etc.), which makes replication harder than it should be.\n\nThese are fixable in revision. The central approach is credible and consistent with prior low-resource MT work, and the authors are not trying to hide the model's shortcomings. But the paper as written overclaims because the evidence for a clean evaluation is missing.\n\nWho should read this: people working on low-resource NLP, ancient language preservation, or digital humanities workflows. A serious referee should engage with it and ask for the missing details—especially dataset split provenance, evaluator independence, and corrected tables—rather than letting it slide. I would accept it for peer review, but I would not cite it as a demonstrated result until those questions are answered.","headline":"A credible LoRA-plus-backtranslation pipeline for Old English whose headline numbers cannot be trusted until the authors prove the test set was held out from all training stages and the expert evaluators were independent.","tokens_in":12125,"tokens_out":1662,"would_cite":false,"duration_ms":18572,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an English-centric large language model can be turned into a fluent Old English writer by staging LoRA fine-tuning, backtranslation, and a dual-agent pipeline.","keywords":["low-resource languages","Old English","synthetic data generation","Low-Rank Adaptation (LoRA)","backtranslation","machine translation","text generation","digital preservation"],"falsifier":"Ask the released code and dataset to name every Old English sentence used in training, validation, and backtranslation, then recompute BLEU, CHRF, and METEOR on a test set whose sentences are provably disjoint from that pool; if the scores drop toward the base-model range, the reported gains are measurement artifacts. A stricter test is to have Old English scholars who have not seen the outputs rate them blind and flag anachronisms or grammatical errors without knowing the texts are machine-generated.","tokens_in":11064,"feed_emoji":"📜","tokens_out":9392,"duration_ms":86216,"temperature":0.7,"pith_summary":"This paper argues that the scarcity of Old English text is not a hard barrier to generating new Old English with large language models. The authors take an English-centric base model, adapt it to Old English with efficient fine-tuning on four translation-like tasks, expand the training data by translating Old English back into Modern English, and then pair the two directions into synthetic parallel sentences. A two-agent pipeline then drafts Modern English passages and translates them into Old English. Reported results are large jumps in automatic translation scores—BLEU from roughly 26 to over 65 for English-to-Old English—and expert ratings near 9 out of 10 for inflection, word order, and lexical choice, with semantic coherence weaker at 7.8. The broader promise is a replicable recipe for expanding corpora and supporting the study of other endangered or ancient languages.","feed_headline":"Old English from an English-centric LLM: BLEU jumps from 26 to 65","feed_subtitle":"LoRA, backtranslation, and a dual-agent pipeline produce Old English that experts rate near 9/10 for grammar and word choice.","key_machinery":"The load-bearing mechanism is the staged adaptation of a single English-centric language model rather than the addition of a separate Old English system. Low-Rank Adaptation (LoRA) keeps pretraining feasible by updating only low-rank weight matrices while the rest of the model stays fixed. Four parallel task templates—Old English text completion, Modern English-to-Old English translation, Old English-to-Modern English backtranslation, and Old English word definition—teach the model Old English grammar and vocabulary through its existing English knowledge. Backtranslation then manufactures synthetic parallel sentence pairs from monolingual Old English, and the dual-agent pipeline (an English fragment generator plus the refined Old English translator) scales this into a corpus-generation loop. The mechanism works by converting the scarcity of bilingual Old English data into an abundance of monolingual data that can be turned into parallel training examples.","core_discovery":"The central claim is that the low-resource bottleneck for Old English can be broken by reframing synthetic corpus expansion as machine translation. The authors start from a general-purpose English-dominant model and adapt it to Old English in two training phases: first, continual domain-adaptive pretraining over four tasks—completing Old English fragments, translating English into Old English, translating Old English into English, and giving English definitions of Old English words—using Low-Rank Adaptation so only a small set of parameters is updated; second, task specialization in which monolingual Old English is backtranslated into Modern English and the resulting pairs are added to the human-annotated parallel data. A final dual-agent stage separates content generation from translation: one component writes fresh Modern English fragments guided by Old English style samples, and a second translates them into Old English. The paper reports that this recipe lifts English-to-Old English BLEU from 25.94 to 65.41, with CHRF 57.82 and METEOR 57.40, and expert ratings of 9.0 for inflection, 9.0 for word order, 9.1 for lexical choice, and 7.8 for semantic coherence.","pith_inferences":["Editorial inference: if this framework is adopted for corpus building, synthetic Old English should be labeled as synthetic; otherwise future models trained on the expanded corpus will inherit the translator's own errors and anachronisms, and the corpus will become circular.","Editorial inference: because backtranslation creates training pairs from the model's own reverse translations, the quality ceiling of the synthetic data is set by the reverse-direction model; a natural stress test is to compare outputs trained on backtranslated data from the same model versus a stronger or ensemble reverse translator.","Editorial inference: the paper's semantic-coherence gap (7.8) suggests a concrete next experiment the authors did not run—retrieval-augmented generation grounded in authentic Old English passages should be tested against the current pipeline on coherence ratings, since the authors name RAG as future work but do not evaluate it.","Editorial inference: if the evaluation set is clean and the expert ratings are independent, this recipe may transfer to other ancient languages with related modern descendants, but the quality numbers themselves would need to be re-established for each new language through its own expert evaluation."],"forward_implications":["If the reported scores hold, English-to-Old English translation becomes a practical capability of an English-centric model: BLEU more than doubles relative to the base model, from about 26 to 65.","The backtranslation step buys roughly 6 additional points on top of domain adaptation, so synthetic parallel data is what turns a stagnating adapted model into a fluent translator.","The dual-agent pipeline can generate large volumes of stylistically anchored Old English, meaning corpus expansion for Old English can proceed at scale and only high-rated outputs need be retained.","High expert scores for inflection, word order, and lexical choice (9.0, 9.0, 9.1) with a lower semantic coherence score (7.8) imply that the generated texts are grammatically reliable, but narration-level coherence is the remaining bottleneck.","The same LoRA-plus-backtranslation-plus-dual-agent recipe is proposed as a transferable template for other low-resource or endangered languages with a small surviving corpus and a related modern language."],"supporting_citations":[{"why":"This corpus supplies the complete surviving written record of Old English, which is the source of monolingual training and backtranslation data.","marker":"[2]"},{"why":"This annotated parallel corpus provides Old English-to-English translation examples used for training and as evaluation references.","marker":"[3]"},{"why":"This work introduces domain- and task-adaptive pretraining, the general strategy the paper adapts to Old English.","marker":"[6]"},{"why":"This work supplies the efficient continual-pretraining approach that motivates the four-task adaptation design.","marker":"[7]"},{"why":"This reference provides the English-dominant 8-billion-parameter checkpoint from which all adaptation stages begin.","marker":"[8]"},{"why":"This work defines Low-Rank Adaptation, the parameter-efficient fine-tuning method that makes continual pretraining computationally feasible.","marker":"[9]"},{"why":"This work supplies the backtranslation technique used to synthesize parallel English–Old English training pairs.","marker":"[10]"},{"why":"This work defines BLEU, the primary automatic metric used to score translation quality.","marker":"[12]"},{"why":"This work defines METEOR, the recall-oriented metric used alongside BLEU in evaluation.","marker":"[13]"},{"why":"This work defines CHRF, the character-level metric used alongside BLEU and METEOR.","marker":"[14]"}],"fun_headline_variants":["Dual-agent LLM lifts Old English BLEU from 26 to 65","Reviving Old English: LLM framework scores 65 BLEU, experts approve","From 26 to 65 BLEU: Generating Old English with an English-centric LLM","Low-resource languages? This LLM framework writes Old English at BLEU 65","LoRA + backtranslation + dual-agent: Old English BLEU hits 65"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's reported quality numbers assume that the Old English sentences used for evaluation were never used in training, domain adaptation, or backtranslation, and that the expert ratings reflect independent Old English specialists rather than the authors themselves.","fun_headline_variants_meta":{"raw":{"variants":["Dual-agent LLM lifts Old English BLEU from 26 to 65","Reviving Old English: LLM framework scores 65 BLEU, experts approve","From 26 to 65 BLEU: Generating Old English with an English-centric LLM","Low-resource languages? This LLM framework writes Old English at BLEU 65","LoRA + backtranslation + dual-agent: Old English BLEU hits 65"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1844,"prompt_tokens":978,"completion_tokens":866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":751}},"tokens_in":594,"tokens_out":866,"duration_ms":7498,"temperature":1.0,"reasoning_tokens":751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:49:07.482086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask the released code and dataset to name every Old English sentence used in training, validation, and backtranslation, then recompute BLEU, CHRF, and METEOR on a test set whose sentences are provably disjoint from that pool; if the scores drop toward the base-model range, the reported gains are measurement artifacts. A stricter test is to have Old English scholars who have not seen the outputs rate them blind and flag anachronisms or grammatical errors without knowing the texts are machine-generated.","supporting_citations":[{"cited_title":"The dictionary of old english web corpus,","cited_arxiv_id":null,"evidence_quote":"This corpus supplies the complete surviving written record of Old English, which is the source of monolingual training and backtranslation data."},{"cited_title":"Parcoroev3: An open access annotated parallel corpus old english- english,","cited_arxiv_id":null,"evidence_quote":"This annotated parallel corpus provides Old English-to-English translation examples used for training and as evaluation references."},{"cited_title":"Efficient continual pre-training for building domain specific large language models,","cited_arxiv_id":null,"evidence_quote":"This work supplies the efficient continual-pretraining approach that motivates the four-task adaptation design."}],"review_version":1}