{"id":"1f1c4ca6-dd72-4b2e-96a1-35a4f05cb29f","arxiv_id":"2605.03742","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Releases Tajik Web Corpus (~1.11B characters) and finds Mistral 7B with QLoRA rank 16 yields lowest perplexity of 5.03 for Tajik generation while full fine-tuning on small models causes forgetting.","lead":"This paper releases the largest open Tajik web corpus of 319k documents and benchmarks 17 fine-tuning setups across model types and PEFT methods on a 10k subsample for Tajik text generation. A smart generalist might read it to see concrete efficiency trade-offs when adapting large models to languages with scarce digital data.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Perplexity on a random 10k subsample may not reliably indicate Tajik generation quality without human or task-based validation.","rationale":"The reader's weakest_assumption directly identifies the evaluation gap. Since the full text follows the abstract's described methodology, this remains the primary risk to the benchmarking conclusions. No other internal inconsistency noted.","tokens_in":1808,"tokens_out":321,"duration_ms":56196,"concrete_test":"Re-evaluate the top three configurations on a stratified 5k-document hold-out set drawn to match the full corpus's document-length and topic distribution (if available); recompute mean perplexity and add a small human study on 50 generations per model for coherence. If the QLoRA advantage disappears or human scores do not correlate with PPL, the central claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Mistral-7B QLoRA r=16 at mean PPL 5.03) is presented as best for Tajik text generation. This depends on two unverified conditions: (1) the 10,000-document random subsample faithfully represents the full 319k-document corpus distribution (no Kolmogorov-Smirnov test or domain breakdown reported), and (2) lower perplexity translates to better generation (no BLEU, human ratings, or downstream metrics like question-answering on Tajik). The abstract notes catastrophic forgetting for full fine-tuning but provides no quantitative measure. If either assumption fails, the ranking of PEFT methods could change.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces the Tajik Web Corpus (319,298 documents, ~1.11B characters) as the largest open Tajik resource and benchmarks 17 PEFT configurations (full fine-tuning, LoRA, QLoRA at ranks 8/16) across autoregressive, encoder-decoder, and encoder-only models on a 10k-document random subsample. Quality is measured via perplexity and cross-entropy loss, with auxiliary GPU memory and training time metrics. The central empirical claim is that Mistral-7B with QLoRA (r=16) yields the best result (mean perplexity 5.03, std 0.03), while full fine-tuning on small GPT-2 models produces lower perplexity but causes catastrophic forgetting; the work offers practical recommendations for low-resource Tajik generation.","tokens_in":1971,"tokens_out":728,"duration_ms":49901,"significance":"If the benchmarking results hold under more rigorous validation, the paper contributes by publicly releasing the largest verified Tajik corpus and delivering the first systematic PEFT comparison for Tajik generative modeling. This supports low-resource language research, with explicit strengths in corpus scale, the breadth of 17 configurations, and reported standard deviations across runs. The practical recommendations on architecture and strategy selection (balancing quality against memory) are potentially useful for similar Cyrillic low-resource settings.","major_comments":[{"comment":"Methods / Data section: The experiments rely on a random 10,000-document subsample drawn from the full 319,298-document corpus, yet no sampling procedure, deduplication details, train-test split criteria, domain stratification, or statistical representativeness test (e.g., Kolmogorov-Smirnov on token distributions) is described. This assumption is load-bearing for the generalizability of all reported perplexity rankings and the claim that QLoRA r=16 is optimal.","section":"Methods / Data Preparation"},{"comment":"Evaluation section: Perplexity and cross-entropy are the sole quality metrics; no human ratings, BLEU/ROUGE, or downstream task performance (e.g., Tajik QA or summarization) is reported. Because the goal is text generation, next-token perplexity alone does not establish that lower values correspond to better generation quality or coherence, directly affecting the headline ranking of Mistral QLoRA.","section":"Evaluation / Results"},{"comment":"Results section: The paper states that full fine-tuning on GPT-2 models induces catastrophic forgetting, but provides no quantitative measure (e.g., degradation on a held-out pre-training task, generation diversity metrics, or comparison of output entropy). This omission weakens the contrast drawn between full fine-tuning and PEFT strategies.","section":"Results / Discussion"}],"minor_comments":[{"comment":"Abstract: The claim of 'statistically insignificant improvement' when increasing LoRA rank from 8 to 16 is stated without naming the test, degrees of freedom, or p-value threshold used.","section":"Abstract"},{"comment":"Tables: Ensure every table reporting mean perplexity and standard deviation explicitly states the number of independent runs or random seeds underlying the statistics.","section":"Tables"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical benchmarking study well within the journal's scope for low-resource NLP; the main concerns are addressable by expanding the methods and evaluation sections without changing the core experimental design."},"author_rebuttal":{"model":"grok-4.3","summary":"We appreciate the referee's thorough review and constructive feedback on our manuscript. Below, we provide detailed responses to each major comment, indicating the revisions we plan to make to address the concerns.","responses":[{"response":"We thank the referee for highlighting this important point regarding reproducibility and generalizability. In the original manuscript, the subsample was selected as a simple random sample of 10,000 documents from the full corpus to enable efficient benchmarking while maintaining scale. To address the concern, we will revise the Methods section to explicitly describe the sampling procedure, including the use of a fixed random seed for reproducibility, the absence of additional deduplication beyond the corpus-level cleaning already performed (as described in Section 3), and the train-validation-test split ratios (80/10/10). We will also include a brief analysis of representativeness by comparing key statistics such as average document length and character distribution between the subsample and the full corpus. While a full Kolmogorov-Smirnov test on token distributions was not performed due to computational constraints, we will add a note acknowledging this limitation and justifying the random sampling approach for this initial benchmarking study. These changes will be incorporated in the revised version.","revision_made":"yes","referee_comment":"[Methods / Data Preparation] Methods / Data section: The experiments rely on a random 10,000-document subsample drawn from the full 319,298-document corpus, yet no sampling procedure, deduplication details, train-test split criteria, domain stratification, or statistical representativeness test (e.g., Kolmogorov-Smirnov on token distributions) is described. This assumption is load-bearing for the generalizability of all reported perplexity rankings and the claim that QLoRA r=16 is optimal."},{"response":"We agree that perplexity alone does not fully capture generation quality, particularly coherence and semantic correctness in text generation tasks. Perplexity was selected as the primary metric because it directly measures the model's predictive performance on the language modeling objective, which is central to our benchmarking of fine-tuning strategies for low-resource settings. To strengthen the evaluation, we will add a new subsection in the Results discussing the limitations of perplexity and include qualitative examples of generated text from the top-performing models in an appendix to illustrate coherence. We note that human evaluations or downstream task benchmarks (such as QA) would provide valuable additional insights but were beyond the scope of this work due to the lack of existing Tajik evaluation datasets and resources for annotation. We will update the Discussion to recommend these as future work.","revision_made":"partial","referee_comment":"[Evaluation / Results] Evaluation section: Perplexity and cross-entropy are the sole quality metrics; no human ratings, BLEU/ROUGE, or downstream task performance (e.g., Tajik QA or summarization) is reported. Because the goal is text generation, next-token perplexity alone does not establish that lower values correspond to better generation quality or coherence, directly affecting the headline ranking of Mistral QLoRA."},{"response":"The claim of catastrophic forgetting in fully fine-tuned small models was based on observed degradation in generation quality, where outputs became repetitive and lost Tajik-specific characteristics, contrasting with the coherent outputs from PEFT methods. However, we acknowledge that this was not supported by quantitative metrics in the original submission. We will revise the Results section to include quantitative evidence, such as measuring the entropy of generated sequences and the proportion of unique n-grams to demonstrate reduced diversity in full fine-tuning cases. Additionally, we will compare performance on a small held-out set of English text to show forgetting of pre-trained capabilities. These additions will provide a more rigorous contrast between full fine-tuning and PEFT approaches.","revision_made":"yes","referee_comment":"[Results / Discussion] Results section: The paper states that full fine-tuning on GPT-2 models induces catastrophic forgetting, but provides no quantitative measure (e.g., degradation on a held-out pre-training task, generation diversity metrics, or comparison of output entropy). This omission weakens the contrast drawn between full fine-tuning and PEFT strategies."}],"tokens_in":1662,"tokens_out":864,"duration_ms":72466,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The clearest addition is the Tajik Web Corpus of 319k documents and 1.11 billion characters, now public. That fills a real gap for a low-resource Cyrillic language. They also ran 17 fine-tuning setups on a 10k-document slice, comparing full tuning against LoRA and QLoRA across Mistral, GPT-2 sizes, and XLM-RoBERTa, while logging memory and time. The reported numbers show Mistral-7B with QLoRA rank 16 at 5.03 mean perplexity with tight standard deviation, and they flag that full fine-tuning on smaller models can beat PEFT on perplexity but triggers forgetting.","headline":"The paper releases a sizable new Tajik corpus and runs straightforward PEFT benchmarks on it, but the results rest on a narrow subsample and perplexity alone.","tokens_in":2432,"tokens_out":213,"would_cite":false,"duration_ms":47456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Mistral 7B with QLoRA rank 16 reaches mean perplexity 5.03 on Tajik text generation after release of the largest open Tajik web corpus.","keywords":["Tajik language","low-resource languages","parameter-efficient fine-tuning","QLoRA","text generation","large language models","perplexity","web corpus"],"falsifier":"Human raters assigning low quality scores to text samples generated by the Mistral 7B QLoRA model, or a new independent collection of Tajik documents producing substantially higher perplexity under the same fine-tuned model.","tokens_in":2710,"feed_emoji":"🌐","tokens_out":753,"duration_ms":65119,"temperature":0.7,"pith_summary":"This paper addresses data scarcity for Tajik, a low-resource language, by building and releasing the Tajik Web Corpus of 319,298 documents containing about 1.11 billion characters. It then runs a benchmark of 17 fine-tuning setups across autoregressive, encoder-decoder, and encoder-only models using full fine-tuning, LoRA, and QLoRA. The strongest result comes from Mistral 7B with QLoRA at rank 16, recording mean perplexity of 5.03 with low variance. The work shows that parameter-efficient methods can adapt large models to Tajik without prohibitive compute costs, while full fine-tuning on smaller models produces lower perplexity but triggers catastrophic forgetting. Encoder-only models perform markedly worse under the same evaluation.","feed_headline":"Mistral 7B QLoRA hits 5.03 perplexity on Tajik text","feed_subtitle":"New 1.1-billion-character Tajik corpus powers first systematic PEFT benchmark for this low-resource language","key_machinery":"The Tajik Web Corpus of 319,298 documents used as training data together with perplexity and cross-entropy evaluation across full fine-tuning, LoRA, and QLoRA applied to models from the GPT-2 family through Mistral 7B.","core_discovery":"By constructing the Tajik Web Corpus the author enables the first systematic comparison of parameter-efficient fine-tuning strategies for Tajik text generation, finding that Mistral 7B equipped with QLoRA at rank 16 attains the lowest mean perplexity of 5.03 on a 10,000-document subsample while remaining computationally practical.","pith_inferences":["The same corpus-construction and benchmarking approach could be replicated for other low-resource languages that have substantial web presence.","Perplexity results would be strengthened by adding human evaluation or downstream task metrics such as Tajik question answering.","The demonstrated viability of QLoRA suggests that similar efficiency gains are possible when adapting models to other Cyrillic-script low-resource languages."],"forward_implications":["QLoRA at rank 16 provides the best observed trade-off between perplexity and memory use for Mistral 7B on Tajik.","Increasing LoRA rank from 8 to 16 produces statistically insignificant perplexity gains at higher memory cost.","Full fine-tuning of GPT-2 family models yields lower perplexity than LoRA but induces catastrophic forgetting.","Encoder-only models such as XLM-RoBERTa deliver the highest perplexity and are least suitable for the generative task."],"fun_headline_variants":["Mistral 7B QLoRA at 5.03 perplexity on Tajik text","Tajik Web Corpus benchmarks 17 PEFT configurations","QLoRA fine-tuning tested on Mistral for Tajik","New corpus of 1.11B chars for Tajik LLM experiments"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"A random 10,000-document subsample from the web corpus represents the overall distribution of Tajik text and that perplexity alone measures generation quality adequately.","fun_headline_variants_meta":{"raw":{"variants":["Mistral 7B QLoRA at 5.03 perplexity on Tajik text","Tajik Web Corpus benchmarks 17 PEFT configurations","QLoRA fine-tuning tested on Mistral for Tajik","New corpus of 1.11B chars for Tajik LLM experiments"]},"model":"grok-4.3","cost_usd":0.018507,"raw_usage":{"total_tokens":7801,"prompt_tokens":743,"num_sources_used":0,"completion_tokens":77,"cost_in_usd_ticks":185065500,"prompt_tokens_details":{"text_tokens":743,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":6981,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":743,"tokens_out":77,"duration_ms":87115,"temperature":1.0,"reasoning_tokens":6981,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-07T16:31:03.654143+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Human raters assigning low quality scores to text samples generated by the Mistral 7B QLoRA model, or a new independent collection of Tajik documents producing substantially higher perplexity under the same fine-tuned model.","supporting_citations":[],"review_version":1}