Pith. sign in

REVIEW 3 major objections 5 minor 26 references

For Japanese-English LLM fine-tuning, choosing training data by semantic quality scores beats lexical and geometric heuristics — even when the chosen subsets differ by only a few percent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:52 UTC pith:MMMN62J6

load-bearing objection A well-controlled experimental sweep whose central consistency claim fails on its own tables, with duplicate numbers that make the reported results untrustworthy. the 3 major comments →

arxiv 2512.11388 v1 pith:MMMN62J6 submitted 2025-12-12 cs.CL

Improving Translation Quality by Selecting Better Data for LLM Fine-Tuning: A Comparative Analysis

Felipe Ribeiro Fujita de Mello , Hideyuki Takada This is my paper
classification cs.CL
keywords machine translationdata selectionfine-tuningLLMquality estimationCOMET-KiwiQURATEJapanese-English translation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the way you pick a small fixed-size training set for fine-tuning a translation LLM matters, and which picking rule is best. Using Japanese–English corpora and 7B-scale models, it compares five selectors: random sampling, TF-IDF, a geometric diversity score, QURATE, and COMET-Kiwi. Its central claim is that semantic selectors — particularly COMET-Kiwi, a reference-free quality estimator — consistently improve translation quality over lexical or geometry-based heuristics under identical training budgets. The paper further reports that even when the selected subsets share most of their content (unique overlap below 3%), the choice of selector can shift downstream COMET and BLEU scores substantially. This matters because data selection is a cheap lever in low-resource translation: better picking per example, not more data, is the proposed route to better models.

Core claim

The paper's core discovery is that semantic-aware data selection for LLM fine-tuning improves Japanese↔English translation more than lexical or geometry-based selection, and that this holds across several 7B-scale architectures. In the paper's controlled setup, each selection method produced a same-size subset (1,000 or 10,000 pairs) from the KFTT corpus, and identical models were fine-tuned with the same hyperparameters. The reported result is that selectors which score translation pairs for semantic adequacy — QURATE and especially COMET-Kiwi — outperform TF-IDF and FD-Score on the majority of evaluations, with the largest gains in the EN→JA direction. The paper also finds that the unique

What carries the argument

The key machinery is the data-selection scoring function s(x_i) applied to each candidate sentence pair, combined with a fixed pipeline that ranks candidates, takes the top-k subset, and fine-tunes a 7B-scale model with low-rank adapters under identical hyperparameters. The central objects are the five selectors: random sampling, mean TF-IDF (lexical relevance), FD-Score (geometric distance from the TF-IDF centroid), QURATE (a general text-quality scorer), and COMET-Kiwi (a reference-free neural quality estimator trained to approximate human/COMET judgments). The claim is carried by the contrast: lexical and geometric scores capture surface properties, while semantic selectors capture adequa

Load-bearing premise

The evaluation metric COMET is treated as the yardstick for translation quality even though the promoted selector, COMET-Kiwi, is trained to approximate that same metric; if COMET bias is the real driver, the reported gains may not reflect human-judged translation improvement.

What would settle it

A human-rated evaluation of the same models on WMT24 test sets: if COMET-Kiwi-selected models do not receive higher human adequacy/fluency scores than TF-IDF-selected models, the paper's central claim collapses. Alternatively, training the same pipelines but evaluating with a metric not correlated with COMET, such as chrF or human judgments, would reveal whether the COMET gains are metric-specific.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If COMET-Kiwi-style selection is adopted, the same fine-tuning budget yields higher translation quality than random or TF-IDF selection, especially in low-resource JA↔EN settings.
  • The sensitivity result implies that fine-tuning datasets do not need to be large; a carefully chosen subset can outperform a larger random one.
  • Semantic selectors reduce variance and produce smoother training loss curves, which suggests more stable convergence during fine-tuning.
  • The method generalizes across several 7B-scale model families, indicating the gain comes from data distribution rather than a specific architecture.
  • Since COMET-Kiwi is reference-free, the selector can be applied to unlabeled or crawled parallel corpora, not just hand-curated ones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed effect sizes are consistent with a metric-alignment story: because COMET-Kiwi is trained to approximate COMET, selecting with it may produce models that score well on COMET without necessarily improving human-perceived translation quality; a human evaluation would separate these.
  • The <3% overlap result suggests a testable extension: deliberately adding a small number of COMET-Kiwi-picked 'unique' sentences to a random or TF-IDF baseline subset should reproduce a large share of the gain, isolating diversity from selection quality.
  • If the effect transfers to other language pairs and domains, data selection via QE models could become a standard preprocessing step in low-resource MT pipelines, replacing or complementing perplexity-based filtering.
  • The paper's comparison treats the selectors as full pipelines; a sharper inference would be to ablate overlap by seeding each method with the same random base and then adding method-specific unique items, to measure the marginal value of each method's exclusive picks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares five data-selection strategies (Random, TF-IDF, FD-Score, QURATE, COMET-Kiwi) for fine-tuning four 7B-scale LLMs (LLaMA-7B, Gemma-7B, Qwen-7B, Mistral-7B) on Japanese-to-English and English-to-Japanese translation. All methods select 1k or 10k examples from KFTT, fine-tune with LoRA under fixed hyperparameters, and are evaluated on WMT24 with BLEU and COMET. The paper claims that semantic selectors, especially COMET-Kiwi, consistently outperform lexical and geometry-based heuristics, and that small differences in the selected subsets can produce substantial performance differences. It also includes a 1k ablation, uniqueness analyses, qualitative examples, score-distribution plots, and training-loss curves. The central claim is that data quality, rather than quantity, drives fine-tuning performance in low-resource settings.

Significance. If the empirical comparison were reliable, the paper would provide useful guidance for low-resource MT data selection: it uses a controlled setup (fixed subset size, identical LoRA configuration, multiple model families, both translation directions) and evaluates on an out-of-domain WMT24 test set. The 1k ablation and the qualitative tables also give concrete illustrations of how selection methods affect outputs. However, the headline conclusion is not supported by the reported numbers. Table III directly contradicts the claim that semantic selectors consistently win: QURATE beats COMET-Kiwi on COMET for LLaMA-7B and Gemma-7B, and FD-Score beats COMET-Kiwi for Gemma-7B and Mistral-7B. Moreover, Tables III and IV contain exact duplicate rows across different model/method conditions, and the experiments use one seed with no repeated runs or released code/data. These issues are load-bearing: they undermine the empirical basis for the paper's main contribution.

major comments (3)
  1. [§V-B, Table III] The conclusion "Across all models, semantic-based selection methods (QURATE and COMET-Kiwi) outperform lexical or geometric heuristics" is contradicted by the paper's own results. On JA→EN: for LLaMA-7B, QURATE scores COMET 0.7873 vs COMET-Kiwi 0.7105; for Gemma-7B, QURATE scores 0.8117 vs COMET-Kiwi 0.7761 and FD-Score scores 0.8077 vs COMET-Kiwi 0.7761; for Mistral-7B, FD-Score scores 0.8116 vs COMET-Kiwi 0.7761. Thus the claim of "consistent" semantic superiority fails on the paper's primary metric before any external evaluation is considered. The abstract and §VI repeat this claim and should be revised or the empirical basis needs to be re-established.
  2. [Tables III and IV] The reported tables contain exact numeric duplicates across different model/method cells, which cannot be explained by the controlled experimental design. Examples: Table III Gemma Random = Mistral Random (17.37 / 0.8095); Gemma FD-Score = Mistral TF-IDF (20.05 / 0.8077); Gemma COMET-Kiwi = Mistral COMET-Kiwi (20.25 / 0.7761). Table IV: Qwen TF-IDF = Mistral TF-IDF = Mistral FD-Score (13.02 / 0.7838). Since §IV-B reports a single seed (42) and no error bars, repeated runs, per-run logs, or released code/data, these coincidences cannot be attributed to expected variance. They undermine the integrity of the comparison and make the results impossible to verify in their current form. The authors should provide the actual per-condition results, repeat experiments with multiple seeds, and report variance.
  3. [§III-C, §IV-D] The evaluation metric COMET is not independent of the promoted selector COMET-Kiwi. Section III-C states that COMET-Kiwi learns to approximate the COMET score (and is trained by distilling COMET signals). Since COMET is the primary evaluation metric in Tables II–IV and Figure 3, the reported COMET gains from COMET-Kiwi-selected data may partly reflect alignment with COMET's learned preferences rather than improved human-facing translation quality. The BLEU columns do not confirm the pattern (e.g., Table III LLaMA-7B: COMET-Kiwi BLEU 25.91 vs QURATE 13.95, but COMET 0.7105 vs 0.7873). The limitations section (§VII) admits only automatic metrics were used, but this concern is not merely a future-work item: it affects the interpretation of the current headline result. The authors should either use a held-out, independently trained metric or include human evaluation before claiming that COME
minor comments (5)
  1. [§II-A, References] Citation mismatch: the text attributes TF-IDF text-classification results to "Das et al. [5]", but reference [5] is Moore and Lewis (2010); reference [6] is Das et al. (2023). The in-text citation should point to [6].
  2. [Throughout] Inconsistent naming: the abstract and text use "COMET-KIWI", "COMET-Kiwi", "QURATE", "QuRate", "FD-Score", and "FD-SCORE" interchangeably. Please standardize selector names.
  3. [Table III] The statement in §V-B that "COMET–Kiwi's reference-free scoring allows it to generalize effectively" is not supported by the table for the JA→EN direction; please make the discussion consistent with the actual rows.
  4. [§V-D] The uniqueness analysis shows overlap counts but does not connect these counts to the model performance differences. If unique samples are claimed to drive improvements, provide a quantitative link (e.g., test-set perplexity or per-sample contribution) rather than only qualitative examples.
  5. [Figure 4] The caption says "training loss curves for all evaluated models" but the figure is difficult to read because the subplots are small and the labels are cramped. Please enlarge the figure and label each subplot clearly.

Circularity Check

1 steps flagged

COMET-Kiwi selection is a distilled approximation of the COMET evaluation metric, so the headline COMET gains are partially self-referential; the QURATE comparison and BLEU columns provide only partial independent grounding.

specific steps
  1. self definitional [Section III-C (COMET vs. COMET-Kiwi), Section IV-D (Evaluation Metrics), Section V-B (conclusion)]
    "COMET-Kiwi ... learns to approximate the COMET score or direct human judgments without needing a gold reference. ... In our work, we use COMET-Kiwi as a scoring function s(xi) in the data selection objective. ... COMET [7] is a neural quality estimation metric trained on human judgments, better reflecting adequacy and fluency. Higher scores on both metrics indicate better translation quality."

    The top-k training pairs are chosen with a scorer that the paper itself describes as approximating the COMET score, and then the fine-tuned models are judged with COMET. Consequently, the reported COMET advantage for COMET-Kiwi-selected data is partly built into the selection input: the model is trained on pairs preferred by a COMET surrogate and then measured with COMET (or its close family). That leg of the abstract's 'consistently outperform' claim is therefore not independent evidence of human-facing quality. The limitations section admits only automatic metrics were used, and the BLEU columns do not reproduce the same pattern (e.g., Table III JA->EN: QURATE COMET 0.7873 > COMET-Kiwi 0.7105 on LLaMA; FD-Score 0.8077 > COMET-Kiwi 0.7761 on Gemma), which is the symptom of metric-alignmen

full rationale

The main circularity is the COMET-Kiwi/COMET pair: the selector is a reference-free distillation of the evaluation metric, so its COMET gains are partially tautological. This is not a full 10 because the comparison also includes QURATE, which was not trained to approximate COMET, and the paper reports BLEU and WMT24 external references as partial independent signals. The self-citation to the authors' prior WMT system [4] is descriptive and not load-bearing. Separately, the empirical tables contain exact duplicate rows and contradictions (e.g., Table III has Gemma Random = Mistral Random and FD-Score > COMET-Kiwi in several cells) that undermine the evidence base, but those are correctness/reporting issues rather than circularity per se. Because the central claim's COMET-Kiwi leg reduces partly to approximating its own evaluation metric, the circularity score is 6.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper's central inference depends on the validity of learned quality scores as proxies for human judgment, on a single training configuration, and on a single seed; these are assumed rather than established. No new entities are introduced.

free parameters (3)
  • Subset size k = 1,000 / 10,000
    Top-k is fixed by hand; the 1k ablation and 10k results differ, so conclusions may depend on this choice.
  • LoRA rank/alpha/target modules = not reported
    LoRA is named (Section IV-B) but rank, alpha, and target layers are omitted, making the configuration non-reproducible and a hand-set choice.
  • FD-Score distance metric = not specified
    FD-Score is described as geometric distance from the TF-IDF centroid (Sections II and IV-C) but no distance formula is given; the implementation is a free choice.
axioms (3)
  • domain assumption COMET/COMET-Kiwi scores are valid proxies for human translation quality
    Used both as the selection signal (Section III-C) and the primary evaluation metric (Section IV-D); if these learned quality models are biased, the conclusion about semantic quality does not follow.
  • ad hoc to paper A single seed (42) and one epoch per subset are sufficient to characterize selector performance
    Section IV-B fixes seed=42 and epochs=1; no repeated runs are reported, so observed differences could be within-run noise.
  • domain assumption KFTT-derived selected subsets and WMT24 are representative of low-resource JA↔EN translation
    Section IV-A and Section VII state the scope is limited to JA-EN; extending to other language pairs, domains, or data scales is an untested assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 12545 in / 16392 out tokens · 134177 ms · 2026-08-03T16:52:53.885278+00:00 · methodology

0 comments
read the original abstract

We investigated the impact of data selection on machine translation fine-tuning for open LLMs. Using Japanese-English corpora, we compare five selectors: TF-IDF, COMET Kiwi, QuRate, FD-Score, and random selection, under controlled training conditions. We observed that semantic selectors consistently outperform lexical and geometry-based heuristics, and that even when the selected data differ by less than 3%, the impact on model performance is substantial, underscoring the sensitivity of fine-tuning to data quality.

Figures

Figures reproduced from arXiv: 2512.11388 by Felipe Ribeiro Fujita de Mello, Hideyuki Takada.

Figure 1
Figure 1. Figure 1: Proposed method. We calculate the scores of different data selection methods and re-rank the samples based on the score. Then, we fine-tune a [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall diagram of the proposed method, consisting of (1) data selection workflow, (2) fine-tuning pipeline, and (3) evaluation architecture. Raw [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: COMET Score Distribution Per Selection Method on LLaMA-7B [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training loss curves for all evaluated models (LLaMA-7B, Gemma-7B, Mistral-7B, and Qwen-7B) fine-tuned on JA [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 7 linked inside Pith

  1. [1]

    Dawei Zhu, Pinzhen Chen, Miaoran Zhang, Barry Haddow, Xiaoyu Shen, and Dietrich Klakow. 2024. Fine-Tuning Large Language Models to Translate: Will a Touch of Noisy Data in Misaligned Languages Suffice?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 388–409, Miami, Florida, USA. Association for Computation...

    2024

  2. [2]

    Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain Adaptation via Pseudo In-Domain Data Selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355–362, Edinburgh, Scotland, UK.. Association for Computa- tional Linguistics

    2011

  3. [3]

    Hainan Xu and Philipp Koehn. 2017. Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2945–2950, Copenhagen, Denmark. Association for Computational Linguistics

    2017

  4. [4]

    Fujita and H

    F. Fujita and H. Takada, Improving Low-Resource Japanese Translation with Fine-Tuning and Backtranslation for the WMT 25 General Trans- lation Task, inProc. 10th Conf. on Machine Translation (WMT 2025), Suzhou, China, 2025

    2025

  5. [5]

    Moore and William Lewis

    Robert C. Moore and William Lewis. 2010. Intelligent Selection of Language Model Training Data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220–224, Uppsala, Sweden. Association for Computational Linguistics

    2010

  6. [6]

    Das, M., Kamalanathan, S., and Alphonse, P. (2023). A Comparative Study on TF-IDF Feature Weighting Method and Its Analysis Using Unstructured Dataset. International Conference on Computational Lin- guistics and Intelligent Systems

    2023

  7. [7]

    Binh-Nguyen Nguyen and Yang He. 2025. Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding. In Proceedings of the 31st International Conference on Computational Linguistics, pages 726–739, Abu Dhabi, UAE. Association for Compu- tational Linguistics

    2025

  8. [8]

    Ricardo Rei, Jos ´e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr ´e F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). As...

    2022

  9. [9]

    Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos ´e G

    Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos ´e G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andr´e F. T. Martins. 2022. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation (W...

    2022

  10. [10]

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang

  11. [11]

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. QuRating: Selecting High-Quality Data for Training Language Models. In Proceedings of the International Conference on Machine Learning (ICML)

    2024

  12. [12]

    Teng Wang, Zhenqi He, Wing-Yin Yu, Xiaojin Fu, and Xiongwei Han

  13. [13]

    Shum et al

    K. Shum et al. 2025. Predictive Data Selection: The Data That Predicts Is the Data That Teaches. InProc. of ACL. Available: https://arxiv.org/abs/2503.00808

    Pith/arXiv arXiv 2025

  14. [14]

    Snover, B

    M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proc. of AMTA. Available: https://aclanthology.org/2006.amta-papers.25/

    2006

  15. [15]

    Albalak, Y

    A. Albalak, Y . Elazar, S. M. Xie, S. Longpre, N. Lambert, X. Wang, N. Muennighoff, B. Hou, L. Pan, H. Jeong, C. Raffel, S. Chang, T. Hashimoto, and W. Y . Wang. 2024. A Survey on Data Selection for Language Models. InarXiv preprint. Available: https://arxiv.org/abs/ 2402.16827

    Pith/arXiv arXiv 2024

  16. [16]

    Toneva, A

    M. Toneva, A. Sordoni, R. Tachet des Combes, A. Trischler, Y . Bengio, and G. J. Gordon. 2018. An Empirical Study of Example Forgetting during Deep Neural Network Learning. InCoRR, vol. abs/1812.05159. Available: http://arxiv.org/abs/1812.05159

    Pith/arXiv arXiv 2018

  17. [17]

    Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ri- cardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag

  18. [18]

    G. Neubig. 2011. The Kyoto Free Translation Task. Available: http: //www.phontron.com/kftt

    2011

  19. [19]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, and G. Lampleet al.. 2023. LLaMA: Open and Efficient Foundation Language Models. InarXiv preprint. Available: https://arxiv.org/abs/2302.13971

    Pith/arXiv arXiv 2023

  20. [20]

    In Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 12257–12284, Vienna, Austria

    WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects. In Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 12257–12284, Vienna, Austria. Association for Computational Linguistics

    2025

  21. [21]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, and M. A. Lachaux et al.. 2023. Mistral 7B. InarXiv preprint. Available: https://arxiv.org/ abs/2310.06825

    Pith/arXiv arXiv 2023

  22. [22]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, and F. Huanget al.. 2025. Qwen3 Technical Report. InarXiv preprint. Available: https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  23. [23]

    Mesnard, C

    Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Love, P. Tafti, L. Hussenot, and A. Chowdheryet al.. 2024. Gemma: Open Models Based on Gemini Research and Technology. InarXiv preprint. Available: https://arxiv.org/ abs/2403.08295

    Pith/arXiv arXiv 2024

  24. [26]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InProc. of ICLR. Available: https://openreview.net/forum?id= nZeVKeeFYf9

    2022

  25. [2023]

    In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23)

    Data selection for language models via importance resampling. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23). Curran Associates Inc., Red Hook, NY , USA, Article 1482, 34201–34227

  26. [2025]

    Large Language Models are Good Multilingual Learners: When LLMs Meet Cross-Lingual Prompts. InProc. of COLING, pages 4442– 4456