Pith. sign in

REVIEW 4 major objections 5 minor 39 references

PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that a small, error-aware language model can compress the inputs of LLM-based translation metrics, cutting token usage 2.37x while preserving—and on segment level improving—the metric's correlation with human judgment.

desk verdict Useful efficiency result for GEMBA-MQM, but the 'no loss' claim is a single-run point estimate; worth reviewing with a demand for error bars and softened claims. read the letter →

arxiv 2412.16120 v1 pith:SB7XE7OB submitted 2024-12-20 cs.CL

Daniil Larionov , Steffen Eger This is my paper · ORCID
classification cs.CL
keywords promptcompressionmachinetranslationevaluationLLMmetricsGEMBA-MQMpreferenceoptimizationORPOtokenusagereductionerrorspanpreservation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the cost of LLM-based machine translation evaluation can be cut by more than half without sacrificing quality, by using a small fine-tuned language model to compress the source text and translation before they go into the evaluator's prompt. The compressed inputs keep the spans that carry translation errors, and the large evaluator (GPT-4o) still scores them accurately. On a 16k-example WMT22 test set, the method reduces token usage from 19M to 8.07M, a 2.37x reduction, while holding pairwise system accuracy at 0.7736 and improving segment-level Kendall tau over the uncompressed baseline on all three language pairs tested. This would make state-of-the-art metrics like GEMBA-MQM practical for large-scale evaluation and reranking.

What carries the argument

The load-bearing mechanism is the two-stage fine-tuned compressor built on LLaMA 3.2 (1B and 3B). In stage one, supervised fine-tuning on MQM-annotated WMT data teaches the model to output three things: a compression rate $r$ from {0.3,...,1.0}, the substrings in source and translation that contain errors, and compressed versions of both texts. The training targets are generated by random token removal that preserves all error span tokens, so the model learns to anchor compression around the evidence the MQM metric needs. In stage two, ORPO preference optimization re-ranks the model's compressions: for each example the chosen compression is the rate whose GPT-4o GEMBA-MQM score is closest to the uncompressed score and the rejected one is furthest, so the model is pushed toward compressions that do not change the metric's verdict. A fixed simplified GEMBA-MQM instruction (short system prompt, JSON output) replaces the verbose original and contributes most of the raw token savings.

What would settle it

Repeat the preference-data generation on a sample of examples, running GPT-4o several times per compression rate: if within-example score variance across runs matches or exceeds the score spread across rates, the chosen/rejected labels are effectively random and the reported quality preservation should not replicate. A second check is to apply the trained compressor to an unseen language pair and require that system-level pairwise accuracy not fall below the uncompressed baseline.

Watch

Extended reading notes

Core claim

The central discovery is that a small model trained to compress translation inputs in an error-aware way preserves—and in some cases improves—the quality of a large LLM-based metric, while generic token-level compression destroys it. PromptOptMe is a LLaMA 3.2 3B model fine-tuned in two stages: first supervised learning on WMT MQM data where random token removal keeps error spans intact, then odds-ratio preference optimization (ORPO) on pairs of compressions labeled by how close their GPT-4o GEMBA-MQM score is to the score for uncompressed text. The resulting compressor, combined with a simplified GEMBA-MQM instruction, achieves the claimed 2.37x token reduction: 8.07M tokens vs. 19M for the uncompressed reference. Segment-level correlation with human judgments improves on all three language pairs tested with GPT-4o (En-Ru $\tau$ 0.4455, En-De 0.4065, Zh-En 0.3738), and system-level pairwise accuracy equals the baseline at 0.7736. The paper also shows the compressor generalizes to GPT-4o mini and LLaMA 3.2-90B backbones, and that the LLMLingua-2 general-purpose compressor, by contrast, drops segment-level correlation to near zero.

Load-bearing premise

The method assumes that a compression is good exactly when its GPT-4o GEMBA-MQM score matches the score for the uncompressed text, and that those scores are stable enough to form reliable training preferences.

Editorial extensions

If this is right

  • At 2.37x fewer input tokens, LLM-based MT evaluation becomes cheap enough for large-scale use cases like system reranking and web-scale dataset filtering, not just small test sets.
  • Because segment-level correlations improve on the tested language pairs, error-aware compression may focus the evaluator on informative spans rather than distracting context.
  • The same prompt compressor works with different backbone evaluators (GPT-4o, GPT-4o mini, LLaMA 3.2-90B), so the training transfer is not tied to one API.
  • Generic token-importance compressors are not a viable shortcut for this task; task-specific training that preserves error spans appears necessary to retain evaluation quality.
  • Simplified instructions alone nearly halve token usage, meaning much of GEMBA-MQM's prompt budget is redundant for evaluation quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 2.37x figure counts only the large evaluator's input tokens; adding the compressor's own inference cost would narrow the true savings, especially for short segments, and a full cost comparison is not in the paper.
  • If the preference labels are noisy because GPT-4o's scores vary across repeated runs, the chosen/rejected pairs may be partly arbitrary; a repeat-sampling study would tell us how much of the quality preservation is real signal.
  • The error-span-preservation principle could transfer to other fine-grained evaluation tasks with annotated spans, such as summarization with factual error annotations, though the paper only demonstrates MT.
  • The success of a fixed simplified instruction suggests prompt designers should quantify how much of a long evaluation prompt is actually load-bearing before optimizing input compression.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PromptOptMe, a two-stage fine-tuned small language model (LLaMA-3.2-1B/3B) that compresses the source and machine-translation inputs to the GEMBA-MQM evaluation prompt, while also outputting a compression rate and error-relevant spans. Stage one is supervised fine-tuning on WMT20-22 MQM annotations with random token removal that preserves gold error spans; stage two is ORPO preference optimization where chosen/rejected compressions are defined by closeness of the GPT-4o-produced MQM score to the score on the uncompressed input. Combined with a simplified prompt template, the method reports a 2.37x reduction in token usage on a 16k-example WMT22 test set, with pairwise accuracy 0.7736 versus 0.7789 for the uncompressed GPT-4o baseline and higher segment-level Kendall tau on En-Ru, En-De, and Zh-En. The paper also reports results on GPT-4o mini and LLaMA-3.2-90B, and contrasts with LLMLingua-2, which degrades quality dramatically.

Significance. If the headline claim is statistically solid, the paper addresses a real cost bottleneck: it cuts GPT-4o's input token usage for GEMBA-MQM by more than half without sacrificing measured correlation with human judgments. The design is task-aware (error-span preservation), uses an external human-judgment benchmark for evaluation, and reports practical details (hyperparameters, GPU budget, planned code release). The work is incremental rather than paradigm-shifting, but it is a useful engineering contribution to LLM-based evaluation. The main caveat is that the 'no loss' assertion rests on single-run point estimates without uncertainty quantification, so the significance currently depends on an act of trust rather than on demonstrated statistical parity.

major comments (4)
  1. [Section 5, Table 1] The central claim 'without any loss in evaluation quality' is not yet supported because all results are single-run point estimates with no confidence intervals, standard errors, or significance tests. In Table 1, pairwise accuracy drops from 0.7789 (GPT-4o ref) to 0.7736 (GPT-4o lite + PromptOptMe-3B), while the segment-level Kendall tau values improve; without uncertainty estimates, one cannot know whether the pairwise drop and the tau gains are within noise. The paper also does not state the number of MT systems used for the pairwise accuracy calculation, which determines the precision of that estimate. I recommend reporting bootstrap confidence intervals over segments and over systems, paired significance tests (e.g., paired bootstrap on Kendall tau, McNemar on pairwise accuracy), or repeated API sampling at least on a subsample, and then basing the 'no loss' wording on the resulting bounds.
  2. [Section 3.2] The preference labels for ORPO are derived from a single GPT-4o score per compression rate, with r_chosen = argmin |s_r - s_1.0| and r_rejected = argmax. If GPT-4o's MQM scores are noisy or insensitive to the compression on some examples, these labels can be effectively random, and the trained model's behavior on those examples becomes unpredictable. Moreover, aligning compressed-prompt scores with GPT-4o's own uncompressed score does not by itself guarantee preservation of human-aligned quality. The paper should quantify the variance of GPT-4o scores, e.g., by repeated evaluations on a sample, measure the stability of the chosen/rejected labels, and ideally validate a subset of preference labels against human judgments or a second LLM. Without this, the training signal's reliability is unquantified and the downstream claims inherit that uncertainty.
  3. [Abstract, Section 5] The abstract's unqualified statement '2.37x reduction in token usage without any loss in evaluation quality' is contradicted by several rows of Table 1. For GPT-4o mini lite + PromptOptMe-3B, En-Ru Kendall tau drops from 0.3723 to 0.3177, and for LLaMA-3.2-90B lite + PromptOptMe-3B, Zh-En tau drops from 0.3576 to 0.3123. The text itself concedes in Section 5 that 'there are cases when metric quality reduces, sometimes considerably.' The claim should be restricted to the specific configuration(s) for which it holds (e.g., GPT-4o with PromptOptMe-3B on the reported language pairs) or be accompanied by significance tests showing that the decreases are within sampling noise. As written, the abstract overstates the findings.
  4. [Section 3.3, Section 5] The paper states in Section 3.3 that the simplified instruction 'did not adversely affect the metric quality' and that evaluation quality 'remained comparable' to the original GEMBA-MQM prompt. Table 1 does not support this: GPT-4o lite alone has visibly lower segment-level Kendall tau on all three language pairs compared to GPT-4o ref (e.g., Zh-En 0.2890 vs 0.3692). The quality recovery in the final system comes only after adding PromptOptMe compression. This discrepancy should be acknowledged and the claim reworded, because the rationale for freezing the simplified prompt depends on this assertion.
minor comments (5)
  1. [Section 5, Table 1] The text reports a reduction rate of 2.04x for GPT-4o lite while Table 1 lists 1.84x; the same paragraph says token usage drops to 8.3M while Table 1 says 8.07M. Please harmonize these numbers and double-check all arithmetic in the results section.
  2. [Introduction] The Introduction states a 2.32x reduction while the Abstract and Section 5 state 2.37x; please make the headline number consistent throughout.
  3. [Appendix A, Figure 4 caption] The caption contains a typo: 'Simplifield Prompt' should be 'Simplified Prompt'.
  4. [Table 1 footnote] The footnote describes 'GPT-4o ref' as 'the original GPT-4' but the experiments use GPT-4o; please correct the model name for consistency.
  5. [Throughout] The method name is written inconsistently: 'PromptOptMe' in the title and abstract, 'PROMPT OPTME' in the body and tables, and 'PromptOptMe' in the figure. Please standardize to a single spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline claim is validated against external human judgments, not against the GPT-4o score-closeness training target.

full rationale

The only candidate for circularity is the preference objective in Section 3.2, where a 'chosen' compression is defined as one minimizing |s_r - s_1.0|, i.e., GPT-4o GEMBA-MQM score distance to the uncompressed input, and the model is then trained with ORPO on that signal. If the paper's 'no loss in evaluation quality' claim were measured by GPT-4o score preservation, this would indeed be partly by construction. However, the evaluation target in Sections 4-5 and Table 1 is pairwise accuracy and segment-level Kendall tau computed against WMT22 human judgments, which are external to the training labels; the GPT-4o score is only a surrogate training signal, not the reported outcome. The held-out test set, the LLMLingua-2 baselines, and the cross-backbone evaluations (GPT-4o mini, LLaMA 3.2-90B) also demonstrate that the results are not forced by the training objective. Self-citations (PrExMe, xCOMET-lite, EffEval, etc.) are contextual and carry no load-bearing argument. The absence of confidence intervals or significance tests is a statistical-evidence concern, not a circularity concern; under the required evidentiary standard, no specific step reduces to its own input.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The efficiency result is anchored in a small number of hand-chosen hyperparameters (compression rate set, ORPO lambda, LoRA and training settings). The quality result rests on three domain assumptions: MQM error spans are a sufficient basis for compression, GPT-4o score closeness is a valid preference signal, and WMT22 human scores are the correct yardstick. No new entities are introduced.

free parameters (4)
  • Compression rate set Rcomp = {0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}
    Chosen candidate rates used to generate SFT and preference examples; the 2.37x token reduction and the preference selection depend on this set and on the tie-breaking rule toward lower rates.
  • ORPO lambda = 0.1
    Preference optimization strength taken from Hong et al. (2024); controls the trade-off between preservation of chosen compressions and rejection of bad ones.
  • LoRA adapter hyperparameters = rank=32, alpha=16, dropout=0.5
    Chosen for fine-tuning LLaMA 3.2 1B/3B; affects how well the compression model learns the task.
  • Training schedule = LR 2e-5 (SFT) and 1e-5 (ORPO), 1 and 3 epochs, batch 64, warmup 6%
    Hand-selected training hyperparameters; the final quality of the compression model depends on them.
assumptions (5)
  • domain assumption Preserving MQM error spans is sufficient to preserve the information needed for high-quality MT evaluation
    Motivates the SFT target in Section 3.1 and the preference signal in Section 3.2; if error spans do not capture all quality-relevant content, compressed prompts may lose information.
  • domain assumption GPT-4o with the uncompressed GEMBA-MQM prompt provides a reliable teacher signal for preferring one compression over another
    Section 3.2 defines chosen and rejected compressions by GPT-4o score differences; score noise or bias in this teacher directly corrupts the ORPO training data.
  • domain assumption WMT MQM human annotations used for SFT are accurate error spans
    Section 3.1 relies on these annotations to mark the spans that must be preserved; annotation errors propagate into training data.
  • domain assumption Human judgments from the WMT22 metrics task are the correct reference for evaluation quality
    Section 4 uses pairwise accuracy and Kendall tau against these judgments as the yardstick for 'no loss in evaluation quality'.
  • domain assumption The simplified prompt template preserves the MQM error categories well enough for scoring
    Section 3.3 fixes a shortened instruction set; the reported quality is achieved only with this template, not with the original verbose prompt.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics." pith.science (2026). https://pith.science/paper/SB7XE7OB

@misc{pith2026241216120,
  author       = {Pith},
  title        = {Pith review of: PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SB7XE7OB}},
  note         = {Machine review of arXiv:2412.16120}
}
abstract

Evaluating the quality of machine-generated natural language content is a challenging task in Natural Language Processing (NLP). Recently, large language models (LLMs) like GPT-4 have been employed for this purpose, but they are computationally expensive due to the extensive token usage required by complex evaluation prompts. In this paper, we propose a prompt optimization approach that uses a smaller, fine-tuned language model to compress input data for evaluation prompt, thus reducing token usage and computational cost when using larger LLMs for downstream evaluation. Our method involves a two-stage fine-tuning process: supervised fine-tuning followed by preference optimization to refine the model's outputs based on human preferences. We focus on Machine Translation (MT) evaluation and utilize the GEMBA-MQM metric as a starting point. Our results show a $2.37\times$ reduction in token usage without any loss in evaluation quality. This work makes state-of-the-art LLM-based metrics like GEMBA-MQM more cost-effective and efficient, enhancing their accessibility for broader use.

Figures

Figures reproduced from arXiv: 2412.16120 by the authors.

Figure 1
Figure 1. The two-stage model training approach used [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of Model-generated compression. Reference translation and Gold MQM annotations are provided for readers convenience and are not part of the model prompt. translation — as well as the outputs — compression rate, error spans and compressed texts. We train the model for one epoch using a batch size of 64 and a learning rate of 2 × 10−5 with cosine warmup over 6% of the training steps. We apply a weight decay of… view at source ↗
Figure 3
Figure 3. Original Prompt for GEMBA-MQM metric. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Simplifield Prompt for GEMBA-MQM metric. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Jonas Belouadi and Steffen Eger. 2023. https://doi.org/10.18653/v1/2023.eacl-main.27 US core: An effective approach to fully unsupervised evaluation metrics for machine translation . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 358--374, Dubrovnik, Croatia. Association for Computatio...

    work page doi:10.18653/v1/2023.eacl-main.27 2023

  3. [3]

    Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. https://aclanthology.org/E06-1032 Re-evaluating the role of B leu in machine translation research . In 11th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages 249--256, Trento, Italy. Association for Computational Linguistics

    2006

  4. [4]

    Yanran Chen and Steffen Eger. 2023. https://doi.org/10.1162/tacl_a_00576 MENLI : Robust evaluation metrics from natural language inference . Transactions of the Association for Computational Linguistics, 11:804--825

    doi:10.1162/tacl_a_00576 2023

  5. [5]

    Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.176 Black-box prompt optimization: Aligning large language models without model training . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    doi:10.18653/v1/2024.acl-long.176 2024

  6. [6]

    Daniel Deutsch, George Foster, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.798 Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914--12929, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.798 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  8. [8]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation . In Proceedings of the Eighth Confer...

    doi:10.18653/v1/2023.wmt-1.100 2023

Show all 39 references
  1. [9]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021 a . https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for C...

    2021 doi

  2. [10]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.2 Results of WMT 22 metrics shared task: Stop using BLEU -- neural metrics ...

    2022

  3. [11]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ond r ej Bojar. 2021 b . https://aclanthology.org/2021.wmt-1.73 Results of the WMT 21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and ne...

    2021

  4. [12]

    Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e FT Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint arXiv:2310.10482

    2023 arXiv

  5. [13]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691

    2024 arXiv

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

    2022

  7. [15]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.825 LLML ingua: Compressing prompts for accelerated inference of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural...

    2023 doi

  8. [16]

    Moussa Kamal Eddine, Guokan Shang, Antoine Tixier, and Michalis Vazirgiannis. 2022. https://doi.org/10.18653/v1/2022.acl-long.93 F rugal S core: Learning cheaper, lighter and faster evaluation metrics for automatic text generation . In Proceedings of the 60th Annual Meeting of...

    2022 doi

  9. [17]

    Marvin Kaster, Wei Zhao, and Steffen Eger. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.701 Global explainability of BERT -based evaluation metrics by disentangling along linguistic factors . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P...

    2021 doi

  10. [18]

    Tom Kocmi and Christian Federmann. 2023 a . https://doi.org/10.18653/v1/2023.wmt-1.64 GEMBA - MQM : Detecting translation quality error spans with GPT -4 . In Proceedings of the Eighth Conference on Machine Translation, pages 768--775, Singapore. Association for Computational ...

    2023 doi

  11. [19]

    Tom Kocmi and Christian Federmann. 2023 b . https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, ...

    2023

  12. [20]

    Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. https://aclanthology.org/2021.wmt-1.57 To ship or not to ship: An extensive evaluation of automatic metrics for machine translation . In Proceedings of the ...

    2021

  13. [21]

    Weize Kong, Spurthi Hombaiah, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. https://doi.org/10.18653/v1/2024.acl-short.54 PR ewrite: Prompt rewriting with reinforcement learning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui...

    2024 doi

  14. [22]

    Daniil Larionov, Jens Gr \"u nwald, Christoph Leiter, and Steffen Eger. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.7 E ff E val: A comprehensive evaluation of efficiency for MT evaluation metrics . In Findings of the Association for Computational Linguistics: EMNLP ...

    2023 doi

  15. [23]

    Daniil Larionov, Mikhail Seleznyov, Vasiliy Viskov, Alexander Panchenko, and Steffen Eger. 2024. xcomet-lite: Bridging the gap between efficiency and quality in learned mt evaluation metrics. arXiv preprint arXiv:2406.14553

    2024 arXiv

  16. [24]

    Christoph Leiter and Steffen Eger. 2024. https://api.semanticscholar.org/CorpusID:270737974 Prexme! large scale prompt exploration of open source llms for machine translation and summarization evaluation . ArXiv, abs/2406.18528

    2024 arXiv

  17. [25]

    Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva, Wei Zhao, Yang Gao, and Steffen Eger. 2024. Towards explainable evaluation metrics for machine translation. Journal of Machine Learning Research, 25(75):1--49

    2024

  18. [26]

    Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. 2023. https://doi.org/10.18653/v1/2023.eval4nlp-1.10 The E val4 NLP 2023 shared task on prompting large language models as explainable metrics . In Proceedings of the 4th Workshop on Evaluati...

    2023 doi

  19. [27]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  20. [28]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  21. [29]

    Juri Opitz and Anette Frank. 2021. https://doi.org/10.18653/v1/2021.eacl-main.129 Towards a decomposable metric for explainable evaluation of text generation from AMR . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisti...

    2021 doi

  22. [30]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor R \"u hle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.57 LLML ingua-2: Data distillation for efficie...

    2024 doi

  23. [31]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  24. [32]

    Jan-Thorsten Peter, David Vilar, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, and Markus Freitag. 2023. https://doi.org/10.18653/v1/2023.wmt-1.50 There ' s no data like better data: Using QE metrics for MT data filtering . In Proceedings of the Eighth Conference on Machine...

    2023 doi

  25. [33]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022 a . https://aclanthology.org/2022.wmt-1.52 COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In ...

    2022

  26. [34]

    de Souza, Pedro G

    Ricardo Rei, Ana C Farinha, Jos \'e G.C. de Souza, Pedro G. Ramos, Andr \'e F.T. Martins, Luisa Coheur, and Alon Lavie. 2022 b . https://aclanthology.org/2022.eamt-1.9 Searching for COMETINHO : The little metric that could . In Proceedings of the 23rd Annual Conference of the ...

    2022

  27. [35]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  28. [36]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  29. [37]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. https://doi.org/10.18653/v1/D19-1053 M over S core: Text generation evaluating with contextualized embeddings and earth mover distance . In Proceedings of the 2019 Conference on Empirical ...

    2019 doi

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.