{"id":"9b42fac5-8be6-4ed7-9333-8dbb04c0c45f","arxiv_id":"2412.16120","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PromptOptMe compresses the inputs of the GEMBA-MQM MT evaluation prompt with a two-stage fine-tuned LLaMA 3.2 model, achieving a 2.37x token reduction without quality loss in the headline GPT-4o configuration.","lead":"PromptOptMe trains a small LLaMA model to compress the source texts and translations inside the GEMBA-MQM evaluation prompt, cutting token use by 2.37x while keeping machine translation evaluation quality roughly unchanged. A generalist reader should care because LLM-based evaluation metrics are costly at scale, and this is a practical way to make them cheaper without retraining the large evaluator.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no loss' claim rests on single-run point estimates; pairwise accuracy drops from 0.7789 to 0.7736 with no error bars or significance tests, so the central claim is not yet established.","rationale":"The paper presents a well-structured method with a clear two-stage training pipeline, a sensible use of MQM error spans for error-preserving compression, and a convincing demonstration that task-agnostic compression (LLMLingua-2) fails catastrophically. The improvement over LLMLingua-2 is large and likely robust. The generalization across backbone LLMs (GPT-4o, GPT-4o mini, LLaMA 3.2-90B) is encouraging and suggests the compression model is not overfit to GPT-4o. However, the central quantitative claim, 'no loss in evaluation quality', is a comparative statement that requires statistical support. The reported differences are small (pairwise accuracy difference of 0.0053, Kendall tau differences of up to 0.0115), and without repeated runs or confidence intervals we cannot distinguish them from sampling noise. The reader's weakest_assumption about the preference data is a valid upstream concern: if GPT-4o scores are noisy, the chosen/rejected labels in Section 3.2 may be arbitrary, and the training signal could be noise. Yet even if the preference signal were perfectly reliable, the final evaluation still needs variance estimates to support the headline. I therefore regard the absence of statistical rigor in the headline comparison as the most load-bearing concern, with the preference-noise issue as a contributing factor. The appropriate verdict remains CONDITIONAL pending this evidence.","tokens_in":15062,"tokens_out":7893,"duration_ms":64865,"concrete_test":"Run the full evaluation pipeline (PromptOptMe compression followed by GPT-4o scoring with the simplified prompt) at least 5 independent times on the same 16k test set, using a fixed temperature (e.g., 0, or a fixed nonzero value for all runs). Bootstrap 95% confidence intervals for pairwise accuracy and segment-level Kendall tau for both GPT-4o ref and GPT-4o lite + PromptOptMe-3B. Then perform a non-inferiority test on pairwise accuracy with a pre-specified margin of 0.005 (the observed difference). If the confidence interval for the difference excludes a loss larger than 0.005, the 'no loss' claim is supported; if not, the claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim in the abstract ('2.37x reduction in token usage without any loss in evaluation quality') is a comparison of two point estimates from Table 1. GPT-4o ref achieves pairwise accuracy 0.7789; GPT-4o lite + PromptOptMe-3B achieves 0.7736, a drop of 0.0053, while segment-level Kendall tau values increase (En-Ru 0.4365->0.4455, En-De 0.3950->0.4065, Zh-En 0.3692->0.3738). All four metrics are computed on a single run of a stochastic API model, with no confidence intervals, bootstrap estimates, significance tests, or repeated evaluations reported. The paper does not even state the number of MT systems used for pairwise accuracy, making the precision of that estimate unknowable. Without variance estimates, the observed pairwise accuracy decrease, which the paper calls 'same level', could be real noise, and the Kendall tau improvements could be within noise as well. The 'without any loss' claim is therefore an interpretation of noisy data, not a demonstrated fact. The preference optimization signal in Section 3.2 is related: chosen/rejected labels are based on a single GPT-4o score per compression rate, so if score variance is high the training labels are essentially random, making the final model's quality unpredictable. But the decisive gap for the central claim is the missing statistical support for the headline comparison itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PromptOptMe, a two-stage fine-tuned small language model (LLaMA-3.2-1B/3B) that compresses the source and machine-translation inputs to the GEMBA-MQM evaluation prompt, while also outputting a compression rate and error-relevant spans. Stage one is supervised fine-tuning on WMT20-22 MQM annotations with random token removal that preserves gold error spans; stage two is ORPO preference optimization where chosen/rejected compressions are defined by closeness of the GPT-4o-produced MQM score to the score on the uncompressed input. Combined with a simplified prompt template, the method reports a 2.37x reduction in token usage on a 16k-example WMT22 test set, with pairwise accuracy 0.7736 versus 0.7789 for the uncompressed GPT-4o baseline and higher segment-level Kendall tau on En-Ru, En-De, and Zh-En. The paper also reports results on GPT-4o mini and LLaMA-3.2-90B, and contrasts with LLMLingua-2, which degrades quality dramatically.","tokens_in":15386,"tokens_out":5440,"duration_ms":44220,"significance":"If the headline claim is statistically solid, the paper addresses a real cost bottleneck: it cuts GPT-4o's input token usage for GEMBA-MQM by more than half without sacrificing measured correlation with human judgments. The design is task-aware (error-span preservation), uses an external human-judgment benchmark for evaluation, and reports practical details (hyperparameters, GPU budget, planned code release). The work is incremental rather than paradigm-shifting, but it is a useful engineering contribution to LLM-based evaluation. The main caveat is that the 'no loss' assertion rests on single-run point estimates without uncertainty quantification, so the significance currently depends on an act of trust rather than on demonstrated statistical parity.","major_comments":[{"comment":"The central claim 'without any loss in evaluation quality' is not yet supported because all results are single-run point estimates with no confidence intervals, standard errors, or significance tests. In Table 1, pairwise accuracy drops from 0.7789 (GPT-4o ref) to 0.7736 (GPT-4o lite + PromptOptMe-3B), while the segment-level Kendall tau values improve; without uncertainty estimates, one cannot know whether the pairwise drop and the tau gains are within noise. The paper also does not state the number of MT systems used for the pairwise accuracy calculation, which determines the precision of that estimate. I recommend reporting bootstrap confidence intervals over segments and over systems, paired significance tests (e.g., paired bootstrap on Kendall tau, McNemar on pairwise accuracy), or repeated API sampling at least on a subsample, and then basing the 'no loss' wording on the resulting bounds.","section":"Section 5, Table 1"},{"comment":"The preference labels for ORPO are derived from a single GPT-4o score per compression rate, with r_chosen = argmin |s_r - s_1.0| and r_rejected = argmax. If GPT-4o's MQM scores are noisy or insensitive to the compression on some examples, these labels can be effectively random, and the trained model's behavior on those examples becomes unpredictable. Moreover, aligning compressed-prompt scores with GPT-4o's own uncompressed score does not by itself guarantee preservation of human-aligned quality. The paper should quantify the variance of GPT-4o scores, e.g., by repeated evaluations on a sample, measure the stability of the chosen/rejected labels, and ideally validate a subset of preference labels against human judgments or a second LLM. Without this, the training signal's reliability is unquantified and the downstream claims inherit that uncertainty.","section":"Section 3.2"},{"comment":"The abstract's unqualified statement '2.37x reduction in token usage without any loss in evaluation quality' is contradicted by several rows of Table 1. For GPT-4o mini lite + PromptOptMe-3B, En-Ru Kendall tau drops from 0.3723 to 0.3177, and for LLaMA-3.2-90B lite + PromptOptMe-3B, Zh-En tau drops from 0.3576 to 0.3123. The text itself concedes in Section 5 that 'there are cases when metric quality reduces, sometimes considerably.' The claim should be restricted to the specific configuration(s) for which it holds (e.g., GPT-4o with PromptOptMe-3B on the reported language pairs) or be accompanied by significance tests showing that the decreases are within sampling noise. As written, the abstract overstates the findings.","section":"Abstract, Section 5"},{"comment":"The paper states in Section 3.3 that the simplified instruction 'did not adversely affect the metric quality' and that evaluation quality 'remained comparable' to the original GEMBA-MQM prompt. Table 1 does not support this: GPT-4o lite alone has visibly lower segment-level Kendall tau on all three language pairs compared to GPT-4o ref (e.g., Zh-En 0.2890 vs 0.3692). The quality recovery in the final system comes only after adding PromptOptMe compression. This discrepancy should be acknowledged and the claim reworded, because the rationale for freezing the simplified prompt depends on this assertion.","section":"Section 3.3, Section 5"}],"minor_comments":[{"comment":"The text reports a reduction rate of 2.04x for GPT-4o lite while Table 1 lists 1.84x; the same paragraph says token usage drops to 8.3M while Table 1 says 8.07M. Please harmonize these numbers and double-check all arithmetic in the results section.","section":"Section 5, Table 1"},{"comment":"The Introduction states a 2.32x reduction while the Abstract and Section 5 state 2.37x; please make the headline number consistent throughout.","section":"Introduction"},{"comment":"The caption contains a typo: 'Simplifield Prompt' should be 'Simplified Prompt'.","section":"Appendix A, Figure 4 caption"},{"comment":"The footnote describes 'GPT-4o ref' as 'the original GPT-4' but the experiments use GPT-4o; please correct the model name for consistency.","section":"Table 1 footnote"},{"comment":"The method name is written inconsistently: 'PromptOptMe' in the title and abstract, 'PROMPT OPTME' in the body and tables, and 'PromptOptMe' in the figure. Please standardize to a single spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a metrics/evaluation venue and the practical motivation is clear. The main blocker is statistical evidence for the headline claim; the authors should be able to address this within the manuscript's scope by adding uncertainty quantification and qualifying the claims. I would also ask the editor to ensure the authors address the inconsistency between the abstract's blanket 'no loss' claim and the per-language degradations reported in Table 1."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely useful idea — compress source/MT inputs for GEMBA-MQM while preserving error spans, trained with SFT then ORPO — but the headline '2.37x without any loss' is built on single-run point estimates with no error bars. I'd send it to review, but the authors need to add variance estimates and soften the claim.\n\nWhat's actually new: prior compression work (LLMLingua, BPO, PRewrite) is task-agnostic or general-purpose. PromptOptMe targets the metric itself, preserves MQM error spans, and defines preferences using the metric's own score stability. That combination is distinct and well-motivated. The simplified prompt alone gives 1.84x with a small pairwise accuracy dip, and adding the 3B compressor gets 2.37x while Kendall tau goes up on all three language pairs. The LLMLingua-2 comparison is a useful sanity check — it collapses the metric, supporting the error-span-preservation story.\n\nSoft spots: the central claim isn't established statistically. Pairwise accuracy drops from 0.7789 to 0.7736, which could be noise; no CIs, no bootstrap, no repeated API runs, and the number of MT systems isn't stated. The Kendall tau improvements are also single-run. Because the preference signal comes from GPT-4o's own scores, the independent evidence is the human-correlation table — but that table carries the same variance problem. The abstract says 'without any loss' while the Discussion admits quality sometimes drops; that's an overclaim. The intro says 2.32x, the abstract says 2.37x; tables and text should match. The LLMLingua-2 baseline uses a meeting-bank model, so the comparison is not apples-to-apples, though it's a minor issue given the authors claim task-agnostic weakness.\n\nCredit: the limitations section is honest about MQM-typology drift and the compression model's inference cost, and the authors plan to release code and models. That's real evidence they know the boundaries.\n\nWho it's for: people building cheap MT evaluation pipelines, and researchers working on prompt compression for downstream metrics. It deserves serious referee time. My recommendation: accept with major revisions — error bars or repeated runs, state the system count, align the abstract with the Discussion, and release code/models before treating the result as definitive.","headline":"Useful efficiency result for GEMBA-MQM, but the 'no loss' claim is a single-run point estimate; worth reviewing with a demand for error bars and softened claims.","tokens_in":15924,"tokens_out":3640,"would_cite":true,"duration_ms":30889,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that a small, error-aware language model can compress the inputs of LLM-based translation metrics, cutting token usage 2.37x while preserving—and on segment level improving—the metric's correlation with human judgment.","keywords":["prompt compression","machine translation evaluation","LLM evaluation metrics","GEMBA-MQM","preference optimization","ORPO","token usage reduction","error span preservation"],"falsifier":"Repeat the preference-data generation on a sample of examples, running GPT-4o several times per compression rate: if within-example score variance across runs matches or exceeds the score spread across rates, the chosen/rejected labels are effectively random and the reported quality preservation should not replicate. A second check is to apply the trained compressor to an unseen language pair and require that system-level pairwise accuracy not fall below the uncompressed baseline.","tokens_in":14868,"feed_emoji":"✂️","tokens_out":7681,"duration_ms":60342,"temperature":0.7,"pith_summary":"The paper tries to show that the cost of LLM-based machine translation evaluation can be cut by more than half without sacrificing quality, by using a small fine-tuned language model to compress the source text and translation before they go into the evaluator's prompt. The compressed inputs keep the spans that carry translation errors, and the large evaluator (GPT-4o) still scores them accurately. On a 16k-example WMT22 test set, the method reduces token usage from 19M to 8.07M, a 2.37x reduction, while holding pairwise system accuracy at 0.7736 and improving segment-level Kendall tau over the uncompressed baseline on all three language pairs tested. This would make state-of-the-art metrics like GEMBA-MQM practical for large-scale evaluation and reranking.","feed_headline":"Prompt compression cuts MT evaluation cost 2.37x with no quality loss","feed_subtitle":"A fine-tuned 3B model shrinks GEMBA-MQM inputs while keeping—even improving—correlation with human judgment.","key_machinery":"The load-bearing mechanism is the two-stage fine-tuned compressor built on LLaMA 3.2 (1B and 3B). In stage one, supervised fine-tuning on MQM-annotated WMT data teaches the model to output three things: a compression rate $r$ from {0.3,...,1.0}, the substrings in source and translation that contain errors, and compressed versions of both texts. The training targets are generated by random token removal that preserves all error span tokens, so the model learns to anchor compression around the evidence the MQM metric needs. In stage two, ORPO preference optimization re-ranks the model's compressions: for each example the chosen compression is the rate whose GPT-4o GEMBA-MQM score is closest to the uncompressed score and the rejected one is furthest, so the model is pushed toward compressions that do not change the metric's verdict. A fixed simplified GEMBA-MQM instruction (short system prompt, JSON output) replaces the verbose original and contributes most of the raw token savings.","core_discovery":"The central discovery is that a small model trained to compress translation inputs in an error-aware way preserves—and in some cases improves—the quality of a large LLM-based metric, while generic token-level compression destroys it. PromptOptMe is a LLaMA 3.2 3B model fine-tuned in two stages: first supervised learning on WMT MQM data where random token removal keeps error spans intact, then odds-ratio preference optimization (ORPO) on pairs of compressions labeled by how close their GPT-4o GEMBA-MQM score is to the score for uncompressed text. The resulting compressor, combined with a simplified GEMBA-MQM instruction, achieves the claimed 2.37x token reduction: 8.07M tokens vs. 19M for the uncompressed reference. Segment-level correlation with human judgments improves on all three language pairs tested with GPT-4o (En-Ru $\\tau$ 0.4455, En-De 0.4065, Zh-En 0.3738), and system-level pairwise accuracy equals the baseline at 0.7736. The paper also shows the compressor generalizes to GPT-4o mini and LLaMA 3.2-90B backbones, and that the LLMLingua-2 general-purpose compressor, by contrast, drops segment-level correlation to near zero.","pith_inferences":["The reported 2.37x figure counts only the large evaluator's input tokens; adding the compressor's own inference cost would narrow the true savings, especially for short segments, and a full cost comparison is not in the paper.","If the preference labels are noisy because GPT-4o's scores vary across repeated runs, the chosen/rejected pairs may be partly arbitrary; a repeat-sampling study would tell us how much of the quality preservation is real signal.","The error-span-preservation principle could transfer to other fine-grained evaluation tasks with annotated spans, such as summarization with factual error annotations, though the paper only demonstrates MT.","The success of a fixed simplified instruction suggests prompt designers should quantify how much of a long evaluation prompt is actually load-bearing before optimizing input compression."],"forward_implications":["At 2.37x fewer input tokens, LLM-based MT evaluation becomes cheap enough for large-scale use cases like system reranking and web-scale dataset filtering, not just small test sets.","Because segment-level correlations improve on the tested language pairs, error-aware compression may focus the evaluator on informative spans rather than distracting context.","The same prompt compressor works with different backbone evaluators (GPT-4o, GPT-4o mini, LLaMA 3.2-90B), so the training transfer is not tied to one API.","Generic token-importance compressors are not a viable shortcut for this task; task-specific training that preserves error spans appears necessary to retain evaluation quality.","Simplified instructions alone nearly halve token usage, meaning much of GEMBA-MQM's prompt budget is redundant for evaluation quality."],"supporting_citations":[{"why":"Defines the GEMBA-MQM metric and its full prompt, the evaluation procedure that PromptOptMe compresses and measures.","marker":"Kocmi and Federmann, 2023a"},{"why":"Supplies the ORPO preference optimization algorithm used in the second training stage.","marker":"Hong et al., 2024"},{"why":"Provides the WMT MQM-annotated datasets used for supervised fine-tuning and preference data construction.","marker":"Freitag et al., 2021b"},{"why":"Introduces the LLaMA 3.2 model family used both as the small compressor and as one of the backbone evaluators.","marker":"Dubey et al., 2024"},{"why":"Presents LLMLingua-2, the general-purpose prompt compression baseline whose quality collapse motivates the task-specific approach.","marker":"Pan et al., 2024"},{"why":"Defines pairwise system-level accuracy, the primary metric used to judge whether evaluation quality is preserved.","marker":"Deutsch et al., 2023"},{"why":"Provides the WMT22 Metrics Challenge Test Set used for the main evaluation with 16k examples.","marker":"Freitag et al., 2022"},{"why":"Supplies LoRA, the efficient fine-tuning method used to train the compressor models.","marker":"Hu et al., 2022"}],"fun_headline_variants":["PromptOptMe: 2.37x fewer tokens, same MT evaluation quality","Error-aware prompt compression cuts LLM MT eval cost 2.37x","Fine-tuned compressor preserves GPT-4 MT metric, 2.37x token cut","Generic compression fails; error-aware version keeps MT quality, 2.37x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a compression is good exactly when its GPT-4o GEMBA-MQM score matches the score for the uncompressed text, and that those scores are stable enough to form reliable training preferences.","fun_headline_variants_meta":{"raw":{"variants":["PromptOptMe: 2.37x fewer tokens, same MT evaluation quality","Error-aware prompt compression cuts LLM MT eval cost 2.37x","Fine-tuned compressor preserves GPT-4 MT metric, 2.37x token cut","Generic compression fails; error-aware version keeps MT quality, 2.37x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1623,"prompt_tokens":1010,"completion_tokens":613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":626,"tokens_out":613,"duration_ms":5820,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:46:26.820992+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the preference-data generation on a sample of examples, running GPT-4o several times per compression rate: if within-example score variance across runs matches or exceeds the score spread across rates, the chosen/rejected labels are effectively random and the reported quality preservation should not replicate. A second check is to apply the trained compressor to an unseen language pair and require that system-level pairwise accuracy not fall below the uncompressed baseline.","supporting_citations":[],"review_version":1}