{"id":"a0ad18a1-b146-4850-bf84-b8bdc33b6afc","arxiv_id":"2506.13070","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Combining gold-entity retrieval with GPT-4o self-refinement improves entity-aware machine translation across ten languages in SemEval-2025 Task 2.","lead":"This paper describes a system for entity-aware machine translation that combines retrieval of gold Wikidata labels with an iterative self-refinement loop driven by GPT-4o. The authors report large gains from retrieval and smaller, consistent gains from self-refinement across ten languages in the SemEval-2025 shared task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RAG gain is an oracle lookup, so the load-bearing weakness is that the +Refine gains may come from a second pass with the gold label in the prompt rather than from the self-evaluation mechanism, with no control or significance test.","rationale":"The reader correctly identified the oracle-ID dependence and the lack of variance reporting. My stress test focuses on a more specific, load-bearing gap: the self-refinement component is the paper's distinctive contribution, but the experimental design does not rule out the trivial explanation that the model simply benefits from another look at the gold label already present in the prompt. This is not an accusation of dishonesty; it is an internal-control problem. The proposed concrete test is a cheap ablation that separates 'second pass with gold information' from 'self-evaluation feedback.' If the ablation matches +Refine, the central claim about self-refinement is not supported. If it does not match, the claim gains real support. Either outcome is informative, and the reader's conditional verdict already accommodates this uncertainty, so no verdict change is required.","tokens_in":7674,"tokens_out":2566,"duration_ms":31336,"concrete_test":"For each language in Table 2, replace the feedback/refine stage with a second-pass prompt that receives the same source text, the same gold entity information e, and the initial +RAG translation, and is instructed only: 'If any entity label differs from the reference label, correct it; otherwise keep the translation.' Use the same number of extra LLM calls and the same maximum iterations as +Refine. If the resulting harmonic mean matches the +Refine column within 0.1 percentage points across languages, then the self-evaluation feedback is not the cause of the reported gains, and the 'self-refinement consistently enhances' claim should be downgraded. Repeat each configuration at least 3 times to assess whether the 0.19-1.66 pp differences are within run-to-run variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two components: RAG with gold IDs substantially improves translation (true by construction, since the prompt already contains the gold target-language label), and self-refinement consistently improves over +RAG (the distinctive contribution). The second component is under-identified. In Equations (2)-(3), both the feedback and refinement prompts include entity information e, i.e., the gold Wikidata label and description. The refinement loop therefore gives the model a second chance to read the correct label and correct its previous output. The reported +Refine gains of 0.19-1.66 percentage points could be caused by this second exposure to e, or by simply using additional decoding passes, rather than by the model's self-scoring/feedback mechanism. No control condition separates these explanations. Table 2 reports a single run per configuration with no variance or significance analysis, so the claim that refinement 'consistently enhances translation quality across all language pairs' rests on very small differences that could be sampling noise. The case study in Table 3 is consistent with this concern: the entity-feedback example simply supplies the correct Korean title in the feedback text. The paper's Section 7 limitation about gold entities is acknowledged, but the attribution problem for the self-refinement component is not addressed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes CHILL, the Samsung Research submission to SemEval-2025 Task 2 on entity-aware machine translation (EA-MT). The system takes the gold Wikidata ID attached to each English test sentence, fetches the corresponding target-language label and description from the Wikidata REST API, and inserts this entity information into a GPT-4o translation prompt; it then runs a self-refinement loop in which the same model scores its own translation on entity correctness and overall quality (5 points each, total 10) and iteratively revises the translation, with a maximum of two iterations. On the official test set, the harmonic mean of M-ETA and COMET rises from 33-62 for a plain GPT-4o baseline to 85-94 with RAG, and by an additional 0.19-1.66 percentage points with refinement. The paper reports per-language results for all ten target languages, per-metric COMET and M-ETA scores in Appendix B, two qualitative case studies, full prompt templates, and a correlation analysis between English-foreign label edit distance and entity accuracy.","tokens_in":1968,"tokens_out":2575,"duration_ms":159918,"significance":"If the results are taken at face value, the paper shows that prompt-injected gold entity labels and descriptions drive large, uniform gains in entity-aware translation across ten languages, and that a second pass with self-feedback yields modest further gains on the official harmonic-mean metric. The manuscript's strengths are its transparency and simplicity: Table 5 reports COMET and M-ETA separately so the headline harmonic means can be independently verified; the initial-translation and feedback prompts are given verbatim in Listings 1-2; the system uses a single off-the-shelf model with no fine-tuning; and Section 7 openly acknowledges the reliance on gold Wikidata IDs. The scientific scope is, however, narrower than the abstract suggests: the 'RAG' step is an oracle lookup keyed to the test set's gold IDs rather than a retrieval system, and the distinctive self-refinement claim is under-identified (major comments 1-2). The paper is a solid, reproducible system description whose conclusions about the refinement mechanism require further controlled evidence before they can be credited beyond the specific task setup.","major_comments":[{"comment":"The attribution of the +Refine gains to the self-evaluation mechanism is under-identified, and this is load-bearing for component (ii) of the central claim in §7. Both the feedback prompt (Eq. 2) and the refinement prompt (Eq. 3) contain the entity information e, which includes the gold target-language label, and the case study in Table 3 shows that the model's own feedback text spells out the correct label verbatim ('붉은군대는가장강력하다'). The same case study shows the model had that label in its initial prompt and did not use it, so the gain is not simply re-exposure to previously withheld information; nevertheless, the current design cannot separate the claimed self-scoring mechanism from (a) the explicit gold-labeled correction that the feedback step inserts into the context, or (b) the benefit of an additional decoding pass. No ablation removes the feedback text, removes e from the refine prompt, or re-translates without feedback. Table 2 further reports a single run per configuration, with no variance or significance testing, so 'consistently enhances translation quality across all language pairs' rests on differences as small as 0.19 percentage points that could be sampling noise. I recommend control conditions on the validation set (refine with feedback but without e; re-translation without feedback; repeated decoding) to identify the active ingredient.","section":"§4.2, Eqs. (2)-(3); Tables 2 and 3; §7"},{"comment":"The paper's own per-metric results partially contradict the claim that refinement improves entity handling. From +RAG to +Refine, M-ETA decreases in 6 of 10 languages (AR 92.17 to 91.86, DE 85.46 to 85.23, ES 90.61 to 89.88, FR 90.61 to 89.95, IT 92.64 to 92.43, ZH 78.06 to 77.77), while COMET increases in all 10 languages; the harmonic mean in Table 2 consequently rises even where the entity-specific metric falls. Since §4.2 motivates the feedback mechanism with an equally weighted 5-point entity-correctness criterion, the mechanism's advertised benefit is absent from the entity metric for most language pairs. The §7 statement that 'self-refinement mechanism consistently enhances translation quality across all language pairs' is defensible only for COMET and the harmonic mean; it should be qualified, and the authors should discuss why the entity score degrades in the majority of language pairs despite the entity-focused feedback criterion.","section":"Appendix B (Table 5); §§6.1 and 7"},{"comment":"The RAG component is an oracle lookup: the test set provides the gold Wikidata IDs, and the retrieved entity information includes the gold target-language label, so Eq. (1) places the answer-key label directly in the prompt. The headline gain in Table 2 (harmonic mean from roughly 33-62 to 85-94) is therefore largely a property of the task setup rather than evidence for retrieval-augmented translation as a general method; the M-ETA values of 78-92 after RAG show the model does not always copy the provided label, so the gain is not fully automatic, but the experiment never tests entity retrieval or linking. Section 7 does acknowledge the gold-entity reliance in its future-work sentence, but the abstract and §7 conclusion still present RAG as a generally effective component. I suggest reframing the contribution as an in-task upper bound for entity-aware prompting with perfect per-sentence entity information, and stating explicitly in the abstract that no retrieval is performed.","section":"§4.1 and §7"}],"minor_comments":[{"comment":"There is a subject-verb agreement error in 'These retrieved information has been successfully applied'; the manuscript needs a careful copy-edit pass.","section":"§2.1"},{"comment":"The title's phrasing 'Make Your LLM to Get Them Right' is ungrammatical; consider 'Make Your LLM Get Them Right'.","section":"Title"},{"comment":"Please report the average number of feedback-refinement cycles actually executed and the fraction of sentences that terminated on a perfect score; this would substantiate the linear-cost discussion in the same section.","section":"§4.2"},{"comment":"Please state whether the GPT-4o-generated few-shot examples for the nine non-Korean languages were manually checked, since the quality of the feedback exemplars directly affects the refinement behavior being evaluated.","section":"§5.2"},{"comment":"Since M-ETA is treated as a binary outcome in this analysis, Spearman's rho and the point-biserial r convey nearly the same association; reporting confidence intervals or significance levels would make the 'little impact' conclusion quantitative.","section":"§6.3, Table 4"},{"comment":"The iteration prompt template is given only with placeholders such as '(entity information)' and '(language)'; a fully instantiated example, matching the level of detail in Listings 1-2, would make the refinement step fully reproducible.","section":"Appendix A"},{"comment":"The column header 'GPT-4o +RAG +Refine' with C/M subcolumns is ambiguous; expanding 'C' and 'M' to 'COMET' and 'M-ETA' would improve readability.","section":"Appendix B, Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a competent shared-task system-description paper with transparent reporting, but it is thin on scientific novelty for a journal venue: the headline RAG gain is an oracle effect of the task setup, and the distinctive refinement claim needs the control experiments and re-analysis requested in the major comments. The single self-citation (Lee and Shin 2024) appears in related work on constrained decoding and is tangential rather than load-bearing; I do not see an improper citation pattern. I would support publication after a revision that adds the ablations and qualifies the two main claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The RAG jump is real but mostly an oracle lookup: the test set ships gold Wikidata IDs, and the system puts the gold target-language label straight into the prompt. The +Refine gains (0.19–1.66 pp) are consistently positive across ten languages, but they come from a single run each and are confounded: the feedback and refinement prompts both re-include the gold entity info, so a second reading of the correct label could explain the improvement. The paper never runs the control that would separate \"the model scored its own output\" from \"the model saw the answer key again.\"\n\nWhat's good: clearly written, the numbers in Tables 2 and 5 are concrete, and the two-criteria feedback (entity label accuracy + overall quality) is a sensible design. The Section 6.3 correlation analysis (label similarity vs M-ETA) is a nice extra. The case studies in Table 3 are illustrative, though the entity-feedback example rather gives away the confound: the feedback text literally supplies the correct Korean title.\n\nWhere it's soft: the oracle dependence is acknowledged only in the final section (\"future work could explore ... without using gold entity\"), but it is the single biggest reason the headline RAG number would not transfer to a real pipeline. That belongs in the abstract. The \"consistently enhances\" claim rests on ten pairwise gains from a single run each, with no variance or significance testing. That is a real gap for a claim about a method's mechanism. And the release is incomplete—the few-shot sets are described but not given, which hurts reproducibility.\n\nBottom line: as a shared-task system description, this is fine and useful to people entering EA-MT. The RAG-with-gold-ID recipe is easy to copy, and the M-ETA numbers are the kind of data practitioners like. But the self-refinement contribution is not demonstrated to be more than a second pass with the answer key. I would send it to peer review—it is a legitimate empirical report with concrete numbers—but I would require the authors to add a control (e.g., +RAG with a plain re-generation, no feedback), report multiple runs with variance, and move the oracle-dependence caveat to the abstract. With those, it is a solid workshop paper; without them, the +Refine claim stays speculative.","headline":"Solid SemEval system description with a big oracle-driven RAG gain and a small, under-identified refinement gain; the +Refine claim needs a control before I would trust it.","tokens_in":8467,"tokens_out":4008,"would_cite":false,"duration_ms":41987,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that feeding a GPT-4o translator gold Wikidata entity labels and descriptions, then having the model critique and revise its own output, raises entity-aware machine translation scores from the low 50s to the low 90s…","keywords":["entity-aware machine translation","retrieval-augmented generation","self-refinement","Wikidata","large language models","transcreation","M-ETA","COMET"],"falsifier":"Run the same pipeline on test sentences whose gold Wikidata IDs are replaced by IDs from an off-the-shelf entity linker, or by randomly corrupted IDs, and measure the M-ETA and COMET harmonic mean. If the score collapses toward the GPT-4o baseline of 33–63, the gains come from the oracle lookup rather than from the RAG-and-refine mechanism being tested.","tokens_in":7449,"feed_emoji":"🌐","tokens_out":6391,"duration_ms":60851,"temperature":0.7,"pith_summary":"This paper addresses entity-aware machine translation, where named entities must be rendered by their established target-language names rather than literal translations. It argues that giving GPT-4o the gold Wikidata labels and descriptions for the entities in each sentence, via retrieval-augmented generation, lifts the harmonic mean of entity accuracy (M-ETA) and overall quality (COMET) from roughly 33–63 to 85–94 across ten languages. Adding a self-refinement loop, in which the same model scores its own translation (entity correctness 5, translation quality 5) and revises accordingly, yields consistent but smaller gains of 0.19–1.66 percentage points. The paper's core message is that entity information must be actively injected and explicitly checked; the model will not reliably use it otherwise.","feed_headline":"Entity-aware MT jumps ~40 points when given Wikidata IDs","feed_subtitle":"A RAG-plus-self-refine system beats the GPT-4o baseline on all ten languages in SemEval-2025 Task 2.","key_machinery":"The load-bearing machinery is the RAG prompt plus the self-refinement loop. Given a source text $x$, guidance prompt $p_{\\mathrm{gen}}$, and entity information $e$ (target-language label, English label, description) fetched from Wikidata, the initial translation is $y_0 = M(p_{\\mathrm{gen}} \\| e \\| x)$. A feedback prompt $p_{\\mathrm{fb}}$ makes the same model score the translation on two equally weighted criteria, entity correctness and overall quality, on a 10-point scale aligned with the task metric; a refinement prompt $p_{\\mathrm{rf}}$ then rewrites the translation using the full history $y_0, fb_0, \\dots, y_t, fb_t$. The design lets the model correct both wrong entity names (e.g., choosing the established Korean title '붉은 군대는 가장 강력하다' over a literal rendering) and semantic errors in the sentence.","core_discovery":"On the paper's own terms, the central discovery is that retrieval-augmented generation with gold entity information from Wikidata substantially improves entity-aware machine translation, and that self-refinement consistently adds further improvement. The pipeline first retrieves entity labels and descriptions using the Wikidata REST API, inserts them into a few-shot prompt, and generates an initial translation $y_0 = M(p_{\\mathrm{gen}} \\| e \\| x)$. A feedback step scores the translation on entity-label accuracy and overall quality, and a refinement step produces $y_{t+1} = M(p_{\\mathrm{rf}} \\| e \\| x \\| y_0 \\| fb_0 \\| \\dots \\| y_t \\| fb_t)$. The process stops at a perfect score or after two iterations. Across Arabic, German, Spanish, French, Italian, Japanese, Korean, Thai, Turkish, and Chinese, the harmonic mean of M-ETA and COMET rises from a GPT-4o baseline of 33–63 to 85–94 after RAG, and then by 0.19–1.66 points after refinement.","pith_inferences":["Without the gold Wikidata IDs, the pipeline would need an entity linking step; the reported margin over the baseline would likely shrink because retrieval errors would propagate into the prompt. This is a testable extension flagged by the authors as future work.","The small gains from self-refinement suggest the model already uses much of the injected entity information in the initial RAG pass; the feedback loop mainly catches residual errors.","The two-criteria scoring scheme could transfer to other knowledge-grounded generation tasks where a named resource (a label, a definition, a canonical form) must be respected in the output.","Because the few-shot feedback examples for nine languages were machine-translated from Korean templates, the feedback quality may vary by language; checking per-language refinement gains (0.19–1.66) against human feedback quality would clarify this."],"forward_implications":["Adding RAG with gold Wikidata entity info improves the harmonic mean of M-ETA and COMET by large margins on all ten language pairs tested.","Self-refinement with the two-criteria scoring rubric improves scores on all ten language pairs, with gains between 0.19 and 1.66 percentage points.","The model's feedback loop can correct both entity label errors (literal translation) and general translation errors (misreading the source intent).","The two-criteria feedback design aligns the model's self-evaluation with the official metric, the harmonic mean of M-ETA and COMET.","Entity label similarity between English and the target language (measured by edit distance) has little influence on translation accuracy, so other factors drive errors."],"supporting_citations":[{"why":"Defines the SemEval-2025 Task 2 dataset and the official harmonic-mean evaluation used for all reported numbers.","marker":"(Conia et al., 2025)"},{"why":"Supplies Wikidata, the knowledge base from which entity labels and descriptions are retrieved.","marker":"(Vrandečić and Krötzsch, 2014)"},{"why":"Provides the self-refine framework of iterative self-feedback that the paper adapts for entity-aware translation.","marker":"(Madaan et al., 2024)"},{"why":"Introduces RAG for cross-cultural MT and the M-ETA metric, the entity accuracy measure in the task score.","marker":"(Conia et al., 2024)"},{"why":"Supplies COMET, the overall translation quality metric in the harmonic mean.","marker":"(Rei et al., 2020)"},{"why":"Provides the basic translation prompt used as the GPT-4o baseline.","marker":"(Xu et al., 2024)"}],"fun_headline_variants":["RAG with Wikidata IDs lifts entity MT scores to 85-94","Entity-aware MT: RAG + self-refine beats GPT-4o baseline","Wikidata IDs in RAG boost entity translation accuracy","SemEval-2025: RAG plus self-refinement for entity MT","From 63 to 94: RAG with entity IDs lifts MT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes every test sentence comes with a correct gold Wikidata ID and that the Wikidata REST API returns an accurate target-language label and description; the model performs no entity linking or retrieval of its own, so the reported gains depend entirely on that oracle lookup.","fun_headline_variants_meta":{"raw":{"variants":["RAG with Wikidata IDs lifts entity MT scores to 85-94","Entity-aware MT: RAG + self-refine beats GPT-4o baseline","Wikidata IDs in RAG boost entity translation accuracy","SemEval-2025: RAG plus self-refinement for entity MT","From 63 to 94: RAG with entity IDs lifts MT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000506,"raw_usage":{"total_tokens":2440,"prompt_tokens":891,"completion_tokens":1549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1453}},"tokens_in":507,"tokens_out":1549,"duration_ms":13745,"temperature":1.0,"reasoning_tokens":1453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:37:09.282011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on test sentences whose gold Wikidata IDs are replaced by IDs from an off-the-shelf entity linker, or by randomly corrupted IDs, and measure the M-ETA and COMET harmonic mean. If the score collapses toward the GPT-4o baseline of 33–63, the gains come from the oracle lookup rather than from the RAG-and-refine mechanism being tested.","supporting_citations":[],"review_version":1}