REVIEW 18 cited by
How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. In this paper, we present a comprehensive evaluation of GPT models for machine translation, covering various aspects such as quality of different GPT models in comparison with state-of-the-art research and commercial systems, effect of prompting strategies, robustness towards domain shifts and document-level translation. We experiment with eighteen different translation directions involving high and low resource languages, as well as non English-centric translations, and evaluate the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002. Our results show that GPT models achieve very competitive translation quality for high resource languages, while having limited capabilities for low resource languages. We also show that hybrid approaches, which combine GPT models with other translation systems, can further enhance the translation quality. We perform comprehensive analysis and human evaluation to further understand the characteristics of GPT translations. We hope that our paper provides valuable insights for researchers and practitioners in the field and helps to better understand the potential and limitations of GPT models for translation.
Forward citations
Cited by 18 Pith papers
-
Translation Heads: Disentangling meaning from language in LLM-based machine translation
Across 20 language directions and three model families, roughly 1% of attention heads — split into separable language-identification and meaning-equivalence groups — causally drive LLM translation, and steering them r...
-
ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
A new open-source benchmark evaluates LLM-generated end-to-end ML pipelines from Kaggle competition descriptions translated into 13 languages, with 6 private tasks to limit data leakage.
-
How Important is `Perfect' English for Machine Translation Prompts?
For LLM machine translation, prompt choice affects output quality more than realistic user errors, with spelling errors hurting most and phrase-level errors often harmless.
-
LLMCup: Ranking-Enhanced Comment Updating with LLMs
LLMCup combines multi-prompt LLM comment generation with a learned ranker, achieving 0.383 exact-match accuracy on the CUP test set versus 0.257 for HebCup and 0.177 for CUP.
-
Prompting LLMs: Length Control for Isometric Machine Translation
Instruction-example alignment and extreme short demonstrations drive length control in LLM translation; selecting among multiple outputs improves the length-quality tradeoff.
-
Exploring In-context Example Generation for Machine Translation
DAT generates query-specific in-context translation examples using only an LLM, improving English-to-low-resource translation over zero-shot in most tested languages.
-
Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity
SNNE estimates LLM uncertainty from pairwise semantic similarities of sampled answers using a log-sum-exp aggregation, and it generalizes semantic entropy as a special case.
-
Compensating for Data with Reasoning: Low-Resource Machine Translation with LLMs
Fragment-Shot Prompting, which retrieves example translations for sentence fragments, substantially improves LLM translation into and between Ladin variants but not into Italian.
-
Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation
Prompt scope and demonstration selection materially change local LLM translation quality and compliance, and dedicated MT systems still outperform them overall.
-
Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
On Swiss legal translation, reinforcement learning with a ChrF reward improves small open models more than supervised fine-tuning, but frontier reasoning models still score higher.
-
The Role of Prompt Language and Translation-Theory-Driven Prompts in Large Language Models: A Case Study on Spanish-Chinese Journalistic Translation
On four EL PAÍS editorials, brief-oriented theory-driven prompts beat a baseline under human MQM (8.66 vs 7.84) but lose under BLEU/BERTScore; prompt language has negligible effect.
-
$M^2PO$: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation
M2PO combines a QE-plus-alignment reward with a dynamic curriculum and multi-pair DPO loss, and reports WMT21-22 gains for a 7B translation model, but the abstract's WMT23/24 9B parity claims are unsupported.
-
Psychology-Driven Enhancement of Humour Translation
A decomposition-and-recomposition prompt method for humor translation reports large gains on LLM-based metrics, but the evaluation lacks human validation and statistical checks.
-
Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation
A two-stage pipeline combining human-curated Singlish examples with embedding-based LLM ranking selects GPT-4o mini for toxicity-preserving translation, reaching gold-level human scores for Chinese and Malay but not Tamil.
-
Invariant-based Robust Weights Watermark for Large Language Models
An invariant-based weights watermark embeds per-user keys into the null space of transformer invariants and uses noise to repel collusion.
-
TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration
TACTIC, a cognitive-inspired six-agent workflow, improves LLM translation quality over direct prompting on FLORES-200 and WMT24, with the best DeepSeek-V3 setup reaching 96.19 XCOMET on English-to-X.
-
Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models
A survey of context-aware machine translation with large language models, categorizing prompting, fine-tuning, and agent-based approaches.
-
Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons
DeepSeek-R1 outperforms GPT-4o and DeepSeek-V3 on family tree and graph reasoning benchmarks at sizes 10 and 20, but all models collapse at size 40.
Discussion (0). Sign in to comment.