Pith. sign in

REVIEW 4 major objections 6 minor 12 references

KoGEC : Korean Grammatical Error Correction with Pre-trained Translation Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a 3.3B translation model with a special token yields a Korean grammatical error corrector that outscores GPT-4o and HCX-3 on BLEU.

desk verdict A genuinely useful engineering result that overclaims its evaluation; the 3.3B NLLB fine-tune is promising but the BLEU-only, same-corpus comparison doesn't yet establish superiority over GPT-4o/HCX-3. read the letter →

arxiv 2506.11432 v1 pith:F7FXV5VB submitted 2025-06-13 cs.CL cs.AI

Taeeun Kim , Semin Jeong , Youngsook Song This is my paper
classification cs.CLcs.AI
keywords KoreanGrammaticalErrorCorrectionNLLBmachinetranslationfine-tuningLLM-as-judgeBLEUtokenvocabularyexpansionsocialmediatext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a compact, task-specific model can outperform much larger general-purpose chatbots at Korean grammatical error correction. The authors fine-tune NLLB, a multilingual translation model, to map erroneous Korean sentences to corrected ones by treating the correction as a translation between two Korean variants marked with a special token. On a 5,253-sentence test set drawn from social-media conversation corpora, the fine-tuned 3.3B model scores 85.73 BLEU, above GPT-4o's 75.03 and HCX-3's 71.24. An LLM-judge analysis finds KoGEC's remaining-error profile more evenly spread across 11 error types, whereas GPT-4o and HCX-3 concentrate on punctuation. The paper also reports that expanding NLLB's Korean token vocabulary degraded performance.

What carries the argument

The machine that carries the argument is the NLLB-200 multilingual translation model fine-tuned with a special <cor_Hang> token. NLLB normally marks source and target languages with tokens such as <kor_Hang>; the authors add <cor_Hang> to mark the corrected Korean sentence, turning grammatical error correction into a translation-like mapping from erroneous to corrected Korean. Fine-tuning with the Adafactor optimizer on a single A100 GPU for about 13 hours produces the 600M and 3.3B variants, and the 3.3B model carries the reported result. The token is what makes the model treat correction as a controllable, minimal-edit translation direction rather than open-ended paraphrase.

What would settle it

A human-annotation study on a held-out corpus from a different genre (e.g., formal essays, news text, or learner Korean) that scores each model's output for grammaticality, meaning preservation, and edit necessity would settle whether the BLEU advantage reflects genuine GEC quality or only n-gram overlap on in-distribution social-media text.

Watch

Extended reading notes

Core claim

The central discovery is that adding a single special token, <cor_Hang>, to NLLB and fine-tuning on roughly 520k pairs of original and corrected Korean social-media sentences converts a multilingual translation model into a Korean GEC system that, by BLEU, produces outputs closer to human references than either GPT-4o or HCX-3 under zero-shot prompting. The <cor_Hang> token tells the model which side of the pair is the corrected Korean, letting it treat error correction as a translation direction rather than a paraphrasing task. In the LLM-as-judge error-type classification, KoGEC shows a more balanced profile: punctuation errors account for 29.8% of its residual errors versus 43.8% for GPT-4o and 52.3% for HCX-3, while word-spacing errors are 21.3% for KoGEC. The authors interpret the balance as evidence that a focused, smaller model can deliver more comprehensive correction for native-speaker errors than general-purpose LLMs. The paper's negative result is that adding a Korean-specific SentencePiece vocabulary of about 22k tokens to NLLB caused the 3.3B model to overfit and underperform, which the authors attribute to insufficient pretraining of the added tokens.

Load-bearing premise

The load-bearing premise is that BLEU score against the reference corrected sentences is a valid measure of grammatical error correction quality; BLEU rewards n-gram overlap and can be inflated by conservative minimal edits, and the test set comes from the same two corpora used for training, so the comparison mixes correction ability with distribution fit.

Editorial extensions

If this is right

  • If the BLEU comparison holds, a 3.3B translation model fine-tuned for one language can beat much larger general-purpose LLMs on Korean GEC without any in-context grammar guidelines.
  • KoGEC's error-type balance implies that users would see fewer uncorrected word-spacing, deletion, pronoun, and verb/adjective errors than with GPT-4o or HCX-3, which concentrate their edits on punctuation.
  • The <cor_Hang> recipe is a direct transfer of the translation paradigm to GEC, so it may extend to other languages with similar typology, such as Japanese or Chinese, as the authors propose.
  • The vocabulary-expansion failure suggests that merely adding Korean tokens to NLLB's tokenizer is not enough; the added tokens need pretraining, so future improvements should target token pretraining rather than tokenizer size alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the test set is drawn from the same two corpora used for training, part of the BLEU gap may reflect distribution fit; a held-out domain such as formal documents or learner writing could narrow the advantage.
  • Inference: the residual-error percentages from the LLM judge do not distinguish between 'error type the model fails to fix' and 'error type the model introduces,' so a human annotation study is needed to interpret the balance claim as correction quality.
  • Inference: the authors' minimal-edit philosophy suggests KoGEC should preserve author style better than rephrasing LLMs; an edit-distance-ratio metric or human style-preservation rating would test this directly.
  • Inference: if the reported 3.3B size threshold is real, scaling the same recipe to larger NLLB-style models may yield further gains, but the tokenizer result warns that scaling must include pretraining of any newly added vocabulary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces KoGEC, a Korean grammatical error correction (GEC) system built by fine-tuning NLLB translation models (600M and 3.3B parameters) on two Korean social-media corpora. The authors compare these models against GPT-4o and HCX-3 using BLEU scores and an LLM-as-judge error-type analysis. They report that the 3.3B KoGEC model achieves a BLEU score of 85.73, substantially higher than GPT-4o (75.03) and HCX-3 (71.24), and that KoGEC shows a more balanced error-type profile. The paper also describes a Chrome extension and an exploration of tokenizer vocabulary expansion, which was found to decrease performance.

Significance. The task is practically important: Korean GEC has real applications in writing assistance and data quality control, and the idea of adapting multilingual translation models to GEC is a reasonable and potentially cost-effective direction. If the empirical claims were fully supported, a compact 3.3B model outperforming much larger proprietary LLMs on Korean GEC would be a noteworthy result. The paper also provides a new error-type taxonomy for Korean and releases its models under an open license, which are useful contributions. However, the central comparison is currently under-supported because the evaluation rests on BLEU over a test set drawn from the same corpora used for training, with no variance or significance testing, no identity baseline, and an LLM judge that evaluates its own system output.

major comments (4)
  1. [Section 5.1, Table 2] The central comparative claim (KoGEC 3.3B BLEU 85.73 vs 75.03 for GPT-4o and 71.24 for HCX-3) rests entirely on BLEU computed on a test split drawn from the same two corpora used for fine-tuning (Section 4.1, Table 1). Because the fine-tuned model was trained on original/corrected pairs from these exact corpora, the score partly measures distribution fit rather than general Korean GEC ability. The paper does not report an identity baseline (copying the input), does not state whether near-duplicate train/test pairs were removed, and reports no variance or significance testing. This evidence is insufficient to support the claim that KoGEC outperforms GPT-4o/HCX-3; the authors should add a held-out or cross-corpus evaluation, an identity baseline, significance tests (e.g., bootstrap resampling), and/or a recognized GEC metric such as ERRANT.
  2. [Section 5.2, Table 3] The error-type analysis uses GPT-4o as the judge to classify the outputs of the systems, and one of the evaluated systems is GPT-4o itself. Although the reference-guided grading method is intended to reduce bias, the paper does not validate the judge's classifications against human annotations or report inter-annotator agreement. The statement that this approach 'minimiz[es] self-enhancement bias' is not demonstrated. The error-type proportions in Table 3 may therefore be artifacts of the judge's own preferences rather than real differences among the systems.
  3. [Abstract, Section 5.1, Section 6] The abstract and conclusion state that the fine-tuned NLLB (KoGEC) models (plural) outperformed GPT-4o and HCX-3, but Table 2 shows that NLLB-200-ko-gec-600M achieved BLEU 58.15, far below both LLMs. Only the 3.3B model outperforms them. The plural claim should be corrected, and the conclusion that 'model size should be at least 3.3B' is based on a single model and is therefore under-supported.
  4. [Section 5.1] The paper justifies BLEU as a GEC metric with the statement that 'Since the metric compares model outputs to human-translated reference text, we determined that it would be appropriate to judge GEC quality as well.' This is insufficient: BLEU rewards n-gram overlap and can be inflated by conservative, minimal-edit outputs, a known issue in GEC evaluation. Without additional metrics (e.g., ERRANT, GLEU) or human evaluation, the headline BLEU gap of 10.7 points may overstate the practical superiority of KoGEC.
minor comments (6)
  1. [Section 3, Table 1] The corpus statistics in Table 1 (393k+127k=520k train; 4k+1k=5k test) do not match the text in Section 4.1, which reports 520,015 training rows and 5,253 test rows (total 525,268). Please reconcile these numbers.
  2. [Section 7.1, Figures 1 and 2] The captions of Figures 1 and 2 introduce 'match rate' and 'matched data' without defining these terms in the text, and the figures are not referenced in the main narrative. Please define the terms and reference the figures where the corresponding analysis is discussed.
  3. [Section 4.2] The sentence 'Training data consisted of original and corrected sentence pairs, with batches generated by randomly selecting two language pairs' is unclear; it is not explained why two language pairs are selected or whether non-Korean pairs were mixed into the fine-tuning data.
  4. [Appendix A, Table 3] The error taxonomy in Appendix A lists 12 categories including SHORT (affix errors), but Table 3 reports 11 categories with no SHORT row. Please clarify whether SHORT was merged with another category or excluded from the analysis.
  5. [Footnote 3] The footnote reports that Unicode normalization increased BLEU by up to 3.12 for NLLB-200-ko-gec-3.3B, but no normalized-versus-un-normalized comparison is given for the other models. Reporting the effect of normalization for all systems would make the evaluation more transparent.
  6. [References] Several references contain formatting errors, including 'HyperCLOV A X AI Team', 'InProceedings', and inconsistent punctuation; please correct these throughout the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the KoGEC result is an empirical fine-tuning and evaluation, not a derivation that reduces to its inputs.

full rationale

The paper's central claim is empirical: NLLB-200-ko-gec-3.3B reaches BLEU 85.73 on a 5,253-row test split after fine-tuning on 520k original/corrected pairs, compared with zero-shot GPT-4o (75.03) and HCX-3 (71.24). There is no fitted parameter renamed as a prediction and no equation whose output is identical to its input. The BLEU score is computed against human-corrected references and is independent of the model's training objective, and the comparison systems are external. The test split comes from the same two corpora used for training, which is a distribution-fit confound rather than circularity: it affects what the BLEU gap means, not whether the evaluation is self-contained. The LLM-as-judge error-type analysis uses GPT-4o both as a compared system and as the judge, but the reference-guided grading compares each model output against the provided corrected reference, so the judge is not grading its own output by construction; this is a validity concern, not a circular reduction. The token-vocabulary-expansion limitation is reported as a failed experiment. No load-bearing argument is supported only by a self-citation, and no uniqueness or ansatz is imported from the authors' prior work. The empirical comparison is therefore not circular.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical, so the ledger captures evaluation and modeling assumptions rather than derivation parameters. The main free choices are training hyperparameters, the special correction token, and the error taxonomy; the key domain assumptions concern BLEU validity, test-set representativeness, and GPT-4o as an unbiased judge.

free parameters (5)
  • Training hyperparameters (batch size, learning rate warm-up, checkpoint interval) = batch 64 (600M) / 16 (3.3B), 1,000 warm-up updates, 2,000-step checkpoint interval
    Hand-chosen values with no reported sweeps or sensitivity analysis; the best checkpoint is selected on a development set, which introduces additional unstated choices (Section 4.2).
  • Maximum sequence length = 128 tokens
    Chosen to fit sentence pairs; truncates longer sentences, which may bias results on longer social media posts (Section 4.2).
  • Special token <cor_Hang> = one added token
    Added to mark corrected sentences during fine-tuning; the authors report better GEC susceptibility but provide no ablation or quantitative comparison without the token (Section 4.2).
  • Error-type taxonomy reduction = 11 types (from 14)
    Researchers manually dropped learner-specific categories from Yoon et al.'s taxonomy; the LLM-as-judge error distribution depends on this subjective choice (Section 5.2, Appendix A).
  • Tokenizer expansion size = 278k tokens (32k Korean SentencePiece added to 256k NLLB)
    Used in the vocabulary expansion experiment, which decreased performance and is not part of the main model; the exact weight transfer procedure is described only qualitatively (Section 7.1).
assumptions (4)
  • domain assumption BLEU is an appropriate metric for GEC quality
    The paper asserts BLEU is appropriate because it compares outputs to reference text, but BLEU does not measure whether grammar errors were actually fixed and can favor conservative outputs. This assumption underlies the main comparison in Table 2 (Section 5.1).
  • domain assumption Test set drawn from the same corpora is representative of Korean GEC
    The 5,253 test sentences come from the same NIKL and NIA corpora used for training; the comparison to zero-shot LLMs therefore conflates in-distribution performance with general GEC ability (Section 4.1).
  • domain assumption GPT-4o can judge error types without self-enhancement bias
    GPT-4o is used as the LLM judge to classify errors in outputs including GPT-4o's own outputs; reference-guided grading reduces but does not eliminate self-enhancement bias (Section 5.2).
  • domain assumption NLLB translation modeling is a valid framework for Korean GEC
    The paper follows Luhtaru et al. in treating error correction as translation; this is a reasonable transfer but unvalidated for Korean beyond BLEU scores (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of KoGEC : Korean Grammatical Error Correction with Pre-trained Translation Models." pith.science (2026). https://pith.science/paper/F7FXV5VB

@misc{pith2026250611432,
  author       = {Pith},
  title        = {Pith review of: KoGEC : Korean Grammatical Error Correction with Pre-trained Translation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7FXV5VB}},
  note         = {Machine review of arXiv:2506.11432}
}
read the original abstract

This research introduces KoGEC, a Korean Grammatical Error Correction system using pre\--trained translation models. We fine-tuned NLLB (No Language Left Behind) models for Korean GEC, comparing their performance against large language models like GPT-4 and HCX-3. The study used two social media conversation datasets for training and testing. The NLLB models were fine-tuned using special language tokens to distinguish between original and corrected Korean sentences. Evaluation was done using BLEU scores and an "LLM as judge" method to classify error types. Results showed that the fine-tuned NLLB (KoGEC) models outperformed GPT-4o and HCX-3 in Korean GEC tasks. KoGEC demonstrated a more balanced error correction profile across various error types, whereas the larger LLMs tended to focus less on punctuation errors. We also developed a Chrome extension to make the KoGEC system accessible to users. Finally, we explored token vocabulary expansion to further improve the model but found it to decrease model performance. This research contributes to the field of NLP by providing an efficient, specialized Korean GEC system and a new evaluation method. It also highlights the potential of compact, task-specific models to compete with larger, general-purpose language models in specialized NLP tasks.

Figures

Figures reproduced from arXiv: 2506.11432 by the authors.

Figure 1
Figure 1. Percentage of matched data by Korean GEC assistant. HCX and GPT-4o have match rates of 35.05% and [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of error types across three models: GPT-4o, HCX-3, and Ko-GEC. Comparative Analysis of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    HyperCLOVA X AI Team . 2024. https://arxiv.org/abs/2404.01954 Hyperclova x technical report . Preprint, arXiv:2404.01954

    arXiv 2024

  4. [4]

    Agnes Luhtaru, Elizaveta Korotkova, and Mark Fishel. 2024. No error left behind: Multilingual grammatical error correction with pre-trained translation models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1209--1222, St. Julian's, Malta. Association for Comput...

    work page 2024

  5. [5]

    Junghwan Maeng, Jinghang Gu, and Sun-A Kim. 2023. https://api.semanticscholar.org/CorpusID:267410996 Effectiveness of chatgpt in korean grammatical error correction . In Pacific Asia Conference on Language, Information and Computation

    work page 2023

  6. [6]

    Ministry of Culture, Sports, and Tourism . 2017. Korean spelling evaluation criteria. Notice No. 2017-12 (March 28, 2017)

    work page 2017

  7. [7]

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The C o NLL -2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1--14, Baltimore, Maryland. Association for Computational Linguistics

    work page 2014

  8. [8]

    NLLB Team . 2022. https://arxiv.org/abs/2207.04672 No language left behind: Scaling human-centered machine translation . Preprint, arXiv:2207.04672

    arXiv 2022

Show all 12 references
  1. [9]

    Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235

    2018 arXiv

  2. [10]

    Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023. ChatGPT or grammarly? evaluating ChatGPT on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648

    2023 arXiv

  3. [11]

    Soyoung Yoon, Sungjoon Park, Gyuwan Kim, Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, and Alice Oh. 2023. Towards standardizing korean grammatical error correction: Datasets and annotation. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin...

    2023

  4. [12]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM -as-a-judge with MT -bench and chatbot arena. arXiv preprint arXiv:2306.05685

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.