REVIEW 7 cited by
Analyzing the Performance of GPT-3.5 and GPT-4 in Grammatical Error Correction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
GPT-3 and GPT-4 models are powerful, achieving high performance on a variety of Natural Language Processing tasks. However, there is a relative lack of detailed published analysis of their performance on the task of grammatical error correction (GEC). To address this, we perform experiments testing the capabilities of a GPT-3.5 model (text-davinci-003) and a GPT-4 model (gpt-4-0314) on major GEC benchmarks. We compare the performance of different prompts in both zero-shot and few-shot settings, analyzing intriguing or problematic outputs encountered with different prompt formats. We report the performance of our best prompt on the BEA-2019 and JFLEG datasets, finding that the GPT models can perform well in a sentence-level revision setting, with GPT-4 achieving a new high score on the JFLEG benchmark. Through human evaluation experiments, we compare the GPT models' corrections to source, human reference, and baseline GEC system sentences and observe differences in editing strategies and how they are scored by human raters.
Forward citations
Cited by 7 Pith papers
-
Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing
Using two AI rewrites of 3,830 student texts, the authors separate grammar errors from unnatural phrasing and map 40 grammar structures into four instructional zones.
-
APIO: Automatic Prompt Induction and Optimization for Grammatical Error Correction and Text Simplification
APIO automatically induces and optimizes instruction-list prompts for grammatical error correction and text simplification, reporting improved scores over prior prompt-based methods on BEA-2019 and ASSET.
-
Adapting LLMs for Minimal-edit Grammatical Error Correction
A lower learning rate on correct examples after training on errors lets Gemma 2 set a new single-model SOTA on BEA-test, aided by adding unedited pairs.
-
Explanation based In-Context Demonstrations Retrieval for Multilingual Grammatical Error Correction
Retrieving in-context demonstrations by matching natural-language grammatical error explanations beats input-text similarity for few-shot multilingual GEC.
-
Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction
Sentence-level GEC metric scores are decomposed into per-edit Shapley attributions, enabling edit-level explanation and error-type analysis.
-
DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models
DSGram is a reference-free GEC evaluation metric that dynamically weights Semantic Coherence, Edit Level, and Fluency using LLM-generated AHP weights, and reports improved correlation with human judgments on the SEEDA...
-
Trigger$^3$: Refining Query Correction via Adaptive Model Selector
Trigger3 uses three trained triggers to route Chinese search queries among a small correction model, an LLM, and the original query, improving F0.5 on two datasets while lowering LLM coverage.
Discussion (0). Continue with ORCID to comment.