Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing

Brendan Flanagan; Hiroaki Ogata; Steve Woollaston

REVIEW 3 major objections 5 minor 7 references

Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that grammatical accuracy and native-like idiomaticity are distinct, measurable gaps in learner writing, and that mapping both on the same texts assigns each grammar structure a different instructional priority.

desk verdict Promising two-axis diagnostic for EFL writing, but the idiomatic half rests on a revision layer that visibly violates its own content-preservation constraint. read the letter →

arxiv 2608.09289 v1 pith:6M3YBBNF submitted 2026-08-10 cs.CL

Steve Woollaston , Brendan Flanagan , Hiroaki Ogata This is my paper · ORCID

classification cs.CL

keywords automatedwritingevaluationlearnercorpusEFLgrammaticalaccuracyidiomaticityLLMCEFR-JJapanese

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that grammatical accuracy and native-like idiomaticity are not the same thing in learner writing, and that both can be measured separately on the same texts. The authors take 3,830 diary-style submissions from 120 Japanese junior high school students, produce two LLM-generated variants of each text—a literal grammatical correction and a freely idiomatic revision—and count how often CEFR-J grammar patterns appear in each version. The frequency shifts define an accuracy gap and an idiomatic gap, and the two gaps place every grammar item in one of four instructional quadrants. If the argument holds, teachers can decide from corpus evidence whether a structure is misused, avoided, or overused, instead of treating every deviation as a single undifferentiated error.

What carries the argument

The load-bearing mechanism is the layered LLM-correction pipeline: a raw learner text is first rewritten with a prompt that fixes only spelling, punctuation, and local grammar while explicitly keeping the student's sentence structure, then rewritten again with a prompt that freely naturalises the phrasing. Counting the same grammar patterns in all three layers with the deterministic regex-based CEFR-J grammar extractor turns each prompt into a measurable frequency shift. The log-likelihood statistic $G^2$ decides which shifts are significant, and the log-ratio measure with a smoothing constant of $0.1$ controls for the baseline-frequency trap, so rare structures are not swamped by high-volume defaults.

What would settle it

Ask independent native-speaker raters to revise a random sample of the same 3,830 texts with the same two instructions, and compare the idiomatic gap rankings from human revisions with those from the model; if the ordering of underused and overused structures changes materially, the typology is an artifact of the revision model rather than a property of learner writing.

Watch

Extended reading notes

Core claim

The paper's central claim is that when the same learner text is passed through a literal error-correction layer and then an idiomatic-revision layer, the normalized frequency differences between consecutive layers are diagnostic measures with distinct pedagogical meanings. The accuracy gap, $\mathrm{Freq}_{\mathrm{literal}}-\mathrm{Freq}_{\mathrm{raw}}$, captures structures attempted but produced with errors; the idiomatic gap, $\mathrm{Freq}_{\mathrm{idiomatic}}-\mathrm{Freq}_{\mathrm{literal}}$, captures structures that are grammatically correct but underused or overused relative to native-like revisions. On the accuracy side, definite articles ($+17.27$ per 10,000 words), third-person singular $-s$, and the modals would and could show the largest gains after correction. On the idiomatic side, $-ing$ forms and would are the most underused, while simple present verbs, subject-verb-object patterns, and can are overused. Mapping the top 40 structures by error rate against idiomatic gap produces four quadrants—high-error/more-natural, low-error/avoided, accurate/overused, and high-error/overused—each with its own instructional target.

Load-bearing premise

The argument stands or falls on whether the idiomatic revisions produced by the local open-source language model are a valid stand-in for native-like English usage, since the paper's manual checks covered only a small subset of outputs and no native-speaker corpus anchors the idiomatic gap.

Editorial extensions

If this is right

Articles and third-person singular $-s$ sit in the high-error, more-natural quadrant, so the paper implies they need both form-focused instruction and production practice rather than exposure alone.
Simple present, can, and SVO patterns land in the accurate/overused quadrant, implying instruction should offer lexical and syntactic alternatives instead of more grammar correction.
Low-error, avoided forms such as non-finite verb forms, first-person pronouns, and determiners like some and any require awareness-raising and input, not further form-focused teaching.
Because the pipeline produces a per-text profile cheaply, it could generate individual diagnostic reports rather than only aggregate class-level patterns.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would be to run the same two prompts on argumentative or academic essays: diary prompts likely inflate first-person pronouns, simple present, and can, so the quadrant assignments may shift with genre.
The idiomatic gap should be read as relative to one revision model's style until it is calibrated against native-speaker revisions or a native corpus; the vertical axis is the least externally anchored part of the method.
The framework would be strengthened by per-learner and per-prompt stability checks, since the current analysis aggregates 3,830 texts and does not yet directly measure individual avoidance or overuse patterns.
Applying the same layered-correction procedure to learners from other first-language backgrounds would test whether the overuse and avoidance profiles are transfer-specific or developmental.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a useful proof-of-concept with a load-bearing flaw in the idiomatic half of its diagnostic. Worth engaging, but it needs major revision, not light editing.

What's new and good: the layered pipeline (literal correction vs. idiomatic revision) applied to 3,830 junior-high samples through a deterministic CEFR-J regex extractor is a genuine new application. The two-dimensional typology (error rate vs. idiomatic gap) is a practical way to present feedback options to teachers. The accuracy-gap results largely replicate known Japanese EFL patterns—articles, third-person -s, modals—which is reassuring. Using log-likelihood with log ratio and an open-source model are sensible choices for reproducibility.

Now the soft spots, in proportion. The idiomatic gap is defined as Freq_idiomatic minus Freq_literal, where both layers are outputs of the same LLM. No native-speaker corpus or human ratings anchor it. The paper admits this, which is honest. But the stress-test note points to something worse, and it holds up on reading: the idiomatic revision prompt says "WITHOUT adding or removing content," while Table 1 shows the revision layer is 18% longer and has 31% fewer sentences, and the paper's own text says the LLM "introduced more explicit connectors, elaborated ideas, and added cohesive devices." That directly contradicts the constraint. So the idiomatic gap conflates elaboration with native-like idiomaticity. The big positive shifts—V-ING +42, would +29, subordinate clauses +26—could simply reflect the model writing more and merging sentences, not learner underuse relative to native norms. The vertical axis of the typology is therefore compromised by an internal inconsistency, not just a missing validation.

The accuracy gap (literal vs. raw) is more defensible, though it still depends on the regex extractor recognizing repaired forms. Minor concern. Code and data are absent; the prompt list is "available on request," which limits reproducibility.

The paper is clearly written and does not overstate its readiness—the authors flag manual validation as ad hoc and call for native correction as future work. But the abstract still frames the results as "relative to native norms," which is not supported.

Who is this for? Applied linguists and educational NLP researchers building automated feedback tools. Teachers might find the typology intuitive, but they should not act on the idiomatic axis until it is re-anchored. This deserves a serious referee: the framework is novel and the main flaw is fixable with native validation, a content-preservation check on the revisions, and a revised abstract. Send it to review, but expect major revision.

Referee Report

3 major / 5 minor

Summary. The paper proposes a layered LLM-correction pipeline to separate grammatical accuracy from native-like idiomaticity in Japanese junior high school EFL writing. For 3,830 writing samples from 120 students, the authors generate a literal error-correction layer and an idiomatic native-revision layer, apply the CEFR-J regex grammar extractor, and define an accuracy gap (Freq_literal - Freq_raw) and an idiomatic gap (Freq_idiomatic - Freq_literal). Results are presented as frequency shifts across three layers, leading to a two-dimensional instructional typology that maps error rates against idiomatic gaps to assign distinct pedagogical priorities to grammar structures such as articles, third-person -s, modals, -ing forms, and SVO patterns.

Significance. If the gaps are validly measured, the framework is a useful and scalable diagnostic tool: it gives teachers a transparent, reproducible way to distinguish structures learners attempt but execute incorrectly from structures they avoid or overuse, and the CEFR-J alignment and deterministic regex extractor are strengths for classroom deployment. The paper also makes a genuine contribution by attempting to operationalize the accuracy/idiomaticity distinction at feature level rather than as a global error count. However, the significance is conditional: the idiomatic gap rests entirely on an unvalidated LLM proxy for native norms, and the reported revision layer violates the very constraint that defines what the idiomatic gap is supposed to measure.

major comments (3)

[§3.3, Table 1, Appendix] The idiomatic revision instruction requires the model to rewrite "WITHOUT adding or removing content," yet Table 1 shows the Idiomatic Revision layer contains 182,671 words versus 154,478 in Literal Correction (an 18.3% increase) and 12,998 sentences versus 18,871 (a 31% decrease), raising mean sentence length from 8.19 to 14.05. This substantial expansion and sentence merging indicate that the model added cohesive devices, elaboration, or subordination, so Freq_idiomatic - Freq_literal is not a clean measure of native-like preference for the same content. A large share of the reported idiomatic shifts (e.g., V-ING +42.21, MD.would.AFF +28.67, CL_after.etc +26.20, TA.PRPF.AFF +13.71) may be driven by the model writing more and with more complex syntax rather than by native-like idiomaticity. Since the vertical axis of Figure 4 is built on this difference, the central typology is compromised by prompt non-compliance. This is directly testable by comparing revisions that honor the length constraint against the current output.
[§4.2, §5.1] The idiomatic gap is operationalized entirely through gemma4:31b's revisions, with no native-speaker corpus, human ratings, or native judgments to anchor the "native norms" against which underuse and overuse are defined. The paper itself acknowledges in §5.1 that "manual validation was ad hoc and restricted to a small subset of outputs" and lists "native speaker correction and revisions" as future work. As a result, the finding that -ing forms and hypothetical would are underused while simple present and can are overused is, by construction, a statement about what gemma4 adds or removes during revision, not an independently validated claim about Japanese learners' distance from native writing. If gemma4's revision preferences differ from real native norms (for example, if it over-generates nonrestrictive relatives or present perfect), the idiomatic gap values and the quadrant assignments in Figure 4 are artifacts of model style. A concrete remedy is to collect native-speaker revisions or native-corpus frequency benchmarks on a held-out sample and compare the resulting gap values.
[§3.4, Table 2] The accuracy gap is computed as the difference in regex-extractor frequencies between raw and literal-corrected text. This presumes that gemma4's literal correction layer preserves each learner's underlying syntactic structure while repairing only surface errors, and that the CEFR-J regex extractor recognizes the repaired forms as instances of the intended structures. No reliability evidence is provided for the literal corrections: there is no comparison with expert human corrections on a sample, and the extractor's precision and recall on raw versus corrected text are not reported. For example, the +17.27 Δ for DT.the could overstate article omission if the literal correction inserts articles in contexts where the learner's raw form would not have been tagged at all. The paper should include a validation sample with human-corrected texts and report extractor agreement on both layers to support the accuracy-gap interpretation.

minor comments (5)

[§4.3] The text contains a typo: "TO.to_d o" should be "TO.to_do".
[§4.3] The Error Rate formula is garbled in the typeset text ("Correctedraw" and "Uncorrectedraw" appear without proper subscripts); it should be presented as (Freq_literal - Freq_raw) / Freq_literal × 100 for readability.
[§3.4] The sentence describing log-ratio thresholds ("≥ 1 and ≥ 2 indicate doubling/halving and quadrupling/quartering, respectively") is ambiguous; log ratio 1 indicates doubling and -1 indicates halving, so the thresholds should be described separately for positive and negative values.
[§5.1] The limitation that manual validation was ad hoc appears only in the final Discussion; it would be more transparent to state this limitation in the Methodology section (§3.3) when the LLM processing is introduced.
[Table 1] The column header "Words / Sentence" is ungrammatical; it should be "Words per Sentence."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy and idiomatic gaps are arithmetic differences between independently generated text layers, not predictions fitted back to their own inputs.

full rationale

The derivation chain is descriptive, not circular. Freq_raw, Freq_literal, and Freq_idiomatic are produced by applying a deterministic regex extractor to three text layers, and the two gaps are defined as arithmetic differences (Section 3.4: Freqliteral – Freqraw and Freqidiomatic – Freqliteral). The structures that show large shifts are empirical outputs of that pipeline, not parameters fitted to reproduce the conclusions. The only self-citation (Woollaston et al., 2026, Section 3.2) merely points to a separate study of chatbot interactions and is not load-bearing. The idiomatic gap does rely on gemma4:31b's revisions as a proxy for 'native-like' English, and Section 5.1 candidly notes that manual validation was ad hoc and native-speaker validation is future work; this is a construct-validity limitation, not circularity, because the LLM output is an external instrument that could in principle be checked against human ratings. Likewise, the observed violation of the 'WITHOUT adding or removing content' instruction (Table 1: +18% words, −31% sentences) is an internal-consistency confound that may make the idiomatic gap partly reflect elaboration, but it does not make any equation identical to its own input. The RQ3 typology is a mapping of the computed gaps, not a derivation that presupposes the gaps it reports.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central measurements rest on two unvalidated domain assumptions: that the LLM literal correction preserves learner structure, and that the LLM idiomatic revision equals native norms. The regex extractor's blind spots in learner text are acknowledged. The quadrant boundary is a post-hoc median. No external benchmarks or independent entities are introduced.

free parameters (3)

Log ratio smoothing constant epsilon = 0.1
Chosen ad hoc in the log ratio formula to avoid undefined logs; affects reported log ratios.
Minimum frequency threshold = 5 per 10,000 words per layer
Chosen post hoc to reduce noise from rare items; changes which grammar items are included in the analysis.
Quadrant boundary median error rate = 8.0%
The median error rate of the top 40 items is used to split the typology into quadrants; the boundary is data-derived and post hoc.

assumptions (5)

domain assumption The literal correction prompt is followed by the LLM, preserving the student's structure and only fixing mechanical errors.
No comprehensive human evaluation is provided; the paper admits manual checking was restricted to a small subset (Section 5.1).
domain assumption The idiomatic revision produced by gemma4:31b reflects native-like idiomatic English.
No native-speaker corpus baseline or human judgment is used; the paper lists native validation as future work (Section 5.1).
domain assumption The CEFR-J regex grammar extractor detects grammar structures reliably in all layers, including non-standard learner writing.
The paper acknowledges the extractor may fail on learner errors where lexical mistakes obscure grammar (Section 5.1).
domain assumption Frequency shifts across the three layers are attributable to learner grammar behavior rather than task or genre effects.
All texts are diary-style prompts; the paper notes genre generalization requires argumentative or academic genres (Section 5.1).
standard math Dunning's log-likelihood G2 test is valid for these comparisons.
This is a standard test in corpus linguistics for non-normal, low-frequency data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing." pith.science (2026). https://pith.science/paper/6M3YBBNF

@misc{pith2026260809289,
  author       = {Pith},
  title        = {Pith review of: Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6M3YBBNF}},
  note         = {Machine review of arXiv:2608.09289}
}

read the original abstract

Second language writing research distinguishes grammatical accuracy from native-like idiomaticity, yet automated writing evaluation often conflates these dimensions. This study introduces a layered LLM-correction pipeline that isolates structural errors from unnaturalness by generating literal error corrections and idiomatic revisions for 3,830 English writing samples from 120 Japanese junior high school students. Applying the regex-based CEFR-J grammar extractor, we quantify two diagnostic measures: accuracy gaps (structures attempted but incorrectly produced) and idiomatic gaps (grammatically correct structures underused or overused relative to native norms). Results reveal distinct patterns: definite articles, third-person singular -s, and modals (would, could) exhibit significant accuracy difficulties, while -ing forms and hypothetical modals (would) show the largest idiomatic underuse, with simple present verbs, subject-verb-object patterns, and modal can conversely exhibiting the most pronounced overuse. A two-dimensional instructional typology maps error rates against idiomatic gaps, distinguishing accurate but overused grammar items from error-prone or avoided complex forms requiring targeted production practice. This framework advances pedagogical feedback by enabling teachers to diagnose whether learner difficulties arise from inaccurate execution, structural avoidance, or L1-mapped overreliance, supporting evidence-based interventions tailored to the specific needs of each learner.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 6 canonical work pages

[1]

A text may be error-free yet sound unnatural due to L1 transfer (Granger, 2015; Jarvis & Pavlenko, 2010)

Introduction Second language (L2) writing research distinguishes grammatical accuracy from native-like (L1) idiomaticity. A text may be error-free yet sound unnatural due to L1 transfer (Granger, 2015; Jarvis & Pavlenko, 2010). This distinction is especially pronounced for Japanese learners of English, given the linguistic distance across article systems,...

work page 1989
[2]

It is ramen

Related Work 2.1 CEFR-J Grammar Profile A key framework for diagnosing Japanese EFL writing development is the CEFR-J (Tono, 2013), an adaptation of the Common European Framework of Reference optimised for English language education in Japan. Beyond providing fine-grained 'Can-Do' descriptors, the framework includes the CEFR-J Grammar Profile (Ishii & Ton...

work page 2018
[3]

Think ‑ Pair ‑ Share

Methodology 3.1 Participants and Setting The raw writing samples (3,912 total) were collected from 120 Japanese junior high school students (aged 14-16) across three EFL classes. Over a four ‑ month period (November 2024 – March 2025), learners used a writing support chatbot daily during English lessons (3-4 sessions per week). Each writing session follow...

work page 2014
[4]

Results To enable valid comparisons across the three text versions, we computed the total word count, number of sentences, and words per sentences for each layer (Table 1). The Literal Correction layer was slightly longer than the Uncorrected layer, while the Idiomatic Revision layer expanded substantially as the LLM introduced more explicit connectors, e...

work page
[5]

the house, which I bought last year, is nearby

Discussion This study introduced and validated a framework for disentangling grammatical accuracy from native-like idiomaticity in Japanese EFL writing, revealing that these dimensions represent distinct developmental challenges with divergent pedagogical implications. Our findings demonstrate that treating all deviations from a native standard as homogen...

work page 2018
[6]

Acknowledgements Many thanks to the English teacher and students who participated in this research, which was supported by Council for Science, 3rd SIP JPJ012347, and JSPS Grant-in-Aid for Scientific Research (B) JP23H01001, (A) JP23H00505, and KAKENHI Grant Number 26KJ1488

work page
[7]

Criterial feature

Bibliographical References Bryant, W. H. (1984). Typical errors in English made by Japanese ESL students. JALT Journal , 6 (1), 1–18. Coyne, S., Sakaguchi, K., Galvan-Sosa, D., Zock, M., & Inui, K. (2023). Analyzing the performance of GPT-3.5 and GPT-4 in grammatical error correction. arXiv . https://doi.org/10.48550/arXiv.2303.14342 Davidson, S. S. (2024...

arXiv doi:10.48550/arxiv.2303.14342 1984

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

A text may be error-free yet sound unnatural due to L1 transfer (Granger, 2015; Jarvis & Pavlenko, 2010)

Introduction Second language (L2) writing research distinguishes grammatical accuracy from native-like (L1) idiomaticity. A text may be error-free yet sound unnatural due to L1 transfer (Granger, 2015; Jarvis & Pavlenko, 2010). This distinction is especially pronounced for Japanese learners of English, given the linguistic distance across article systems,...

work page 1989

[2] [2]

It is ramen

Related Work 2.1 CEFR-J Grammar Profile A key framework for diagnosing Japanese EFL writing development is the CEFR-J (Tono, 2013), an adaptation of the Common European Framework of Reference optimised for English language education in Japan. Beyond providing fine-grained 'Can-Do' descriptors, the framework includes the CEFR-J Grammar Profile (Ishii & Ton...

work page 2018

[3] [3]

Think ‑ Pair ‑ Share

Methodology 3.1 Participants and Setting The raw writing samples (3,912 total) were collected from 120 Japanese junior high school students (aged 14-16) across three EFL classes. Over a four ‑ month period (November 2024 – March 2025), learners used a writing support chatbot daily during English lessons (3-4 sessions per week). Each writing session follow...

work page 2014

[4] [4]

Results To enable valid comparisons across the three text versions, we computed the total word count, number of sentences, and words per sentences for each layer (Table 1). The Literal Correction layer was slightly longer than the Uncorrected layer, while the Idiomatic Revision layer expanded substantially as the LLM introduced more explicit connectors, e...

work page

[5] [5]

the house, which I bought last year, is nearby

Discussion This study introduced and validated a framework for disentangling grammatical accuracy from native-like idiomaticity in Japanese EFL writing, revealing that these dimensions represent distinct developmental challenges with divergent pedagogical implications. Our findings demonstrate that treating all deviations from a native standard as homogen...

work page 2018

[6] [6]

Acknowledgements Many thanks to the English teacher and students who participated in this research, which was supported by Council for Science, 3rd SIP JPJ012347, and JSPS Grant-in-Aid for Scientific Research (B) JP23H01001, (A) JP23H00505, and KAKENHI Grant Number 26KJ1488

work page

[7] [7]

Criterial feature

Bibliographical References Bryant, W. H. (1984). Typical errors in English made by Japanese ESL students. JALT Journal , 6 (1), 1–18. Coyne, S., Sakaguchi, K., Galvan-Sosa, D., Zock, M., & Inui, K. (2023). Analyzing the performance of GPT-3.5 and GPT-4 in grammatical error correction. arXiv . https://doi.org/10.48550/arXiv.2303.14342 Davidson, S. S. (2024...

arXiv doi:10.48550/arxiv.2303.14342 1984