pith. sign in

arxiv: 2605.20043 · v1 · pith:3B4A4XQVnew · submitted 2026-05-19 · 💻 cs.CL

Mind Your Moras: Orthography-Aware Error Analysis of Neural Japanese Morphological Generation

Wen Zhang This is my paper

Pith reviewed 2026-05-20 05:47 UTC · model grok-4.3

classification 💻 cs.CL
keywords Japanese morphologyorthography-aware analysiserror taxonomygemination errorsneural sequence-to-sequencepast-tense inflectionhiragana representationmodel generalization
0
0 comments X

The pith

Neural models for Japanese past-tense verbs make systematic gemination errors driven by hiragana orthography.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes why high-accuracy neural models still fail on Japanese past-tense morphological inflection in consistent ways. It treats hiragana not as neutral letters but as a system that encodes morphophonological distinctions affecting generalization. Analysis of character-level sequence-to-sequence models on SIGMORPHON datasets shows errors clustering around orthographic features, with gemination accounting for 75-80 percent of residual mistakes, especially in e-stem verbs before the past suffix. These patterns hold steady across architectures and random seeds. A seven-mode error taxonomy makes the failures linguistically interpretable rather than random noise.

Core claim

Despite high aggregate accuracy, models exhibit systematic, linguistically interpretable errors that cluster around specific orthographic properties of hiragana. The work introduces a concise error taxonomy capturing seven primary failure modes. Gemination-related errors dominate residual failures, accounting for 75-80% of errors, particularly in verbs whose stems end in the vowel e and require gemination before the past-tense suffix. Error patterns remain highly consistent across architectures and random seeds, suggesting a robust interaction between orthographic representation, morphological structure, and data frequency effects in shaping model generalization.

What carries the argument

Orthography-aware error taxonomy of seven failure modes that isolates how hiragana encodes morphophonological distinctions, especially gemination in e-stem verbs.

If this is right

  • Gemination errors dominate residual failures at 75-80 percent for e-stem verbs requiring consonant doubling before the past-tense suffix.
  • Error patterns stay consistent across different sequence-to-sequence architectures and random seeds.
  • Orthographic representation interacts with morphological structure and data frequency to shape generalization.
  • A seven-mode taxonomy captures the main linguistically interpretable failure types in Japanese inflection.
  • Orthography-aware evaluation is required to diagnose generalization limits in morphologically complex languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Balancing training frequencies for e-stem verbs could reduce the dominant error class without changing model architecture.
  • The same orthography-driven clustering may appear in other languages that use scripts encoding moraic or geminate distinctions.
  • Explicit rules for mora boundaries or gemination could be added to character-level models to test whether the interaction is causal.
  • Switching to a non-hiragana input script offers a direct way to measure how much of the residual error traces to the writing system itself.

Load-bearing premise

The observed error clusters are primarily driven by orthographic properties of hiragana rather than unmeasured interactions with data frequency or model capacity.

What would settle it

A test in which gemination error rates fall below 50 percent after balancing the frequency of e-stem verbs in training data or after switching inputs to romanized form instead of hiragana.

read the original abstract

We present an orthography-aware error analysis of Japanese past-tense morphological inflection, treating hiragana not merely as a transcriptional medium, but as a representational system encoding morphophonological distinctions that may influence model generalization. We evaluate two character-level sequence-to-sequence architectures on past-tense formation using datasets formatted according to the SIGMORPHON 2020 and 2023 shared task conventions. Despite high aggregate accuracy, models exhibit systematic, linguistically interpretable errors that cluster around specific orthographic properties of hiragana. We introduce a concise error taxonomy capturing seven primary failure modes and provide both quantitative and qualitative analyses. Gemination-related errors dominate residual failures, accounting for 75-80% of errors, particularly in verbs whose stems end in the vowel e and require gemination before the past-tense suffix. Error patterns remain highly consistent across architectures and random seeds, suggesting a robust interaction between orthographic representation, morphological structure, and data frequency effects in shaping model generalization. These results underscore the necessity of orthography-aware evaluation for understanding neural generalization in morphologically complex languages.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents an orthography-aware error analysis of character-level seq2seq models for Japanese past-tense morphological inflection on SIGMORPHON 2020/2023 datasets. It introduces a seven-category error taxonomy and reports that gemination errors dominate residual failures (75-80%), especially for e-stem verbs before the past-tense suffix, with patterns consistent across architectures and random seeds. The work interprets these as evidence of a robust interaction between hiragana orthography, morphological structure, and data frequency effects.

Significance. If the quantitative dominance and cross-model consistency are confirmed with proper controls, the paper offers a useful case study on how orthographic representations shape neural generalization in morphologically complex languages. It could guide orthography-sensitive evaluation practices and targeted data augmentation for Japanese and similar languages, moving beyond aggregate accuracy to linguistically interpretable failure modes.

major comments (2)
  1. [Abstract and Results] Abstract and quantitative results section: The central claim that gemination-related errors account for 75-80% of residual failures (particularly e-stem verbs) is load-bearing for the orthography-morphology interaction thesis, yet the manuscript provides no details on total error counts, test-set size, exact annotation protocol for the taxonomy, or statistical tests supporting the percentage and consistency across seeds.
  2. [Discussion] Discussion or interpretation section: The attribution of error clusters primarily to representational properties of hiragana orthography (rather than unmeasured data frequency or capacity effects) lacks isolation. No frequency-binned ablations, stem-frequency matching, or capacity-controlled comparisons are described, despite the abstract acknowledging 'data frequency effects'; this leaves the causal interpretation vulnerable to the alternative that sparsity in geminated e-stem forms drives the observed patterns.
minor comments (1)
  1. [Error taxonomy] The error taxonomy is introduced concisely but would benefit from an explicit table or figure showing example inputs/outputs for each of the seven failure modes to improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript. The comments identify key areas where additional transparency and controls would strengthen the presentation of our orthography-aware error analysis. We respond to each major comment below and describe the revisions we will implement.

read point-by-point responses

  1. Referee: [Abstract and Results] The central claim that gemination-related errors account for 75-80% of residual failures (particularly e-stem verbs) is load-bearing for the orthography-morphology interaction thesis, yet the manuscript provides no details on total error counts, test-set size, exact annotation protocol for the taxonomy, or statistical tests supporting the percentage and consistency across seeds.

    Authors: We agree that these supporting details are necessary to allow readers to fully assess the quantitative results. In the revised manuscript we will add: (i) the exact sizes of the SIGMORPHON 2020 and 2023 test sets, (ii) absolute error counts per category rather than only percentages, (iii) a precise description of the annotation protocol (including decision criteria for classifying gemination failures and identifying e-stem verbs), and (iv) statistical support such as standard deviations across the five random seeds and a simple test of proportion consistency. These additions will be placed in a new subsection of the results and will not alter the reported 75-80% figure. revision: yes

  2. Referee: [Discussion] The attribution of error clusters primarily to representational properties of hiragana orthography (rather than unmeasured data frequency or capacity effects) lacks isolation. No frequency-binned ablations, stem-frequency matching, or capacity-controlled comparisons are described, despite the abstract acknowledging 'data frequency effects'; this leaves the causal interpretation vulnerable to the alternative that sparsity in geminated e-stem forms drives the observed patterns.

    Authors: We acknowledge that the current version does not contain explicit frequency-binned ablations or stem-frequency matching, leaving room for the alternative explanation the referee raises. The cross-architecture and cross-seed consistency we report provides evidence that the pattern is not an artifact of any single model's capacity, but we agree this does not fully isolate orthographic from frequency contributions. In revision we will add a frequency-stratified analysis: verbs will be binned by corpus frequency, error rates for gemination failures will be compared within frequency bands, and we will report whether the e-stem gemination bias persists at matched frequencies. This new analysis will be presented alongside the existing discussion of orthography-morphology interaction. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical error analysis with post-hoc taxonomy

full rationale

The paper reports experimental results on seq2seq models for Japanese past-tense inflection using SIGMORPHON data. It defines a seven-mode error taxonomy from observed failures, quantifies gemination errors at 75-80%, and notes consistency across architectures and seeds. No derivations, equations, fitted parameters, or predictions appear; the central observations are direct measurements from held-out test sets. The abstract's mention of orthography-morphology-data frequency interaction is an interpretive summary of empirical patterns, not a self-referential claim or reduction to inputs. No self-citations, ansatzes, or uniqueness theorems are invoked as load-bearing steps. This is a standard observational study whose claims rest on external experimental outcomes rather than internal construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical error analysis paper; no mathematical axioms, free parameters, or invented physical entities. The seven-mode error taxonomy is a post-hoc categorization tool rather than a postulated entity with independent evidence.

pith-pipeline@v0.9.0 · 5710 in / 1010 out tokens · 32843 ms · 2026-05-20T05:47:32.547593+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages

  1. [1]

    Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018)

    work page 2018

  2. [2]

    Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared task—morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10--22. Association for Computational Linguistics

    work page 2016

  3. [3]

    Daniels and William Bright

    Peter T. Daniels and William Bright. 1996. The World’s Writing Systems. Oxford University Press, Oxford

    work page 1996

  4. [4]

    Omer Goldman, Khuyagbaatar Batsuren, Salam Khalifa, Aryaman Arora, Garrett Nicolai, Reut Tsarfaty, and Ekaterina Vylomova. 2023. SIGMORPHON – UniMorph 2023 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 20th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 117--125, Toronto,...

    work page 2023

  5. [5]

    Haruo Kubozono, Junko Ito, and Armin Mester. 2008. Consonant gemination in J apanese loanword phonology. In Proceedings of the 18th International Congress of Linguistics (Seoul), pages 953--973

    work page 2008

  6. [6]

    Laurence Labrune. 2012. The Phonology of Japanese. Oxford University Press

    work page 2012

  7. [7]

    Peter Makarov and Simon Clematide. 2018. Imitation learning for neural morphological string transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2877--2882, Brussels, Belgium. Association for Computational Linguistics

    work page 2018

  8. [8]

    Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609--4622, Online. Association for Computational Linguistics

    work page 2020

  9. [9]

    Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM -networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338--348, Copenhagen, Denmark. Association for Computational Linguistics

    work page 2017

  10. [10]

    Richard Sproat. 2000. A Computational Theory of Writing Systems. Cambridge University Press

    work page 2000

  11. [11]

    Timothy J. Vance. 1989. An Introduction to J apanese Phonology . State University of New York Press

    work page 1989

  12. [12]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), pages 5998--6008. Curran Associates, Inc

    work page 2017

  13. [13]

    Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tiago Pimentel, Lucas Torroba Hennigen, Christo Kirov, Garre...

    work page 2020

  14. [14]

    Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1530--1537, Florence, Italy. Association for Computational Linguistics

    work page 2019

  15. [15]

    Wen Zhang. 2023. Pronunciation ambiguities in J apanese K anji. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 50--60, Toronto, Canada. Association for Computational Linguistics

    work page 2023

  16. [16]

    and Kaiser,

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser,. Attention is all you need , booktitle =

    work page

  17. [17]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =

    Wu, Shijie and Cotterell, Ryan , title =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages =

    work page

  18. [18]

    Proceedings of the 20th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , pages =

    Goldman, Omer and Batsuren, Khuyagbaatar and Khalifa, Salam and Arora, Aryaman and Nicolai, Garrett and Tsarfaty, Reut and Vylomova, Ekaterina , title =. Proceedings of the 20th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , pages =

    work page

  19. [19]

    Vylomova, Ekaterina and White, Jennifer and Salesky, Elizabeth and Mielke, Sabrina J. and Wu, Shijie and Ponti, Edoardo Maria and Maudslay, Rowan Hall and Zmigrod, Ran and Valvoda, Josef and Toldova, Svetlana and Tyers, Francis and Klyachko, Elena and Yegorov, Ilya and Krizhanovsky, Natalia and Czarnowska, Paula and Nikkarinen, Irene and Krizhanovsky, And...

    work page

  20. [20]

    Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , pages =

    Cotterell, Ryan and Kirov, Christo and Sylak-Glassman, John and Yarowsky, David and Eisner, Jason and Hulden, Mans , title =. Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , pages =

    work page

  21. [21]

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages =

    Pimentel, Tiago and Valvoda, Josef and Maudslay, Rowan Hall and Zmigrod, Ran and Williams, Adina and Cotterell, Ryan , title =. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages =

    work page

  22. [22]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =

    Makarov, Peter and Clematide, Simon , title =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =

    work page 2018

  23. [23]

    Proceedings of the 6th International Conference on Learning Representations (ICLR 2018) , year =

    Belinkov, Yonatan and Bisk, Yonatan , title =. Proceedings of the 6th International Conference on Learning Representations (ICLR 2018) , year =

    work page 2018

  24. [24]

    Sproat, Richard , title =

    work page

  25. [25]

    and Bright, William , title =

    Daniels, Peter T. and Bright, William , title =

    work page

  26. [26]

    , title =

    Vance, Timothy J. , title =

    work page

  27. [27]

    Labrune, Laurence , title =

    work page

  28. [28]

    Proceedings of the 18th International Congress of Linguistics (Seoul) , pages =

    Kubozono, Haruo and Ito, Junko and Mester, Armin , title =. Proceedings of the 18th International Congress of Linguistics (Seoul) , pages =

    work page

  29. [29]

    2011 , series =

    Catalan Speecon database , publisher =. 2011 , series =

    work page 2011

  30. [30]

    2004 , islrn =

    The EMILLE/CIIL Corpus , publisher =. 2004 , islrn =

    work page 2004

  31. [31]

    2004 , islrn =

    The OrienTel Moroccan MCA (Modern Colloquial Arabic) database , publisher =. 2004 , islrn =

    work page 2004

  32. [32]

    Roventini, Adriana and Marinelli, Rita and Bertagna, Francesca , pid =

    work page

  33. [33]

    Proceedings of the Workshop on Computation and Written Language (CAWL 2023) , year =

    Zhang, Wen , title =. Proceedings of the Workshop on Computation and Written Language (CAWL 2023) , year =

    work page 2023

  34. [34]

    Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages =

    Reimers, Nils and Gurevych, Iryna , title =. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages =

    work page 2017