Pith. sign in

REVIEW 3 major objections 5 minor 5 references

Shared multi-task training helps recover intended words from L2 speech but hurts recovery of what was actually pronounced, especially in English, because the shared encoder collapses the two tasks into nearly the same representation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 18:21 UTC pith:EJQSYJLL

load-bearing objection Clean empirical pattern on joint MTL for dual-output L2 ASR; the CKA entanglement story is suggestive but confounded by capacity and a single metric. the 3 major comments →

arxiv 2606.06065 v4 pith:EJQSYJLL submitted 2026-06-04 cs.CL cs.SDeess.AS

Multi-task Learning is Not Enough: Representational Entanglement in Dual-output Second Language Speech Recognition

Seung Hwan Cho , Young-Min Kim This is my paper
classification cs.CL cs.SDeess.AS
keywords multi-task learningsecond-language speech recognitiondual-output ASRrepresentational entanglementsurface vs meaning transcriptionCentered Kernel AlignmentL2 pronunciation assessment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language-learning and pronunciation-assessment systems need two transcriptions from one audio clip: the surface form (what the learner actually said) and the meaning form (what they intended). Multi-task learning that shares one encoder for both outputs looks like the natural solution. This paper shows that the natural solution is not enough. On Korean and English L2 data, joint training improves the meaning transcription but degrades the surface transcription. The surface degradation is far larger in English and grows steadily with how different the two transcriptions already are (measured by Levenshtein edit distance). Representation comparisons locate the problem inside the shared encoder: Korean keeps the two tasks separable, while English collapses them into nearly identical representations. The meaning decoder can still invent its own workaround; the surface decoder cannot, because it must stay aligned to the acoustic frames. The authors therefore argue that future multi-task designs must actively prevent this encoder-level entanglement.

Core claim

Joint multi-task learning for dual-output L2 speech recognition produces an asymmetric trade-off: meaning transcription improves while surface transcription degrades. The surface penalty is substantially larger for English than for Korean and scales monotonically with surface-meaning edit distance. Layer-wise similarity analysis attributes the pattern to encoder-level task entanglement that appears in English but not in Korean; the English meaning decoder can bypass the entangled encoder, but the surface decoder cannot.

What carries the argument

Encoder-level representational entanglement, measured by layer-wise Centered Kernel Alignment (CKA) between single-output and dual-output Conformer encoders (and, secondarily, between their decoders). High CKA between independently trained surface and meaning encoders indicates that the two tasks have collapsed into a single representation that the dual-output encoder cannot disentangle.

Load-bearing premise

The claim rests on treating layer-wise CKA similarity as a faithful, non-confounded measure of whether two tasks have become entangled inside the encoder.

What would settle it

Train the same dual-output architecture on English with an explicit disentanglement regularizer (or with a larger, task-specific encoder capacity) and check whether surface CER degradation disappears or no longer scales with edit distance while meaning CER stays improved.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dual-output L2 ASR systems that simply share an encoder will systematically under-perform on surface (pronunciation) transcription relative to separately trained models, especially for English.
  • The severity of that under-performance will be predictable from the edit-distance distribution between surface and meaning labels.
  • Decoder-only adaptation is insufficient; methods that keep encoder representations task-separable (sparse decomposition, adversarial losses, gating) are required.
  • Language identity itself modulates how much entanglement occurs, so cross-lingual transfer of multi-task designs cannot be assumed safe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same entanglement pattern may appear in any multi-task ASR setting where one output is a near-monotonic acoustic sequence and the other is a more abstract rewrite (e.g., ASR + punctuation restoration, ASR + translation).
  • If surface degradation scales with edit distance, curriculum or sample-reweighting strategies that first train on low-divergence utterances could reduce the observed trade-off without architectural change.
  • The finding suggests that ‘shared representation is always better’ is an unsafe default for any dual-output speech task whose targets diverge at the token level.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies joint multi-task learning for dual-output L2 ASR that must produce both surface (verbatim pronunciation) and meaning (canonical written) transcriptions from the same acoustic input. Using Korean and English L2 read-speech corpora, it compares single-output Conformer/Whisper baselines against a dual-output Conformer that shares one encoder and two task-specific Transformer decoders (with an auxiliary CTC head on surface targets). Empirically, joint training improves meaning CER while degrading surface CER; the surface degradation is larger in English and grows monotonically with surface–meaning Levenshtein edit distance (Figure 2). Layer-wise CKA analyses of encoder and decoder activations (Tables 3–4) are offered as evidence that English single-output encoders remain highly similar (encoder-level entanglement), whereas Korean encoders diverge, and that the English meaning decoder can construct a distinct representation while the surface decoder remains constrained by the shared encoder. The authors conclude that decoder-level adaptation alone cannot compensate for encoder entanglement and call for structured MTL methods that mitigate it.

Significance. If the reported CER asymmetry and its scaling with edit distance hold under broader conditions, the work supplies a concrete, language-dependent counter-example to the default assumption that joint MTL is uniformly beneficial for dual-output L2 ASR. The controlled SO-versus-DO comparison, the stratified analysis by surface–meaning divergence, and the explicit localization attempt via CKA are useful contributions for the speech-education and multi-task ASR communities. The paper does not claim a new architecture or a theoretical guarantee; its value lies in the empirical demonstration and the mechanistic hypothesis that motivates future disentanglement techniques (sparse decomposition, adversarial training, gating). The raw performance patterns are cleanly presented and falsifiable; the mechanistic interpretation is more provisional.

major comments (3)
  1. Section 4 / Tables 3–4: The central causal claim that English surface degradation is produced by encoder-level task entanglement rests on layer-wise linear CKA between 32 M SO and 40 M DO Conformer encoders. Two confounds remain unaddressed. First, residual capacity: the DO encoder is trained with two decoders and a larger parameter budget, so elevated English SSO–MSO similarity could reflect under-capacity of the independent SO models rather than language-intrinsic entanglement. Second, only a single similarity metric (linear CKA) is reported; RSA, PWCCA or SVCCA, or an equal-capacity ablation (matched encoder size or frozen encoder), are absent. Without these controls the mechanistic story is under-determined even though the CER and ED-stratified patterns themselves are clear.
  2. Section 3.2 / Table 2: Only a single dual-output architecture (Conformer encoder + two Transformer decoders) is evaluated. Whisper-base/small appear only as single-output baselines. Consequently it is impossible to know whether the surface degradation and the CKA pattern generalize beyond this particular encoder–decoder pairing or are artifacts of the Conformer’s inductive bias. At least one additional DO architecture (or a DO Whisper variant) would be needed to support the language-level claim.
  3. Section 2.2 / Eq. (2): The dual-output loss weights (α, β, γ) = (0.2, 0.5, 0.3) are fixed after “pre-experiments on the validation set.” No sensitivity analysis or alternative weightings are reported. Because surface degradation is the key negative result, it is important to show that the English surface penalty is not an artifact of the particular surface-heavy weighting chosen for the DO objective.
minor comments (5)
  1. Table 1: Surface–meaning edit distance is computed at character/syllable level for Korean and word level for English. A short justification of this choice (and a check that the stratified pattern is robust to a common tokenization) would strengthen the cross-lingual comparison.
  2. Figure 2: Error bars or confidence intervals on the CER gaps would help the reader judge whether the monotonic English trend is statistically reliable, especially in the sparsely populated ED ≥ 11 bin.
  3. Section 4.1: The notation “SSO ↔ MSO”, “SDO”, “MDO” is introduced only in the table caption; a brief definition in the main text would improve readability.
  4. References: The discussion of joint CTC-attention and dual-decoder ASR/ST models is appropriate; a pointer to recent work on multi-task representation disentanglement (e.g., gradient surgery or task-specific adapters) would better situate the proposed future directions.
  5. Abstract and Conclusion: The phrase “nearly identical ones” for English encoder representations is slightly stronger than the CKA numbers (final-layer SSO–MSO = 0.40); a more measured wording would be preferable.

Circularity Check

0 steps flagged

No significant circularity: empirical CER/CKA measurements on held-out data with no definitional reduction or load-bearing self-citation.

full rationale

The paper is a controlled empirical comparison of single-output vs dual-output Conformer models for dual-output L2 ASR. Performance claims rest on Character Error Rate measured on held-out test partitions of the AI-Hub Korean and English corpora (Table 2, Figure 2 stratified by Levenshtein edit distance). Representational claims rest on layer-wise CKA computed between independently trained SO and jointly trained DO encoders/decoders (Tables 3-4). Loss weights (α,β,γ) are fixed once on validation and never re-used as predictions. No equation equates a claimed result to an input by construction; no uniqueness theorem or ansatz is imported from the authors' prior work; references are standard external literature (Ruder, Kim et al., Kornblith et al., etc.). The entanglement interpretation is post-hoc but does not redefine the measured quantities. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 1 invented entities

The paper is an empirical ML study. Its load-bearing premises are standard architectural and evaluation choices plus one interpretive leap (CKA similarity = task entanglement). No new physical entities or free parameters are fitted to produce the central claim; the loss weights are fixed once on validation and do not define the language asymmetry.

free parameters (2)
  • dual-output loss weights (α, β, γ) = (0.2, 0.5, 0.3)
    Fixed to (0.2, 0.5, 0.3) via pre-experiments on the validation set (Eq. 2). Different weights could change the surface/meaning trade-off magnitude.
  • single-output CTC-attention mix α
    Appears in Eq. 1; value not numerically stated beyond the dual-output setting, but is a free hyper-parameter of the baseline.
axioms (4)
  • domain assumption Shared-encoder dual-decoder architecture with surface-only auxiliary CTC is a fair test of joint MTL for dual-output L2 ASR.
    Section 2.2 and Figure 1; the only architectural difference claimed between SO and DO is the second decoder, yet CTC supervision remains surface-only.
  • ad hoc to paper Layer-wise linear CKA between encoder/decoder activations measures task-specific representational entanglement.
    Section 4; CKA is used as the sole similarity metric to support the mechanistic claim.
  • domain assumption Levenshtein edit distance (character syllables for Korean, word tokens for English) is a valid proxy for surface-meaning divergence.
    Table 1 and Section 3.3; stratification of CER gaps rests on this measure.
  • domain assumption Character Error Rate is the appropriate primary metric for both surface and meaning transcription.
    Section 3.1; used throughout Tables 2 and Figure 2.
invented entities (1)
  • encoder-level task entanglement no independent evidence
    purpose: Interpretive label for the observation that English SO surface and meaning encoders remain highly similar by CKA while Korean SO encoders diverge.
    Introduced in Section 4.1; no independent external measurement or formal definition beyond the CKA tables in this paper.

pith-pipeline@v1.1.0-grok45 · 11534 in / 3226 out tokens · 28250 ms · 2026-07-14T18:21:37.329492+00:00 · methodology

0 comments
read the original abstract

Second-language (L2) speech recognition often requires transcriptions of pronunciations and intended meanings. Multi-task learning (MTL) is a natural approach because it assumes that shared representations benefit both outputs. However, this paper shows that this assumption does not hold across Korean and English. MTL improves meaning but degrades surface transcription, especially in English, where the degradation scales with surface-meaning divergence measured by Levenshtein edit distance. Encoder analysis links these patterns to encoder-level entanglement, with Korean preserving disentangled representations while English produces nearly identical ones. Cross-output decoder analysis shows that the meaning dual-output decoder adapts with a unique representation, while the surface dual-output decoder remains constrained by the encoder. These findings motivate the design of MTL frameworks that mitigate encoder-level entanglement to reduce surface degradation in dual-output L2 automatic speech recognition.

Figures

Figures reproduced from arXiv: 2606.06065 by Seung Hwan Cho, Young-Min Kim.

Figure 1
Figure 1. Figure 1: (a) Single-output configuration trains separate models for surface-level (left) and meaning-oriented (right) transcription, each with its own encoder, decoder, and auxiliary CTC head. (b) Dual-output configuration shares a single encoder with two separate Transformer decoders that jointly produce both outputs, along with an auxiliary CTC head on the encoder output. the intended written form. Each token dra… view at source ↗
Figure 2
Figure 2. Figure 2: shows the stratified results and reveals two patterns. In Korean, MTL effects are minor and inconsistent with divergence. The surface gap ranges from +0.19 to +1.03 in the lower ranges and becomes slightly negative at ED>10 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 2 canonical work pages

  1. [1]

    Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y ., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y ., and Pang, R

    doi: 10.1016/j.specom.2009.04.005. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y ., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y ., and Pang, R. Con- former: Convolution-augmented transformer for speech recognition. InProc. Interspeech, pp. 5036–5040, 2020. doi: 10.21437/Interspeech.2020-3015. URL https: //www.isca-archive.org/interspeech_ 2020/gu...

    work page doi:10.1016/j.specom.2009.04.005 2009

  2. [2]

    difficult

    PMlR, 2019. Le, H., Pino, J., Wang, C., Gu, J., Schwab, D., and Be- sacier, L. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. InProceedings of the 28th International Conference on Computational Linguistics, pp. 3520–3533, 2020. Munro, M. J. On the difficulty of defining “difficult” in second-language v...

    Pith/arXiv arXiv doi:10.3389/fcomm.2021 2019

  3. [3]

    doi: 10.21437/ interspeech.2019-2680

    ISCA, September 2019. doi: 10.21437/ interspeech.2019-2680. URL http://dx.doi.org/ 10.21437/Interspeech.2019-2680. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision, 2022. URL https://arxiv. org/abs/2212.04356. Ruder, S. An overview of multi-task learning in deep neur...

    Pith/arXiv arXiv doi:10.21437/interspeech.2019-2680 2019

  4. [4]

    doi: https://doi.org/10.1016/j.csl.2003.09

    work page doi:10.1016/j.csl.2003.09 2003

  5. [5]

    Shakeel, M., Sudo, Y ., Peng, Y ., Lin, C.-J., and Watanabe, S

    URL https://www.sciencedirect.com/ science/article/pii/S0885230803000548. Shakeel, M., Sudo, Y ., Peng, Y ., Lin, C.-J., and Watanabe, S. Unifying diarization, separation, and asr with multi- speaker encoder.arXiv preprint arXiv:2508.20474, 2025. Watanabe, S., Hori, T., Kim, S., Hershey, J. R., and Hayashi, T. Hybrid ctc/attention architecture for end-to-...

    Pith/arXiv arXiv 2025