REVIEW 3 major objections 4 minor 29 references
The paper claims that a decoupled agentic parser, paired with an ambiguity-aware metric, substantially improves dependency parsing of spoken code-switching without retraining.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 03:56 UTC pith:FK24VCWT
load-bearing objection The resource is real but the evaluation is incoherent: the paper's headline numbers are contradicted by its own tables, so it needs major revision before it can be trusted. the 3 major comments →
Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that parsing spoken code-switching improves decisively when the problem is decomposed: a spoken-phenomena handler flags disfluencies, repetitions, and discourse markers; a language resolver normalizes contractions and multiword expressions; a core UD assigner builds the tree under those constraints; and a verifier enforces single-root acyclicity. On the authors' SpokeBench, DECAP reaches the highest Flex-UD score (76.2) versus 70.7 for an LLM pipeline and 30.4 for a traditional bilingual parser, with large reductions in catastrophic structural errors. The paper further claims that standard LAS/UAS metrics compress these gains, and that Flex-UD's graded penalties reveal t
What carries the argument
The load-bearing machinery is the decoupled agent pipeline (SPH → LSR → Core → V/R) combined with Flex-UD's severity multiplier. The four agents turn spoken-language annotation rules into explicit constraints: reparandum relations for repetitions, INTJ/discourse for fillers, dotted MWE nodes, contraction splits with integer-shifted IDs. Flex-UD aggregates five component scores (split, ID, UPOS, HEAD, DEPREL) and multiplies by (1 − P), where P penalizes catastrophic errors such as missing dotted MWEs or invalid heads. This design is what lets the paper claim interpretability and no-retraining extensibility.
Load-bearing premise
That the authors' reformulated UD guidelines and hand-tuned severity weights define what a correct parse is, so the benchmark, metric, and parser prompts all encode the same standard rather than an independent ground truth.
What would settle it
If an independent set of trained annotators following conventional UD guidelines re-annotated the same 126 utterances, and DECAP's parses matched the authors' gold far better than the independent gold, the reported superiority would be an artifact of the annotation standard. Also, if DECAP underperformed a simple fine-tuned parser on a larger spoken code-switching corpus measured by human preference, the no-retraining advantage would be questioned.
If this is right
- If DECAP generalizes, spoken-language dependency parsing for code-switching can be improved purely via prompt/agent design, without fine-tuning on expensive spoken corpora.
- Flex-UD provides a template for evaluating parsers on other ambiguous or non-canonical data, where single-gold matching underestimates performance.
- The taxonomy of nine phenomena offers a checklist for future spoken treebanks and parser stress tests.
- The reported gains imply that standard UD metrics may be systematically under-reporting progress on conversational speech, affecting how leaderboards are read.
Where Pith is reading between the lines
- A closed-loop risk is left implicit: because DECAP's prompts encode the same annotation rules that defined SpokeBench's gold, the comparison may reflect rule adherence rather than broad parsing competence; an independent gold standard would test this.
- The benchmark's small size means the claimed difficulty hierarchies (e.g., repetition easiest under UPOS-LAS, ellipsis hardest under LAS) would be more convincing if replicated on a larger or multi-pair corpus.
- Flex-UD's severity weights are hand-set; an empirical calibration against human acceptability judgments would make the metric portable to other languages and domains.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that standard Universal Dependencies (UD) parsing and evaluation are ill-suited to spoken code-switched conversation. It introduces a taxonomy of ten spoken-language phenomena; SpokeBench, an expert-annotated benchmark derived from the Miami Corpus; Flex-UD, a weighted ambiguity-aware evaluation metric; and DECAP, a four-stage LLM-based agentic parser. The authors claim that DECAP substantially improves LAS, UPOS-LAS, and Flex-UD over traditional and LLM baselines, with up to 52.6% improvement and a top Flex-UD score of 76.2. I read the full text, including the appendices, tables, and prompts.
Significance. The resource and system-design ideas are timely. The expert annotation and dispute-resolution protocol, the plan to release data and code, the deterministic agent interaction, and the separation of spoken-phenomena handling from core parsing are genuine strengths. If the claims were reliable, Flex-UD would be a useful diagnostic for spoken treebanks, and DECAP would demonstrate a no-retraining route to spoken code-switching parsing. However, the manuscript's empirical foundation is compromised by internal contradictions between the prose and the tables, and by the close coupling among the gold standard, the metric, and the system prompts. The headline results therefore cannot be accepted without substantial additional evidence and likely re-analysis.
major comments (3)
- [§5.2 and Tables 1–2] The headline quantitative claims are contradicted by the paper's own tables. §5.2 states that DECAP improves overall LAS from 0.31 (BiLingua) to 0.48 and UPOS-LAS from 0.70 to 0.87, and that DECAP achieves the highest overall FLEX-UD score (76.2) versus 70.7 for BiLingua and 30.4 for the traditional parser. Table 1 Overall shows DECAP LAS 0.26, below BiLingua's 0.32; DECAP U-LAS is 0.70, not 0.87; and no row in Table 1 contains a DECAP LAS of 0.48. Table 2 Overall shows DECAP's Final FLEX-UD as 66.6, below BiLingua's 72.2. This is not a rounding issue; the reported rankings are reversed by the tables. Because these numbers are the empirical basis for the abstract and conclusion, the central claim is unsupported as presented.
- [§3.2, Appendix C, Appendix E] The evaluation is coupled to the system prompt. SpokeBench was annotated with a 'reformulated version of UD guidelines adapted for spoken, disfluent bilingual data' (§3.2, Appendix C). The Core UD Assigner prompt in Appendix E instructs the model to follow 'Miami Gold Subset spoken-language rules' and to use 'Miami-approved lists' for UPOS and DEPREL. In effect, DECAP is prompted with the annotation conventions used to create the gold benchmark and to define Flex-UD's tolerance classes. The FLEX-UD gains may therefore measure adherence to the authors' own guidelines rather than general parsing ability. This concern should be addressed by evaluating on an independently created gold standard or on existing spoken-UD treebanks, and by reporting standard LAS/UAS on such data.
- [§4.2] Flex-UD is not specified sufficiently for reproduction. The formal definition leaves the component scores s_i, the weights w_i, and the severity penalty P to an appendix reference ('see App. B'), but Appendix B contains only illustrative examples of spoken phenomena, not the penalty formula or the values of w_i. The text gives per-issue penalty ranges (P=0.25–0.6 for catastrophic errors, 0.01–0.05 for minor errors) without a clipping rule or aggregation procedure. Without these details, the aggregated FLEX-UD scores in Table 2 cannot be independently computed, and the claimed 'ambiguity-aware' behavior is not checkable.
minor comments (4)
- [Table 3] The category counts sum to 127 sentences (10+15+10+10+10+15+10+15+12+20), but the text in §3.2 and §5.1 says 126. Please correct the arithmetic or the reported number.
- [Figures 3 and 4] The captions say 'DECAP ... performs the best across all categories,' but Table 1 shows several categories where BiLingua exceeds DECAP in LAS (e.g., Discourse and None). The captions/figures should be reconciled with the table values.
- [Abstract] The abstract claims 'over 60% improvements in UPOS-F1 Score over baselines,' but the experiments report UPOS-LAS (U-LAS) and not UPOS-F1. The metric name and the reported numbers should be aligned.
- [Overall presentation] There are numerous formatting inconsistencies, including irregular spacing in 'FLEX-UD' and 'DECAP' in the abstract and running text, and inconsistent category labels (e.g., 'Ellipsis' vs. 'Ellipses' in Table 2). These should be cleaned up.
Circularity Check
Evaluation loop is closed: DECAP's prompts import the same spoken-language annotation rules used to build SpokeBench, and the paper's own tables contradict its headline gains.
specific steps
-
self definitional
[Appendix E (Core UD Assigner prompt); §3.2; Appendix C]
"Your task is to assign UD-style annotations (UPOS, HEAD_ID, DEPREL) following the Miami Gold Subset spoken-language rules. | Annotations in SpokeBench follow UD conventions with targeted extensions for spoken, disfluent, and code-switched data."
SpokeBench's gold annotations are produced under the same reformulated UD rules that DECAP's Core agent is explicitly prompted to follow ('Miami Gold Subset spoken-language rules'). The prompt's label mappings (reparandum→rep, discourse→INTJ/discourse, dep for unresolved ellipsis) mirror the gold-annotation rules in Appendix C. Evaluating DECAP against this gold therefore measures prompt compliance with the authors' own annotation conventions; the comparison systems are not given these rules. Part of DECAP's claimed improvement is an artifact of sharing the answer standard, rather than an independent test of parsing quality.
full rationale
SpokeBench is expert-annotated and human adjudication provides some external linguistic grounding, so the benchmark is not vacuous. However, the specific evaluation loop is substantially closed: the gold annotations were created with the authors' reformulated UD rules, DECAP's Core prompt explicitly instructs the model to follow the same 'Miami Gold Subset spoken-language rules', and FLEX-UD's severity weights are hand-set by the same team. Consequently, DECAP's high reported scores partly measure instruction-following against the authors' own standard rather than an independent parsing capability. The comparison is uneven because BiLingua and Stanza baselines are not given the gold rules. Separately, the abstract and §5.2 headline numbers (FLEX-UD 76.2, LAS 0.48) are contradicted by the paper's own Table 1 (DECAP Overall LAS 0.26) and Table 2 (DECAP Overall Final 66.6, below BiLingua's 72.2); I treat that as a correctness and verification flaw rather than circularity. Self-citations such as Kellert et al. (2025) are used for baselines and framing but are not load-bearing for the circularity finding. Overall, the central 'improvement' claim is partially circular because the benchmark, metric, and system prompt share the same authored conventions.
Axiom & Free-Parameter Ledger
free parameters (4)
- FLEX-UD component weights w_Split, w_ID, w_UPOS, w_HEAD, w_DEPREL
- FLEX-UD severity penalty P and per-issue penalties =
catastrophic 0.25–0.6; minor 0.01–0.05
- FLEX-UD tolerance classes (VERB↔AUX, obj↔obl, etc.)
- Single-root priority ordering in DECAP Verifier =
finite VERB > AUX > NOUN/PRON > communicative token
axioms (5)
- domain assumption Gold annotations in SpokeBench are correct and valid ground truth for spoken CSW parse quality
- ad hoc to paper Single-root UD constraint is appropriate for fragmentary and elliptical spoken utterances
- ad hoc to paper FLEX-UD severity weighting maps monotonically to linguistic quality
- domain assumption GPT-4.1 with temperature=0 reliably follows the multi-agent DECAP protocol
- domain assumption Miami Corpus sample is representative of spoken English-Spanish CSW challenges
Cite this review
Pith. "Pith review of Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions." pith.science (2026). https://pith.science/paper/FK24VCWT
@misc{pith2026260206307,
author = {Pith},
title = {Pith review of: Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions},
year = {2026},
howpublished = {\url{https://pith.science/paper/FK24VCWT}},
note = {Machine review of arXiv:2602.06307}
}
read the original abstract
Spoken bilingual conversations pose substantial challenges for syntactic parsing because they often include disfluencies and discourse-driven structures that complicate dependency parsing under standard Universal Dependencies (UD) assumptions and evaluation practices. To systematically study these challenges, in this work, we first introduce a linguistically grounded taxonomy of conversational bilingual phenomena, together with SpokeBench, an expert-annotated English-Spanish benchmark for structurally complex speech. To address the limitations of existing evaluation practices, we propose Flex-UD, an ambiguity-aware evaluation metric that distinguishes catastrophic structural failures from linguistically acceptable variations. Finally, we introduce DECAP, a decoupled agentic parsing framework that separates spoken-phenomena handling from core syntactic analysis, enabling robust and interpretable dependency parsing without retraining. Experiments across both proprietary and open-weight LLMs show that DECAP substantially improves performance on complex conversational phenomena and achieves over 60% improvements in UPOS-F1 Score over baselines, while Flex-UD evaluations reveal gains that otherwise remain partially hidden under standard attachment-based metrics.
Figures
Reference graph
Works this paper leans on
-
[1]
Xuefeng Bai, Jialong Wu, Yulong Chen, Zhongqing Wang, Kehai Chen, Min Zhang, and Yue Zhang. 2025. Constituency parsing using llms. IEEE Transactions on Audio, Speech and Language Processing
2025
-
[2]
Damir Cavar and 1 others. 2024. Syntactic annotation of spoken and disfluent language. In Proceedings of LREC-COLING 2024
2024
-
[3]
O zlem C etino g lu and C a g r C \
\"O zlem C etino g lu and C a g r C \"o ltekin. 2019. Challenges of annotating a code-switching treebank. In Proceedings of the 18th international workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019), pages 82--90
2019
-
[4]
Clark and Thomas Wasow
Herbert H. Clark and Thomas Wasow. 1998. Using ``uh'' and ``um'' in spontaneous speaking. In David E. Meyer and Steven Kornblum, editors, Lexical and Syntactic Processing, pages 199--228. Lawrence Erlbaum Associates
1998
-
[5]
Frances Adriana Laureano De Leon, Harish Tayyar Madabushi, and Mark Lee. 2024. Code-mixed probes show how pre-trained models generalise on code-switched text. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3457--3468
2024
-
[6]
Parafita Couto, and Dan Carter
Margaret Deuchar, Peter Davies, Judith Herring, María C. Parafita Couto, and Dan Carter. 2014. Building bilingual corpora. In Enlli M. Thomas and Ineke Mennen, editors, Advances in the Study of Bilingualism, pages 93--110. Multilingual Matters, Bristol
2014
-
[7]
Kaja Dobrovoljc. 2022 a . Annotating ellipsis in dependency treebanks. Language Resources and Evaluation
2022
-
[8]
Kaja Dobrovoljc. 2022 b . Spoken language treebanks in universal dependencies: An overview. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1798--1806
2022
-
[9]
Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2019. Improving surface-syntactic universal dependencies (sud): surface-syntactic relations and deep syntactic features. In TLT 2019-18th International Workshop on Treebanks and Linguistic Theories, pages 126--132. Association for Computational Linguistics
2019
-
[10]
Kim Gerdes and Sylvain Kahane. 2016. Surface-syntactic universal dependencies. In Proceedings of COLING 2016, pages 223--235. ACL
2016
-
[11]
Carlos G \'o mez-Rodr \' guez, Muhammad Imran, David Vilares, Elena Solera, and Olga Kellert. 2024. Dancing in the syntax forest: fast, accurate and explainable sentiment analysis with salsa. In SEPLN–CEDI-PD 2024. Seminar of the Spanish Society for Natural Language Processing: Projects and System Demonstrations, volume 3729 of CEUR Workshop Proceedings, ...
2024
-
[12]
Daniel Hardt and Maribel Romero. 2004. Ellipsis and the structure of discourse. In Proceedings of SALT 14
2004
-
[13]
Muhammad Huzaifah, Weihua Zheng, Nattapol Chanpaisit, and Kui Wu. 2024. Evaluating code-switching translation with large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6381--6394
2024
-
[14]
Muhammad Imran, Olga Kellert, and Carlos G \'o mez-Rodr \' guez. 2024. A syntax-injected approach for faster and more accurate sentiment analysis. arXiv preprint arXiv:2406.15163
arXiv 2024
-
[15]
Muhammad Imran, Olga Zamaraeva, and Carlos G \'o mez-Rodr \' guez. 2025. Synner: Syntax-infused named entity recognition in the biomedical domain. JAMIA Open
2025
-
[16]
Jesin James, Vithya Yogarajan, Isabella Shields, Catherine I Watson, Peter Keegan, Keoni Mahelona, and Peter-Lucas Jones. 2022. Language models for code-switch detection of te reo m \=a ori and english in a low-resource setting. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 650--660
2022
-
[17]
Sylvain Kahane. 2019. Interpreting and defining connections in dependency structures. In 5th international conference on Dependency Linguistics (Depling), pages 89--99. Association for Computational Linguistics
2019
-
[18]
Sylvain Kahane, Bernard Caron, Emmett Strickland, and Kim Gerdes. 2021. Annotation guidelines of ud and sud treebanks for spoken corpora. In Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021), pages pp--35. Association for Computational Linguistics
2021
-
[19]
Olga Kellert, Nemika Tyagi, Muhammad Imran, Nelvin Licona-Guevara, and Carlos G \'o mez-Rodr \' guez. 2025. Parsing the switch: Llm-based universal dependency annotation for code-switched language. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics
2025
-
[20]
Boda Lin, Xinyi Zhou, Binghao Tang, Xiaocheng Gong, and Si Li. 2023. Chatgpt is a potential zero-shot dependency parser. arXiv preprint arXiv:2310.16654
Pith/arXiv arXiv 2023
-
[21]
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Deep learning for ellipsis resolution. In Proceedings of ACL 2016. ACL
2016
-
[22]
Jason Merchant. 2001. The Syntax of Silence: Sluicing, Islands, and the Theory of Ellipsis. Oxford University Press
2001
-
[23]
Leif Arda Nielsen. 2004. A Logical Approach to Ellipsis Resolution. Ph.D. thesis, University of London
2004
-
[24]
Igor Sterner and Simone Teufel. 2025 a . Code-switching and syntax: A large-scale experiment. arXiv preprint arXiv:2506.01846
Pith/arXiv arXiv 2025
-
[25]
Igor Sterner and Simone Teufel. 2025 b . Minimal pair-based evaluation of code-switching. arXiv preprint arXiv:2506.01840
Pith/arXiv arXiv 2025
-
[26]
Yuanhe Tian, Fei Xia, and Yan Song. 2024. Large language models are no longer shallow parsers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7131--7142
2024
-
[27]
Ziyan Zhang, Yang Hou, Chen Gong, and Zhenghua Li. 2025. Self-correction makes llms better parsers. arXiv preprint arXiv:2504.14165
Pith/arXiv arXiv 2025
-
[28]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.