REVIEW 4 major objections 6 minor 25 references
Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A prompt-guided LLM, checked by native speakers, parses code-switched text at up to 95.29% labeled attachment accuracy and produces the first Spanish-Guaraní dependency treebank.
desk verdict Genuinely new Spanish-Guaraní UD code-switched corpus, but the 95.29% LAS headline is agreement with a human-corrected version of the model's own output, not independent accuracy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the BiLingua Parser pipeline. It feeds each code-switched sentence to GPT-4.1 (a large language model) with temperature 0 and a few-shot prompt that specifies the UD output fields (ID, FORM, LEMMA, UPOS, HEAD, DEPREL) and encodes rules for one root per sentence, contraction splitting, repetition and ellipsis handling, and punctuation attachment. The pipeline's second component is expert review: native speakers correct the model output, and the evaluation compares model output to the revised output while treating groups of semantically similar UD tags (e.g., xcomp vs. ccomp, nmod vs. obl) as equivalent. This prompt-plus-review-plus-flexible-metric combination is what carries the argument that LLM annotation is practically usable.
What would settle it
Have two independent teams of native-speaker linguists, shown no LLM output, annotate the same random sample of roughly 200 Spanish-Guaraní sentences; if the parser's LAS against that independent gold standard lands near the paper's 59.90% gold-standard figure rather than the 77.42% human-review figure, the revised-reference metric is measuring self-consistency.
Extended reading notes
Core claim
The paper's central claim is that an LLM-based annotation pipeline, built on few-shot prompting with explicit UD rules and corrected by native speakers, can produce syntactically valid Universal Dependencies annotations for code-switched text where monolingually trained parsers fail. In the paper's evaluation, the BiLingua Parser reaches 95.29% labeled attachment score on Spanish-English and 77.42% on Spanish-Guaraní when compared with human-revised output, and 76.32% and 59.90% respectively against an independent gold standard; the authors read these results as evidence that LLMs can bootstrap syntactic resources for under-resourced code-switched language pairs. The paper also reports a dependency-level analysis of switch points, finding that nominal subjects and root verbs are frequent switch sites and that Spanish-Guaraní shows higher variation, including switches inside the verbal complex. On the basis of these results, the authors release the first publicly available UD-annotated code-switched datasets for Spanish-English and Spanish-Guaraní and argue that current UD evaluation metrics are too rigid for conversational multilingual data.
Load-bearing premise
The scores that carry the paper's main claim assume the human-revised LLM output is a trustworthy target, even though it starts from the same model being evaluated; if the corrections mostly ratify the model's choices, the high accuracy is agreement-with-self, not evidence of quality.
Editorial extensions
If this is right
- Code-switched language pairs with no existing syntactic treebank can get a first UD-annotated corpus through prompting plus native-speaker review, rather than months of from-scratch annotation.
- Multilingual parsers trained only on monolingual treebanks are not a reliable fallback for code-switched text: the paper's comparison parser scores 14.71% LAS on Spanish-English, far below the prompted LLM.
- Dependency-based switch-point analysis reveals syntactic positions that POS tagging misses, notably the high frequency of subjects and root predicates as switch sites.
- Spanish-Guaraní switching is less constrained inside the verbal complex than Spanish-English switching, suggesting that typological distance shapes where bilingual speakers switch.
- The first public Spanish-Guaraní UD code-switched corpus gives the dependency parsing community a new low-resource evaluation and training resource.
Reading between the lines
- If the gold-standard numbers (76.32% Spanish-English, 59.90% Spanish-Guaraní) are the true quality estimates, then the headline 95.29% and 77.42% figures, measured against human-revised LLM output, mostly show that the pipeline is self-consistent; readers should quote the gold-standard numbers when describing annotation quality.
- The same few-shot recipe should transfer to other low-resource code-switched pairs, but the transferable bottleneck is likely the availability of trained native-speaker annotators, not the model's ability to follow the prompt.
- The semantic-equivalence tag groups amount to a new evaluation convention; if adopted widely, published LAS scores will no longer be comparable across papers that do and do not use the mapping, so the mapping should be published and standardized.
- The switch-site asymmetry between Spanish-English and Spanish-Guaraní predicts that other agglutinative-plus-Spanish pairs, such as Spanish-Quechua or Spanish-Basque, will also show frequent switches inside the verbal complex; a replication corpus would test that prediction.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BiLingua Parser, an LLM-based (GPT-4.1) prompting pipeline for producing Universal Dependencies (UD) annotations for code-switched text. The authors apply it to Spanish-English (Miami Corpus) and Spanish-Guaraní (GUA-SPA shared task) data, release the resulting annotated datasets (including the first Spanish-Guaraní UD-parsed corpus), and perform a switch-point analysis of the annotations. They evaluate the parser using LAS against two references: a manually annotated gold standard and human-revised LLM output, reporting 76.32% / 95.29% LAS for Spanish-English and 59.90% / 77.42% for Spanish-Guaraní, with a UDSL baseline at 14.71% on the Spanish-English gold standard. The paper concludes that, with careful prompting and expert review, LLMs can bootstrap syntactic resources for under-resourced code-switched languages.
Significance. The released datasets and the detailed prompting protocol are valuable contributions: the Spanish-Guaraní UD corpus is, to my knowledge, the first of its kind, and the paper's gold-standard evaluation (76.32% / 59.90% LAS) provides a credible, if modest, estimate of what an LLM prompting pipeline can achieve under expert review. The paper also advances the under-explored use of dependency structures rather than POS tags for switch-point analysis. However, the headline claim of 95.29% LAS is based on a non-independent, tolerant reference and is therefore not an accuracy estimate; the central performance claim as stated in the abstract is not supported by the evidence. The paper's own gold-standard numbers are the only non-circular accuracy estimates and should be foregrounded.
major comments (4)
- [Abstract; §4.3; §5.1 (Table 9)] The abstract's central claim — "achieves up to 95.29% LAS after expert revision" — is based on comparing the LLM output with human-revised versions of the model's own output, not with an independent reference. Section 4.3 explicitly states that this second evaluation "accepts the LLM's annotations if they fall within a linguistically plausible range," and Table 8 treats semantically similar UD tags as equivalent. This measures agreement between the model and a tolerant, model-derived standard; it is not an accuracy measure. The paper's independent gold-standard numbers (76.32% and 59.90%) are credible but substantially lower. The abstract and the discussion in §5.1 should lead with the gold-standard results and present the 95.29% figure only as agreement with human-revised output under a relaxed equivalence metric, or drop it from the abstract.
- [§2 (Table 1); §5.1] The claim of "significantly outperforming prior baselines" is not supported by a controlled comparison. Table 1 lists prior work on Hindi-English, Turkish-German, and Spanish-English POS tagging, but the LAS scores from Özateş et al. and Bhat et al. are on different datasets and different language pairs. The only same-dataset baseline, UDSL, is reported in Table 9 as 14.71%, but without confidence intervals or the size of the test set. To support the outperformance claim, the authors should run the same baselines on their own Spanish-English and Spanish-Guaraní test sets, or restrict the claim to the UDSL comparison and add uncertainty quantification.
- [§5.3] The paper's own concession that "The high frequency of code-switches at the root level may partly reflect parser errors, such as incorrectly analyzing modals or auxiliaries as roots" directly undermines one of its main linguistic claims — that root-level switching is common in Spanish-English and requires a reexamination of classic constraints. Since the root-level analysis is a central finding, the paper should quantify the effect (e.g., manually validate root-level switch cases and report the error rate) or present the finding as a provisional observation with the caveat given equal prominence in the abstract and conclusion.
- [§4.3 (item 1)] The paper does not report the size of the manually annotated gold-standard subset, the number of annotators who contributed to it, or inter-annotator agreement for that subset. Without these details, the 76.32% and 59.90% LAS numbers cannot be evaluated for statistical reliability, and the paper's own emphasis on annotator variability (§4.3) makes this information essential. Please provide the gold-standard sample size, the annotation procedure, and confidence intervals for all reported LAS values.
minor comments (6)
- [§4.3] The inter-annotator agreement (Cohen's Kappa = 0.85) is reported for the human-revised subset, but the size of that subset is never stated; please add it.
- [Table 9] The UDSL row reports "–%" for the Human Review column; please clarify whether this condition was not run or not applicable.
- [§3.1 (Table 2); §4.4] The code-switched sentence counts in Table 2 (2,837 for Spanish-English, 1,140 for Spanish-Guaraní) differ from the analysis counts in §4.4 (1,711 and 877 after filtering). Please explain the filtering step (at least three tokens) explicitly in the main text rather than only in §4.4.
- [§2] The paper states that "A draft UD treebank exists for Spanish-English code-switching, but it is not publicly released," while the Introduction claims the released Spanish-English dataset is "first" in some sense. Please clarify the relationship between the existing draft and the released dataset to avoid a contradiction.
- [§5.3] The sentence "Our findings suggest that this restriction of the linguistic theory needs to be reexamined" is too strong given the acknowledged parser-error confound; consider softening it to propose a reexamination once validated.
- [Table 3] In the root token row, the HEAD column shows "0 root" while other rows show the head form; for consistency, the HEAD column for the root should be empty or "_", with HEAD ID 0.
Circularity Check
The 95.29% LAS headline is computed against the model's own human-corrected output with semantically similar UD tags merged, so the central accuracy claim is self-referential; the only independent gold-standard estimates are 76.32% and 59.90%.
-
self definitional
[Sec. 4.3 (Annotation Validation and Evaluation), Table 8, Table 9; Abstract]
"In a faster second round, two bilingual annotators reviewed and corrected the model’s own parse outputs. ... This approach accepts the LLM’s annotations if they fall within a linguistically plausible range, even when differing from canonical UD labels. ... we treat sets of semantically related UD tags (see Table 8) as equivalent. Differences within each group are not counted as errors under our human-aligned evaluation."
The 95.29% LAS in Table 9 (also quoted in the Abstract as 'up to 95.29% LAS after expert revision') is not measured against an independent gold standard. The reference set is the model's own output after correction by annotators instructed to accept linguistically plausible LLM annotations, and the scoring rule merges semantically similar UD tags (Table 8). Agreement with a corrected version of itself under a tolerance relation is an agreement-with-self measure, not an accuracy estimate. The paper's own independent manually annotated gold standard yields 76.32% LAS for Spanish-English and 59.90% for Spanish-Guaraní, substantially below the headline, confirming that the headline number reduces by construction to self-consistency after revision.
full rationale
The paper contains one genuinely independent evaluation: a small random subset manually annotated by linguistic experts (Sec. 4.3, method 1), giving 76.32% LAS for Spanish-English and 59.90% for Spanish-Guaraní. That part is not circular. The circularity is in the second evaluation method and in the Abstract's use of its 95.29% figure as the headline result: the 'Human Review' reference is the LLM's own output corrected by annotators, with semantically similar UD tag groups treated as equivalent and plausible LLM variants accepted. The 95.29% therefore measures agreement with a tolerant, model-derived reference, not parser accuracy. The paper is transparent about the method and even concedes in Sec. 5.2 that LLM responses are inconsistent and in Sec. 5.3 that root-level code-switch frequencies may be inflated by parser errors, which further undercuts the linguistic generalization built on the same annotations. No load-bearing self-citation chain is present; the UDSL baseline uses CoDeLin, a framework by one of the current authors, but that is an implementation choice, not the basis of the accuracy claim. Overall, because the central quantitative claim in the Abstract reduces by construction to agreement with the model's own revised output, the circularity score is high.
Assumptions & free parameters
free parameters (2)
- UD tag equivalence groups =
9 groups of semantically similar UD tags (Table 8)
- Minimum token filter for switch-point analysis =
3 tokens
assumptions (3)
- domain assumption Human-revised LLM output, with semantically similar UD tags grouped as equivalent, is a valid reference for computing LAS
- domain assumption GPT-4.1 with temperature=0 and top_p=1 produces deterministic, stable annotations
- domain assumption The UDSL parser trained on monolingual UD treebanks is a representative baseline for code-switched parsing
Cite this review
Pith. "Pith review of Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages." pith.science (2026). https://pith.science/paper/MTCHCM25
@misc{pith2026250607274,
author = {Pith},
title = {Pith review of: Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages},
year = {2026},
howpublished = {\url{https://pith.science/paper/MTCHCM25}},
note = {Machine review of arXiv:2506.07274}
}
read the original abstract
Code-switching presents a complex challenge for syntactic analysis, especially in low-resource language settings where annotated data is scarce. While recent work has explored the use of large language models (LLMs) for sequence-level tagging, few approaches systematically investigate how well these models capture syntactic structure in code-switched contexts. Moreover, existing parsers trained on monolingual treebanks often fail to generalize to multilingual and mixed-language input. To address this gap, we introduce the BiLingua Parser, an LLM-based annotation pipeline designed to produce Universal Dependencies (UD) annotations for code-switched text. First, we develop a prompt-based framework for Spanish-English and Spanish-Guaran\'i data, combining few-shot LLM prompting with expert review. Second, we release two annotated datasets, including the first Spanish-Guaran\'i UD-parsed corpus. Third, we conduct a detailed syntactic analysis of switch points across language pairs and communicative contexts. Experimental results show that BiLingua Parser achieves up to 95.29% LAS after expert revision, significantly outperforming prior baselines and multilingual parsers. These results show that LLMs, when carefully guided, can serve as practical tools for bootstrapping syntactic resources in under-resourced, code-switched environments. Data and source code are available at https://github.com/N3mika/ParsingProject
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Bhat, Manish Shrivastava, and Dipti Sharma
Irshad Bhat, Riyaz A. Bhat, Manish Shrivastava, and Dipti Sharma. 2018. https://doi.org/10.18653/v1/N18-1090 U niversal D ependency parsing for H indi- E nglish code-switching . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages ...
-
[2]
Barbara E Bullock and Almeida Jacqueline Toribio. 2009. Trying to hit a moving target: On the sociophonetics of code-switching. International Journal of Bilingualism, 13(2):165--193
work page 2009
-
[3]
Jie Chi and Peter Bell. 2024. https://doi.org/10.18653/v1/2024.findings-eacl.120 Analyzing the role of part-of-speech in code-switching: A corpus-based study . In Findings of the Association for Computational Linguistics: EACL 2024, pages 1801--1811, St. Julian's, Malta. Association for Computational Linguistics
-
[4]
Luis Chiruzzo, Marvin Agüero-Torales, Gustavo Giménez-Lugo, Aldo Alvarez, Yliana Rodríguez, Santiago Góngora, and Thamar Solorio. 2023. http://arxiv.org/abs/2309.06163 Overview of gua-spa at iberlef 2023: Guarani-spanish code switching analysis
work page Pith review arXiv 2023
-
[5]
Parafita Couto, and Dan Carter
Margaret Deuchar, Peter Davies, Judith Herring, María C. Parafita Couto, and Dan Carter. 2014. Building bilingual corpora. In Enlli M. Thomas and Ineke Mennen, editors, Advances in the Study of Bilingualism, pages 93--110. Multilingual Matters, Bristol
work page 2014
-
[6]
Melinda Fricke and Gerrit Jan Kootstra. 2016. https://doi.org/10.1016/j.jml.2016.04.003 Primed codeswitching in spontaneous bilingual dialogue . Journal of Memory and Language, 91:181--201
-
[7]
Ryan Georgi, Yating Wang, and Fei Xia. 2021. https://aclanthology.org/2021.tlt-1.4 Evaluating dependency parsers on spoken language transcripts . In Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021), pages 33--43, Online. Association for Computational Linguistics
work page 2021
-
[8]
Víctor Soto Martínez. 2020. https://www.cs.columbia.edu/speech/ThesisFiles/victor_soto.pdf Identifying and Modeling Code-Switched Language . Ph.D. thesis, Columbia University, New York, NY
work page 2020
Show all 25 references
-
[9]
Carol Myers-Scotton. 1993. Duelling languages: Grammatical structure in code-switching. Oxford University Press
1993
-
[10]
Carol Myers-Scotton. 2002. Contact Linguistics: Bilingual Encounters and Grammatical Outcomes. Oxford University Press, Oxford, New York
2002
-
[11]
Maria Carmen Parafita Couto, Marianne Gullberg, and Pieter Muysken. 2015. Subject positioning in spanish--english code-switching. Linguistic Approaches to Bilingualism, 5(3):277--300
2015
-
[12]
Shana Poplack. 1980. Sometimes i'll start a sentence in spanish y termino en español: Toward a typology of code-switching. Linguistics, 18(7-8):581--618
1980
-
[13]
Shruti Rijhwani and Thamar Solorio. 2016. https://doi.org/10.18653/v1/W16-5812 Estimating code-switching on twitter with a novel generalized word-level classification model . In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 33--42, Aus...
2016 doi
-
[14]
Shruti Rijhwani, Lawrence Wolf-Sonkin, Victor Kuperman, Timothy Baldwin, and Thamar Solorio. 2017. Analyzing code-switched social media text. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
2017
-
[15]
Diego Roca, David Vilares, and Carlos G \'o mez-Rodr \'i guez. 2023. https://api.semanticscholar.org/CorpusID:252912242 A system for constituent and dependency tree linearization . Kalpa Publications in Computing
2023
-
[16]
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for E nglish. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (...
2014
-
[17]
Thamar Solorio, Emily Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Di Lin, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Wo...
2014
-
[18]
Thamar Solorio and Yang Liu. 2008. https://aclanthology.org/D08-1111 Part-of-speech tagging for english-spanish code-switched text . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 1051--1060, Honolulu, Hawaii. Association for C...
2008
-
[19]
Michalina Strzyz, David Vilares, and Carlos G \'o mez-Rodr \'i guez. 2019. https://api.semanticscholar.org/CorpusID:67855320 Viable dependency parsing as sequence labeling . ArXiv, abs/1902.10505
2019 arXiv
-
[20]
Mariona Taul \'e , Maria Ant \`o nia Mart \' , and Marta Recasens. 2008. Ancora: Multilevel annotated corpora for catalan and spanish. In Lrec, volume 2008, pages 96--101
2008
-
[21]
Almeida Jacqueline Toribio. 2001. Accessing bilingual code-switching competence. International Journal of Bilingualism, 5(4):403--436
2001
-
[22]
Daniel Zeman, Jan Haji c , Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. https://doi.org/10.18653/v1/K18-2001 C o NLL 2018 shared task: Multilingual parsing from raw text to U niversal D ependencies . In Proceedings of the C o ...
2018 doi
-
[23]
Şazi Murat Özateş, Özlem Çetinoğlu, Reut Tsarfaty, Dilek Küçük, and Olcay Taner Yıldız. 2022. https://aclanthology.org/2022.findings-naacl.87 Improving code-switching dependency parsing with semi-supervised auxiliary tasks . In Findings of the Association for Computational Lin...
2022
-
[24]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[25]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.