{"id":"eebf2363-8b90-4393-98ce-200286195bf3","arxiv_id":"2506.07274","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A prompt-based GPT-4.1 pipeline with expert review produces UD-style dependency annotations for code-switched Spanish-English and Spanish-Guaraní text, including the first Spanish-Guaraní UD code-switched corpus.","lead":"The paper introduces BiLingua Parser, a GPT-4.1-based pipeline that produces Universal Dependencies annotations for code-switched Spanish-English and Spanish-Guaraní text, and releases two annotated datasets. A smart generalist should read it because it claims LLMs can bootstrap syntactic resources for low-resource languages, but the headline accuracy is computed against a reference derived from the model's own output.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95.29% LAS headline is computed against a human-corrected version of the model's own output with semantically similar UD tags merged (Table 8); the only independent gold-standard estimates are 76.32% and 59.90%, so the central accuracy claim is unsupported as stated.","rationale":"The central claim is the abstract's quantitative superiority statement: 'BiLingua Parser achieves up to 95.29% LAS after expert revision, significantly outperforming prior baselines and multilingual parsers.' That claim rests entirely on the evaluation methodology of Section 4.3. Reading Section 4.3 carefully, the second reference set is not independent: annotators review and correct the model's own output, and Table 8 explicitly groups semantically related UD tags as equivalent, with the stated policy of accepting outputs that are 'linguistically plausible.' Under that protocol, the 95.29% LAS measures consistency between the model and a human-edited version of the model, not accuracy against an external standard. The only non-circular estimate is the manually annotated gold standard (Reference Set 1), which yields 76.32% for Spanish-English and 59.90% for Spanish-Guaraní — a large gap that the paper does not reconcile. The comparison to prior work is also not controlled: Table 1 reports LAS from Hindi-English and Turkish-German corpora, so 'outperforming prior baselines' is not demonstrated on matched data. The one same-dataset baseline (UDSL, 14.71%) has no confidence interval or described test-set size. Weaker but additional evidence: Section 5.2 and Appendix D document systematic inconsistencies in the model's parses, and Section 5.3 concedes that the observed root-level switch frequencies may be inflated by parser errors, which undermines the syntactic generalization derived from those same annotations. The dataset release and the prompting framework are genuine contributions that could support a revised submission leading with the gold-standard evaluation and controlled baselines; but as written, the paper's headline does not survive scrutiny. The reader's weakest assumption identifies exactly this circularity, so I agree with the reader's analysis and verdict.","tokens_in":17195,"tokens_out":3668,"duration_ms":45812,"concrete_test":"Select a random sample of 200 Spanish-English and 100 Spanish-Guaraní code-switched sentences; have two independent expert annotators annotate them from scratch without ever seeing the LLM output, then compute strict LAS (no Table 8 tag equivalence) of the original GPT-4.1 output against this blind gold standard, reporting per-token agreement and 95% confidence intervals. If strict LAS approximates the existing 76.32%/59.90% figures rather than 95.29%/77.42%, the 'human review' headline is an artifact of comparing to a corrected version of the model's own predictions. A minimal variant: recompute Table 9 using only Reference Set 1 (manual gold standard) and disclose the number of sentences and confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 4.3's second evaluation method. The 95.29% LAS in Table 9 is not measured against an independent gold standard; it is agreement between the LLM parse and the same parse after bilingual annotators corrected it, with Table 8 treating semantically similar UD tags as equivalent and instructions to accept LLM annotations when 'linguistically plausible.' Agreement with a human-corrected version of itself conflates annotator deference with accuracy. The paper's own independent reference, the manually annotated gold standard from Section 4.3 item 1, yields 76.32% for Spanish-English and 59.90% for Spanish-Guaraní — substantially below the headline. The abstract's claim of outperforming prior baselines is also uncontrolled: Table 1 lists scores on Hindi-English, Turkish-German, and other datasets, not on the same Spanish-English/Spanish-Guaraní test sets, and the single same-dataset UDSL baseline (14.71%) is reported without confidence intervals or test-set size. Section 5.3 additionally concedes that root-level code-switch frequencies may be inflated by parser errors, undercutting the derived linguistic generalization. The released datasets and prompting recipe are valuable, but the central quantitative claim is not established by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BiLingua Parser, an LLM-based (GPT-4.1) prompting pipeline for producing Universal Dependencies (UD) annotations for code-switched text. The authors apply it to Spanish-English (Miami Corpus) and Spanish-Guaraní (GUA-SPA shared task) data, release the resulting annotated datasets (including the first Spanish-Guaraní UD-parsed corpus), and perform a switch-point analysis of the annotations. They evaluate the parser using LAS against two references: a manually annotated gold standard and human-revised LLM output, reporting 76.32% / 95.29% LAS for Spanish-English and 59.90% / 77.42% for Spanish-Guaraní, with a UDSL baseline at 14.71% on the Spanish-English gold standard. The paper concludes that, with careful prompting and expert review, LLMs can bootstrap syntactic resources for under-resourced code-switched languages.","tokens_in":17273,"tokens_out":3698,"duration_ms":41546,"significance":"The released datasets and the detailed prompting protocol are valuable contributions: the Spanish-Guaraní UD corpus is, to my knowledge, the first of its kind, and the paper's gold-standard evaluation (76.32% / 59.90% LAS) provides a credible, if modest, estimate of what an LLM prompting pipeline can achieve under expert review. The paper also advances the under-explored use of dependency structures rather than POS tags for switch-point analysis. However, the headline claim of 95.29% LAS is based on a non-independent, tolerant reference and is therefore not an accuracy estimate; the central performance claim as stated in the abstract is not supported by the evidence. The paper's own gold-standard numbers are the only non-circular accuracy estimates and should be foregrounded.","major_comments":[{"comment":"The abstract's central claim — \"achieves up to 95.29% LAS after expert revision\" — is based on comparing the LLM output with human-revised versions of the model's own output, not with an independent reference. Section 4.3 explicitly states that this second evaluation \"accepts the LLM's annotations if they fall within a linguistically plausible range,\" and Table 8 treats semantically similar UD tags as equivalent. This measures agreement between the model and a tolerant, model-derived standard; it is not an accuracy measure. The paper's independent gold-standard numbers (76.32% and 59.90%) are credible but substantially lower. The abstract and the discussion in §5.1 should lead with the gold-standard results and present the 95.29% figure only as agreement with human-revised output under a relaxed equivalence metric, or drop it from the abstract.","section":"Abstract; §4.3; §5.1 (Table 9)"},{"comment":"The claim of \"significantly outperforming prior baselines\" is not supported by a controlled comparison. Table 1 lists prior work on Hindi-English, Turkish-German, and Spanish-English POS tagging, but the LAS scores from Özateş et al. and Bhat et al. are on different datasets and different language pairs. The only same-dataset baseline, UDSL, is reported in Table 9 as 14.71%, but without confidence intervals or the size of the test set. To support the outperformance claim, the authors should run the same baselines on their own Spanish-English and Spanish-Guaraní test sets, or restrict the claim to the UDSL comparison and add uncertainty quantification.","section":"§2 (Table 1); §5.1"},{"comment":"The paper's own concession that \"The high frequency of code-switches at the root level may partly reflect parser errors, such as incorrectly analyzing modals or auxiliaries as roots\" directly undermines one of its main linguistic claims — that root-level switching is common in Spanish-English and requires a reexamination of classic constraints. Since the root-level analysis is a central finding, the paper should quantify the effect (e.g., manually validate root-level switch cases and report the error rate) or present the finding as a provisional observation with the caveat given equal prominence in the abstract and conclusion.","section":"§5.3"},{"comment":"The paper does not report the size of the manually annotated gold-standard subset, the number of annotators who contributed to it, or inter-annotator agreement for that subset. Without these details, the 76.32% and 59.90% LAS numbers cannot be evaluated for statistical reliability, and the paper's own emphasis on annotator variability (§4.3) makes this information essential. Please provide the gold-standard sample size, the annotation procedure, and confidence intervals for all reported LAS values.","section":"§4.3 (item 1)"}],"minor_comments":[{"comment":"The inter-annotator agreement (Cohen's Kappa = 0.85) is reported for the human-revised subset, but the size of that subset is never stated; please add it.","section":"§4.3"},{"comment":"The UDSL row reports \"–%\" for the Human Review column; please clarify whether this condition was not run or not applicable.","section":"Table 9"},{"comment":"The code-switched sentence counts in Table 2 (2,837 for Spanish-English, 1,140 for Spanish-Guaraní) differ from the analysis counts in §4.4 (1,711 and 877 after filtering). Please explain the filtering step (at least three tokens) explicitly in the main text rather than only in §4.4.","section":"§3.1 (Table 2); §4.4"},{"comment":"The paper states that \"A draft UD treebank exists for Spanish-English code-switching, but it is not publicly released,\" while the Introduction claims the released Spanish-English dataset is \"first\" in some sense. Please clarify the relationship between the existing draft and the released dataset to avoid a contradiction.","section":"§2"},{"comment":"The sentence \"Our findings suggest that this restriction of the linguistic theory needs to be reexamined\" is too strong given the acknowledged parser-error confound; consider softening it to propose a reexamination once validated.","section":"§5.3"},{"comment":"In the root token row, the HEAD column shows \"0 root\" while other rows show the head form; for consistency, the HEAD column for the root should be empty or \"_\", with HEAD ID 0.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The gold-standard numbers in this paper are credible and the released corpora are a genuine contribution to a low-resource area. My main concern is the framing: the abstract's 95.29% figure is not an accuracy estimate, and the 'outperforming prior baselines' claim is based on comparing across datasets. A careful revision that foregrounds the gold-standard results, adds sample sizes and confidence intervals, and runs comparable baselines on the released datasets would strengthen the paper considerably. I believe the issues are addressable within the scope of a revision and do not warrant rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, it ships something genuinely useful: the first public UD-annotated Spanish-Guaraní code-switched corpus, plus a Spanish-English CSW UD dataset and a detailed, replicable prompt recipe with sensible handling of contractions, repetitions, and ellipsis. Second, the headline result in the abstract—95.29% LAS 'after expert revision'—is not an independent accuracy estimate. It is agreement between the LLM output and the same output after human correction, with Table 8 forgiving semantically similar dependency labels and annotators told to accept 'linguistically plausible' alternatives. That makes the number a consistency measure with a tolerant reference, not a benchmark score.\n\nThe paper actually contains the more credible numbers: 76.32% LAS against a manually annotated gold standard for Spanish-English, 59.90% for Spanish-Guaraní. No confidence intervals or subset sizes are given, and the 14.71% UDSL baseline is trained on monolingual data, so the comparison is not apples-to-apples in the sense of a fair CSW parser baseline—though it does show the pipeline has real headroom over an off-the-shelf model. The authors are transparent about the two evaluation methods and even concede in Section 5.3 that root-level switch frequencies may be inflated by parser errors, which undercuts the subsequent claim that the classical prohibition on verb-phrase switching 'needs to be reexamined.'\n\nWhat the paper does well: the corpus release is the kind of resource the field needs; the prompt engineering is thoughtful and documented in the appendices; the qualitative error analysis is honest and useful. The switch-point analysis is a nice demonstration but should be read as provisional until the annotation noise is cleaned up.\n\nProportionally, the central accuracy claim is overstated as written, but the underlying assets are real and the evaluation problem is fixable. A revised version that leads with the gold-standard numbers, reports the size of the gold subset and intervals, and reframes the human-review agreement as exactly what it is would be a solid contribution.\n\nWho's this for: anyone bootstrapping UD treebanks for low-resource or code-switched languages, and anyone using LLMs for annotation who wants to see a careful prompt design with native-speaker review.\n\nMy recommendation: send it to peer review, not desk reject. The corpus alone earns referee time. I'd expect heavy revision, and I'd want the central claim reframed, but there's a publishable core here.","headline":"Genuinely new Spanish-Guaraní UD code-switched corpus, but the 95.29% LAS headline is agreement with a human-corrected version of the model's own output, not independent accuracy.","tokens_in":18014,"tokens_out":2738,"would_cite":true,"duration_ms":30371,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A prompt-guided LLM, checked by native speakers, parses code-switched text at up to 95.29% labeled attachment accuracy and produces the first Spanish-Guaraní dependency treebank.","keywords":["code-switching","Universal Dependencies","dependency parsing","large language models","low-resource languages","Spanish-Guaraní","Spanish-English","syntactic annotation"],"falsifier":"Have two independent teams of native-speaker linguists, shown no LLM output, annotate the same random sample of roughly 200 Spanish-Guaraní sentences; if the parser's LAS against that independent gold standard lands near the paper's 59.90% gold-standard figure rather than the 77.42% human-review figure, the revised-reference metric is measuring self-consistency.","tokens_in":16826,"feed_emoji":"🌐","tokens_out":8712,"duration_ms":86784,"temperature":0.7,"pith_summary":"This paper sets out to show that a carefully prompted large language model, followed by native-speaker review, can produce Universal Dependencies (UD) annotations for code-switched text that are good enough to bootstrap linguistic resources in low-resource settings. The supporting experiments annotate Spanish-English conversation transcripts and a Spanish-Guaraní social-media and news corpus, releasing the first UD-parsed Spanish-Guaraní code-switched dataset. After expert revision the pipeline reaches 95.29% labeled attachment score on Spanish-English (76.32% against an independent gold standard) and 77.42% on Spanish-Guaraní after revision (59.90% against gold). The paper also uses the resulting annotations to analyze where intra-sentential switches occur, finding that subjects and roots are frequent switch sites and that Spanish-Guaraní permits more verbal-context switches than Spanish-English. If taken at face value, the result would make LLM prompting a practical route to syntactic annotation for under-resourced code-switched language pairs.","feed_headline":"LLM pipeline parses code-switched text at 95.29%","feed_subtitle":"Prompted few-shot GPT-4.1 plus native-speaker review also yields the first Spanish-Guaraní dependency treebank.","key_machinery":"The carrying object is the BiLingua Parser pipeline. It feeds each code-switched sentence to GPT-4.1 (a large language model) with temperature 0 and a few-shot prompt that specifies the UD output fields (ID, FORM, LEMMA, UPOS, HEAD, DEPREL) and encodes rules for one root per sentence, contraction splitting, repetition and ellipsis handling, and punctuation attachment. The pipeline's second component is expert review: native speakers correct the model output, and the evaluation compares model output to the revised output while treating groups of semantically similar UD tags (e.g., xcomp vs. ccomp, nmod vs. obl) as equivalent. This prompt-plus-review-plus-flexible-metric combination is what carries the argument that LLM annotation is practically usable.","core_discovery":"The paper's central claim is that an LLM-based annotation pipeline, built on few-shot prompting with explicit UD rules and corrected by native speakers, can produce syntactically valid Universal Dependencies annotations for code-switched text where monolingually trained parsers fail. In the paper's evaluation, the BiLingua Parser reaches 95.29% labeled attachment score on Spanish-English and 77.42% on Spanish-Guaraní when compared with human-revised output, and 76.32% and 59.90% respectively against an independent gold standard; the authors read these results as evidence that LLMs can bootstrap syntactic resources for under-resourced code-switched language pairs. The paper also reports a dependency-level analysis of switch points, finding that nominal subjects and root verbs are frequent switch sites and that Spanish-Guaraní shows higher variation, including switches inside the verbal complex. On the basis of these results, the authors release the first publicly available UD-annotated code-switched datasets for Spanish-English and Spanish-Guaraní and argue that current UD evaluation metrics are too rigid for conversational multilingual data.","pith_inferences":["If the gold-standard numbers (76.32% Spanish-English, 59.90% Spanish-Guaraní) are the true quality estimates, then the headline 95.29% and 77.42% figures, measured against human-revised LLM output, mostly show that the pipeline is self-consistent; readers should quote the gold-standard numbers when describing annotation quality.","The same few-shot recipe should transfer to other low-resource code-switched pairs, but the transferable bottleneck is likely the availability of trained native-speaker annotators, not the model's ability to follow the prompt.","The semantic-equivalence tag groups amount to a new evaluation convention; if adopted widely, published LAS scores will no longer be comparable across papers that do and do not use the mapping, so the mapping should be published and standardized.","The switch-site asymmetry between Spanish-English and Spanish-Guaraní predicts that other agglutinative-plus-Spanish pairs, such as Spanish-Quechua or Spanish-Basque, will also show frequent switches inside the verbal complex; a replication corpus would test that prediction."],"forward_implications":["Code-switched language pairs with no existing syntactic treebank can get a first UD-annotated corpus through prompting plus native-speaker review, rather than months of from-scratch annotation.","Multilingual parsers trained only on monolingual treebanks are not a reliable fallback for code-switched text: the paper's comparison parser scores 14.71% LAS on Spanish-English, far below the prompted LLM.","Dependency-based switch-point analysis reveals syntactic positions that POS tagging misses, notably the high frequency of subjects and root predicates as switch sites.","Spanish-Guaraní switching is less constrained inside the verbal complex than Spanish-English switching, suggesting that typological distance shapes where bilingual speakers switch.","The first public Spanish-Guaraní UD code-switched corpus gives the dependency parsing community a new low-resource evaluation and training resource."],"supporting_citations":[{"why":"Supplies the Miami Spanish-English conversation corpus whose code-switched subset is annotated.","marker":"(Deuchar et al., 2014)"},{"why":"Supplies the Spanish-Guaraní dataset and its shared-task context for the low-resource extension.","marker":"(Chiruzzo et al., 2023)"},{"why":"Prior semi-supervised code-switching parser whose LAS baseline (~73% Turkish-German) the new results are compared against.","marker":"(Özateş et al., 2022)"},{"why":"Prior Hindi-English code-switching parsing baseline (~71% LAS) that the paper's performance claims outperform.","marker":"(Bhat et al., 2018)"},{"why":"Prior code-switched social-media tagging and parsing work used as a baseline and motivation.","marker":"(Rijhwani et al., 2017)"},{"why":"Provides the sequence-labeling parser encoding used to train the UDSL multilingual baseline.","marker":"(Roca et al., 2023)"},{"why":"UD English EWT treebank used as part of the training data for the baseline parser.","marker":"(Silveira et al., 2014)"},{"why":"UD Spanish AnCora treebank used as the other half of the baseline parser's training data.","marker":"(Taulé et al., 2008)"},{"why":"CoNLL 2018 evaluation script used to compute labeled attachment scores.","marker":"(Zeman et al., 2018)"},{"why":"Classic typology of code-switching that the switch-point analysis tests and partially challenges, especially root-level switches.","marker":"(Poplack, 1980)"}],"fun_headline_variants":["LLM parser hits 95.29% on code-switched Spanish-English","First Spanish-Guaraní UD treebank from LLM pipeline","LLM bootstraps syntax for code-switched low-resource languages","95.29% LAS on code-switched parsing via LLM guidance","LLMs outperform parsers on code-switched UD annotation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scores that carry the paper's main claim assume the human-revised LLM output is a trustworthy target, even though it starts from the same model being evaluated; if the corrections mostly ratify the model's choices, the high accuracy is agreement-with-self, not evidence of quality.","fun_headline_variants_meta":{"raw":{"variants":["LLM parser hits 95.29% on code-switched Spanish-English","First Spanish-Guaraní UD treebank from LLM pipeline","LLM bootstraps syntax for code-switched low-resource languages","95.29% LAS on code-switched parsing via LLM guidance","LLMs outperform parsers on code-switched UD annotation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3338,"prompt_tokens":1030,"completion_tokens":2308,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":2214}},"tokens_in":646,"tokens_out":2308,"duration_ms":17688,"temperature":1.0,"reasoning_tokens":2214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:37:41.442138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent teams of native-speaker linguists, shown no LLM output, annotate the same random sample of roughly 200 Spanish-Guaraní sentences; if the parser's LAS against that independent gold standard lands near the paper's 59.90% gold-standard figure rather than the 77.42% human-review figure, the revised-reference metric is measuring self-consistency.","supporting_citations":[{"cited_title":"Parafita Couto, and Dan Carter","cited_arxiv_id":null,"evidence_quote":"Supplies the Miami Spanish-English conversation corpus whose code-switched subset is annotated."},{"cited_title":"Overview of GUA-SPA at IberLEF 2023: Guarani-Spanish Code Switching Analysis","cited_arxiv_id":"2309.06163","evidence_quote":"Supplies the Spanish-Guaraní dataset and its shared-task context for the low-resource extension."},{"cited_title":"Bhat, Manish Shrivastava, and Dipti Sharma","cited_arxiv_id":null,"evidence_quote":"Prior Hindi-English code-switching parsing baseline (~71% LAS) that the paper's performance claims outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior code-switched social-media tagging and parsing work used as a baseline and motivation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sequence-labeling parser encoding used to train the UDSL multilingual baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UD English EWT treebank used as part of the training data for the baseline parser."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CoNLL 2018 evaluation script used to compute labeled attachment scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Classic typology of code-switching that the switch-point analysis tests and partially challenges, especially root-level switches."}],"review_version":1}