REVIEW 4 major objections 5 minor 45 references
UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The first officially released UD treebank for CHILDES child speech brings 48,183 gold-standard sentences from 11 children under a single annotation scheme.
desk verdict A genuinely useful resource paper that delivers the first official UD treebank for CHILDES, with honest limitations that keep the gold label softer than the headline suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the harmonization-and-validation pipeline rather than a new algorithm. Transcripts are collected through a CHILDES database interface; sentence IDs, speaker metadata, capitalization, and punctuation are normalized; reparandum and parataxis subtypes are moved into the MISC column; UD v1 flat direction and deprecated relations are converted to UD v2; stanza supplies UPOS tags for untagged trees; and every sentence is passed through the UD validation tool, with failures fixed manually. The same stanza parser, run over unsampled utterances from the same conversations, generates the silver-standard set.
What would settle it
Re-annotate a random sample of, say, 500 released gold sentences with two trained UD annotators working blind to the release, then compare their trees against the released trees on the phenomena the pipeline touched—reparanda, phrasal particles, auxiliaries, disfluent fragments, and overall attachment. If agreement is low on those categories, or if a non-trivial share of released files fail the UD validation tool, the claim of a consistent gold-standard treebank is falsified.
Extended reading notes
Core claim
The paper's central claim is that a UD v2 treebank for CHILDES can be produced by compilation rather than annotation from scratch. It takes three existing UD-style treebanks—S+24, LP21, and LP23—and harmonizes their metadata, punctuation, reparandum annotations, and dependency subtypes, runs every sentence through the UD validation tool, and manually fixes failures, making roughly 8,000 corrections. The gold output covers 48,183 sentences and 236,941 tokens from 11 children, split into train/dev/test sets that keep children separate; the silver output adds 1,197,471 sentences and 6,892,314 tokens parsed by stanza, with parser accuracy estimated at 84.2 LAS overall, 81.2 on children's speech, and 86.3 on parents' speech. The paper also claims the release is the first official UD treebank for CHILDES and that it unifies previously divergent annotation practices.
Load-bearing premise
The load-bearing premise is that the grammatical analyses inherited from the three source treebanks are correct; the manual pass fixed about 8,000 validation failures but did not independently re-verify every tree, and morphological features were not verified.
Editorial extensions
If this is right
- Researchers can train and evaluate UD parsers on child and child-directed speech under a single annotation scheme, with the reported LAS gap between children's (81.2) and parents' (86.3) speech as a baseline.
- Because each tree carries speaker role, age, gender, and original sentence ID, age-binned syntactic analyses and reconstruction of conversational turns are possible even though the released trees themselves are not conversationally ordered.
- The 1.19M-sentence silver set provides large-scale training data for adapting parsers and language models to spontaneous, disfluent speech, provided users account for its automatic origin.
- The high proportion of questions in child-directed speech (nearly half as frequent as declaratives, versus 9% in the adult GUM corpus) gives acquisition researchers a quantified input statistic that was previously hard to measure consistently.
Reading between the lines
- Editorial inference: because no inter-annotator agreement is reported for the roughly 8,000 corrections, the gold label currently rests on a single trained pass; a re-annotation study on a sample would tell how much of the apparent consistency is annotator judgment.
- Editorial inference: the silver set's child-speech LAS is 5.1 points lower than parent-speech LAS, suggesting that concentrating future manual correction on child utterances would be the cheapest path to expanding the gold set.
- Editorial inference: if the stanza-based silver trees are used as training data, the lower accuracy on disfluent fragments and function-word heads could reinforce UD-internal biases; an age- or disfluency-stratified evaluation would reveal whether the gold correction patterns are actually learned.
- Editorial inference: a testable extension would be to fine-tune a parser on the silver set and measure whether the child/parent LAS gap shrinks; if it does not, the gap may reflect annotation bias in the gold set rather than linguistic difficulty.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces UD-English-CHILDES, which the authors describe as the first officially released Universal Dependencies treebank for CHILDES data. The resource combines three earlier dependency-annotated CHILDES datasets (S+24, LP21, LP23), harmonizes them to UD v2 conventions through a manual correction pass of roughly 8,000 fixes plus automatic validation, and adds a large silver-standard set parsed with stanza. The gold set covers 48,183 sentences and 236,941 tokens from 11 children and their caregivers; the silver set covers 1,197,471 sentences and 6,892,314 tokens. The paper describes the annotation pipeline, the harmonization decisions, the data splits, and a parser-quality estimate for the silver data, and it releases the data through a public GitHub repository. The authors also disclose several limitations, including missing dialogue structure, unverified morphological features, and the fact that the official main branch is currently missing roughly 10K of the claimed gold sentences.
Significance. If the resource is taken at face value, it would fill a genuine gap: a single, consistently annotated, UD-compliant dependency treebank for child and child-directed speech would replace the divergent annotation schemes currently in use and would support both acquisition research and parser development. The paper is commendably concrete about provenance: the source corpora are enumerated, the correction categories are listed with examples, the official UD validation tool is run, and the limitations are stated explicitly rather than hidden. The release is also reproducible in principle, with a public repository, per-child statistics, and metadata such as original sentence IDs that allow conversation reconstruction. However, the gold label rests largely on inherited annotations plus a validation-driven correction pass, and the paper provides no direct evidence about the linguistic accuracy of the corrections or the silver parser output. The numerical inconsistency in the reported LAS and the current incompleteness of the official release further weaken the central claims as they now stand.
major comments (4)
- [§3.2, §4] The gold-standard claim is the central claim of the paper, but Section 3.2 states that the manual pass intervenes only on sentences that fail the UD validation tool, and Section 4 states that morphological features have not been annotated or independently verified. Because the validation tool checks structural conformance (format, projectivity, UPOS/dependency consistency) rather than linguistic correctness for child-speech phenomena, and because no inter-annotator agreement or error analysis is reported for the roughly 8,000 corrections, inherited linguistic errors from LP21, LP23, and S+24 can survive in the released 48K gold sentences undetected. The paper should provide direct evidence that the correction pass addresses linguistic accuracy, for example a sampled reannotation study or an error analysis on a held-out subset of the source treebanks.
- [§1 (footnote), §3.1] The footnote to Section 1 says the official main branch is missing approximately 10K of the claimed 48K gold sentences and asks users to use the dev branch until November 2025. This makes the headline statistics in Tables 1 and 2 not fully reproducible from the official artifact at the time of publication, which undercuts the 'first officially released' claim. The authors should either complete the main branch before acceptance or clearly state in the abstract and introduction that the released artifact currently contains only part of the described gold data.
- [§3.4] The silver-quality estimate is computed by evaluating stanza on the gold sentences from the same 11 children rather than on a sample of the silver sentences themselves. Because the silver sentences are the unsampled utterances from the same conversations and may differ in child age, disfluency rate, and sentence complexity, the reported LAS/UAS in Table 5 do not directly measure the quality of the released silver annotations. The paper should either add a sample-based evaluation of the silver output or explicitly reframe the current numbers as a proxy with known limitations.
- [§3.4, Table 5] The prose in Section 3.4 reports an overall LAS of 83.3, while Table 5 reports an overall LAS of 84.2 for the same evaluation. This numerical discrepancy affects the central silver-quality estimate and must be reconciled before the paper can be accepted.
minor comments (5)
- [§3.2] The sentence 'Sentence normalization can be found in the paper' is a placeholder rather than a description; the normalization procedure should be specified or referenced precisely.
- [Table 2] The table header says ages are given in months, but the values are in years;months format (e.g., 1;3-7;0), and the parenthetical ages for the silver corpus are not labeled with a unit. Please clarify the format.
- [Figure 2] The caption describes the example as being from the CHILDES-Providence corpus, but the metadata in the example shows corpus_name = Kuczaj and child_name = Abe. The caption should be corrected.
- [Appendix A] The title contains a typo: 'Coprora' should be 'Corpora'.
- [§3.3] The discussion of the Adam merge is confusing: the text says S+24 and LP23 overlap, but the footnote says 887 sentences from S+24 were removed because S+24 and LP21 use different data sources. Clarify which source pairs overlap and which pairs were incompatible.
Circularity Check
No self-definitional or fitted-input circularity: the treebank's claims rest on transparent compilation from prior released treebanks and external UD validation.
full rationale
This is a resource-compilation paper rather than a derivation, so the main patterns for circularity do not apply. The gold treebank is explicitly assembled from three prior released treebanks (S+24, LP21, LP23), which are external artifacts and available for independent inspection; the paper does not redefine its inputs in terms of its outputs. The central claim of UD v2 conformance is checked by the external UD validation tool, and the manual corrections are reported as approximately 8,000 fixes, not as a fitted parameter. The silver-quality estimate in Section 3.4 evaluates a fixed stanza parser against the gold data; this is a standard intrinsic evaluation, not a prediction that is forced by construction, because the gold annotations were not derived from the parser's silver outputs. The paper's own limitation statements (morphological features not annotated or independently verified; main branch missing about 10K sentences due to a postprocessing error; no inter-annotator agreement reported) concern quality and release integrity, not circularity. Self-citations to LP21/LP23/S+24 are load-bearing in the sense that those sources supply most of the annotations, but this is transparent prior-work reuse rather than an unverified uniqueness claim or an imported ansatz, and the contribution—harmonization, correction, and official release—is independently checkable against the external UD repository and validation tooling.
Assumptions & free parameters
assumptions (4)
- domain assumption The three source treebanks (S+24, LP21, LP23) are linguistically accurate enough that targeted correction yields gold-standard quality.
- domain assumption Capitalizing the first word and inferring sentence-final punctuation from metadata preserves the linguistic content of the transcripts.
- domain assumption stanza's automatic UPOS and dependency annotations are a sound starting layer for gold sentences lacking prior annotations.
- domain assumption UD v2 relations are adequate for child-speech phenomena under the paper's specific choices (flat for pure interjection utterances, reparandum and parataxis subtypes demoted to MISC).
Cite this review
Pith. "Pith review of UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions." pith.science (2026). https://pith.science/paper/7O3WDLLJ
@misc{pith2026250420304,
author = {Pith},
title = {Pith review of: UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions},
year = {2026},
howpublished = {\url{https://pith.science/paper/7O3WDLLJ}},
note = {Machine review of arXiv:2504.20304}
}
read the original abstract
CHILDES is a widely used resource of transcribed child and child-directed speech. This paper introduces UD-English-CHILDES, the first officially released Universal Dependencies (UD) treebank. It is derived from previously dependency-annotated CHILDES data, which we harmonize to follow unified annotation principles. The gold-standard trees encompass utterances sampled from 11 children and their caregivers, totaling over 48K sentences (236K tokens). We validate these gold-standard annotations under the UD v2 framework and provide an additional 1M~silver-standard sentences, offering a consistent resource for computational and linguistic research.
Figures
Reference graph
Works this paper leans on
-
[1]
Smith, Sharon Goldwater, and Mark Steedman
Omri Abend, Tom Kwiatkowski, Nathaniel J. Smith, Sharon Goldwater, and Mark Steedman. 2017. http://www.sciencedirect.com/science/article/pii/S0010027717300495 Bootstrapping language acquisition . Cognition, 164:116--143
work page 2017
-
[2]
Ruth A. Berman. 1990. https://doi.org/doi:10.1515/ling.1990.28.6.1135 On acquiring an (S)VO language: subjectless sentences in children’s Hebrew . Linguistics, 28(6):1135--1166
- [3]
-
[4]
Roger Brown. 1973. A first language: The early stages. Harvard University Press
1973
-
[5]
Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. https://doi.org/10.48550/arXiv.2502.10645 BabyLM turns 3: Call for papers for the 2025 BabyLM workshop . arXiv preprint arXiv:2502.10645
-
[6]
Noam Chomsky. 1976. Reflections on language. Temple Smith London
work page 1976
-
[7]
Leshem Choshen, Ryan Cotterell, Michael Y Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. https://doi.org/10.48550/arXiv.2404.06214 [ C all for papers] the 2nd BabyLM challenge: Sample-efficient pretraining on a developmentally plausible corpus . arXiv preprint arXiv:2404.06214
-
[8]
Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, and Roger Levy. 2023. https://doi.org/10.1162/tacl_a_00589 A cross-linguistic pressure for U niform I nformation D ensity in word order . Transactions of the Association for Computational Linguistics, 11:1048--1065
Show all 45 references
-
[9]
Katherine Demuth, Jennifer Culbertson, and Jennifer Alter. 2006. https://doi.org/10.1177/00238309060490020201 Word-minimality, epenthesis and coda licensing in the early acquisition of E nglish . Language and speech, 49(2):137--173
2006 doi
-
[10]
Kaja Dobrovoljc. 2022. https://aclanthology.org/2022.lrec-1.191/ Spoken language treebanks in U niversal D ependencies: an overview . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1798--1806, Marseille, France. European Language Resources...
2022
-
[11]
Feng, Noah Goodman, and Michael Frank
Steven Y. Feng, Noah Goodman, and Michael Frank. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1231 Is child-directed speech effective training data for language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22055-...
2024 doi
-
[12]
Stephani Foraker, Terry Regier, Naveen Khetarpal, Amy Perfors, and Joshua Tenenbaum. 2009. https://doi.org/10.1111/j.1551-6709.2009.01014.x Indirect evidence and the poverty of the stimulus: The case of anaphoric one . Cognitive Science, 33(2):287--300
2009
-
[13]
Heidi R Getz. 2019. https://doi.org/10.1080/10489223.2018.1470242 Acquiring wanna: Beyond U niversal G rammar . Language Acquisition, 26(2):119--143
2019
-
[14]
Ronald Bradley Gillam and Nils A Pearson. 2004. Test of narrative language. Pro-ed Austin, TX
2004
-
[15]
Jean Berko Gleason. 1980. https://doi.org/10.1016/B978-0-08-024696-3.50009-0 The acquisition of social speech routines and politeness formulas . In Language, pages 21--27. Elsevier
1980 doi
-
[16]
Shai Gretz, Alon Itai, Brian MacWhinney, Bracha Nir, and Shuly Wintner. 2015. https://link.springer.com/content/pdf/10.1007/s10579-013-9256-x.pdf Parsing H ebrew CHILDES transcripts . Language Resources and Evaluation, 49:107--145
2015 doi
-
[17]
Michael Hahn, Dan Jurafsky, and Richard Futrell. 2020. https://doi.org/10.1073/pnas.1910923117 Universals of word order reflect optimization of grammars for efficient communication . Proceedings of the National Academy of Sciences, 117(5):2347--2353
2020 doi
-
[18]
Rui Huang. 2016. https://academicworks.cuny.edu/gc_etds/1577 An evaluation of POS taggers for the CHILDES corpus . CUNY Academic Works
2016
- [19]
-
[20]
Sylvain Kahane, Martine Vanhove, Rayan Ziane, and Bruno Guillaume. 2021. https://aclanthology.org/2021.tlt-1.5/ A morph-based and a word-based treebank for B eja . In Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021), pa...
2021
-
[21]
Stan Kuczaj. 1977. https://doi.org/10.1016/S0022-5371(77)80021-2 The acquisition of regular and irregular past tense forms . Journal of Verbal Learning and Verbal Behavior, 16(5):589--600
1977 doi
-
[22]
Elena Lieven, Doroth\' e Salomo, and Michael Tomasello. 2009. https://doi.org/10.1515/COGL.2009.022 Two-year-old children's production of multiword utterances: A usage-based analysis . Cognitive Linguistics, 20(3):481--507
2009 doi
-
[23]
Houjun Liu and Brian MacWhinney . 2024. https://lps.library.cmu.edu/LDR/article/id/810/ Morphosyntactic analysis for CHILDES . Language Development Research, 4(1)
2024
-
[24]
Zoey Liu and Emily Prud ' hommeaux. 2021. https://aclanthology.org/2021.adaptnlp-1.16/ Dependency parsing evaluation for low-resource spontaneous speech . In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 156--165, Kyiv, Ukraine. Association for Computa...
2021
-
[25]
Zoey Liu and Emily Prud ' hommeaux. 2023. https://doi.org/10.1162/tacl_a_00624 Data-driven parsing evaluation for child-parent interactions . Transactions of the Association for Computational Linguistics, 11:1734--1753
2023 doi
-
[26]
Brian MacWhinney. 2000. The CHILDES project: Tools for analyzing talk, Volume I: Transcription format and programs . Psychology Press
2000
-
[27]
Rochelle S Newman, Meredith L Rowe, and Nan Bernstein Ratner. 2016. https://doi.org/10.1017/S0305000915000446 Input and uptake at 7 months predicts toddler vocabulary: the role of child-directed speech and infant processing skills in language development . Journal of Child Lan...
2016 doi
-
[28]
Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Haji c , Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. https://aclanthology.org/L16-1262/ U niversal D ependencies v1: A mu...
2016
- [29]
-
[30]
Mai Omura, Hiroshi Matsuda, Masayuki Asahara, and Aya Wakasa. 2023. https://doi.org/10.18653/v1/2023.sigdial-1.29 UD \_ J apanese- CEJC : Dependency relation annotation on corpus of everyday J apanese conversation . In Proceedings of the 24th Annual Meeting of the Special Inte...
2023 doi
-
[31]
Juri Opitz, Shira Wein, and Nathan Schneider. 2025. https://doi.org/10.1162/coli_a_00560 Natural language processing RELIES on linguistics . Computational Linguistics, pages 1--23
2025 doi
-
[32]
Lisa Pearl and Benjamin Mis. 2011. https://escholarship.org/uc/item/8wc5w9d2 How far can indirect evidence take us? Anaphoric one revisited . In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33
2011
-
[33]
Lisa Pearl and Jon Sprouse. 2013. https://doi.org/10.1080/10489223.2012.738742 Syntactic islands and learning biases: Combining experimental syntax and computational modeling to investigate the language acquisition problem . Language Acquisition, 20(1):23--68
2013
-
[34]
Tenenbaum, and Terry Regier
Amy Perfors, Joshua B. Tenenbaum, and Terry Regier. 2011. https://doi.org/10.1016/j.cognition.2010.11.001 The learnability of abstract syntactic principles . Cognition, 118(3):306--338
2011 doi
-
[35]
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. https://doi.org/10.18653/v1/2020.acl-demos.14 S tanza: A Python Natural Language Processing Toolkit for many human languages . In Proceedings of the 58th Annual Meeting of the Association for Co...
2020 doi
-
[36]
Kenji Sagae, Eric Davis, Alon Lavie, Brian MacWhinney, and Shuly Wintner. 2007. https://aclanthology.org/W07-0604/ High-accuracy annotation and parsing of CHILDES transcripts . In Proceedings of the Workshop on Cognitive Aspects of Computational Language Acquisition, pages 25-...
2007
-
[37]
Kenji Sagae, Eric Davis, Alon Lavie, Brian MacWhinney, and Shuly Wintner. 2010. https://doi.org/10.1017/S0305000909990407 Morphosyntactic annotation of CHILDES transcripts . Journal of Child Language, 37(3):705–729
2010 doi
-
[38]
Kenji Sagae, Alon Lavie, and Brian MacWhinney. 2005. https://aclanthology.org/P05-1025/ Automatic measurement of syntactic development in child language . In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics ( ACL `05) , pages 197--204, An...
2005
-
[39]
Kenji Sagae, Brian MacWhinney, and Alon Lavie. 2004. https://aclanthology.org/L04-1484/ Adding syntactic annotations to transcripts of parent-child dialogs . In Proceedings of the Fourth International Conference on Language Resources and Evaluation ( LREC `04) , Lisbon, Portug...
2004
-
[40]
Meylan, Mika Braginsky, Kyle E
Alessandro Sanchez, Stephan C. Meylan, Mika Braginsky, Kyle E. MacDonald , Daniel Yurovsky, and Michael C. Frank. 2019. https://doi.org/10.3758/s13428-018-1176-7 childes-db: A flexible and reproducible interface to the child language data exchange system . Behavior Research Me...
2019 doi
-
[41]
Ida Szubert, Omri Abend, Nathan Schneider, Samuel Gibbon, Louis Mahon, Sharon Goldwater, and Mark Steedman. 2024. https://doi.org/10.1007/s10579-024-09734-y Cross-linguistically consistent semantic and syntactic annotation of child-directed speech . Language Resources and Evaluation
2024 doi
-
[42]
Richard M Weist and Andrea A Zevenbergen. 2008. https://doi.org/10.1080/15475440802293490 Autobiographical memory and past time reference . Language Learning and Development, 4(4):291--308
2008 doi
-
[43]
Amir Zeldes. 2017. http://dx.doi.org/10.1007/s10579-016-9343-x The GUM corpus: creating multilayer resources in the classroom . Language Resources and Evaluation, 51(3):581--612
2017 doi
-
[44]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[45]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.