Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

FinchGPT: a Transformer based language model for birdsong analysis

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transformer trained from scratch on Bengalese finch songs predicts the next syllable better than Markov, RNN, and LSTM models, and its attention maps reveal long-range dependencies in the song structure.

desk verdict A novel and promising application of transformers to birdsong, but the key evidence for long-range dependencies rests on an uncontrolled synthetic corpus. read the letter →

arxiv 2502.00344 v1 pith:PRH4G6HB submitted 2025-02-01 cs.CL

Kosei Kobayashi , Kosuke Matsuzaki , Masaya Taniguchi , Keisuke Sakaguchi , Kentaro Inui , Kentaro Abe This is my paper · ORCID
classification cs.CL
keywords birdsongBengalesefinchtransformerlanguagemodelnext-syllablepredictionlong-rangedependenciesattentionanalysisHVCablationanimalvocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a GPT-2-style transformer, trained from scratch on text-converted Bengalese finch songs, predicts the next syllable more accurately than Markov, RNN, and LSTM models. It argues the transformer's advantage comes from long-range dependencies in the songs, not merely local transition statistics. Evidence includes attention visualization showing distant syllables attended to in deeper layers, and experiments where restricting the attention span to a few syllables hurts prediction. The paper also shows that songs from birds after HVC ablation lose the long-range structure the model relies on, linking the model's behavior to a biological manipulation of song syntax. If correct, transformer language models become a general tool for detecting hierarchical structure in non-human vocalizations.

What carries the argument

The load-bearing object is the self-attention mechanism of a GPT-2-style transformer applied to a tokenized song corpus, with each syllable as a token and start and end markers. The model is compared against a 6th-order Markov model on natural songs and on a synthetic corpus with the same local statistics but no dependencies beyond six syllables; that synthetic corpus is the null model that isolates the contribution of long-range context. Attention span restriction, implemented by masking tokens farther away than a fixed distance in every layer, is the experiment that turns the architecture's access to distant context on and off. The paper also uses a graph-theoretic extraction of dependency trees from attention weights to visualize which syllables depend on which, and HVC ablation as a biological manipulation that disrupts long-range ordering without changing syllable acoustics.

What would settle it

Regenerate the artificial corpus from the 6th-order Markov model while explicitly matching the natural corpus' song-length distribution and start/end token handling, then retrain FinchGPT on it. If the transformer's accuracy no longer drops relative to the Markov model, the reported decline is an artifact of the generation procedure, not evidence that natural songs contain dependencies beyond six syllables.

Watch

Extended reading notes

Core claim

The central discovery is that Bengalese finch songs contain predictive dependencies spanning more than six syllables, and a transformer can exploit them. FinchGPT, a GPT-2-style model with 6 layers, 6 attention heads, and 384 hidden units, outperforms a 6th-order Markov model and recurrent networks at next-syllable prediction on corpora from three birds. When the same architecture is trained and tested on an artificial corpus generated by a 6th-order Markov process that by construction contains no dependencies beyond six syllables, the transformer's accuracy drops significantly while Markov's does not. Attention weights in later layers reach across motifs, and limiting the attention span to fewer than roughly six preceding tokens monotonically worsens cross-entropy. The paper interprets these results as evidence that the songs are not Markovian and that long-range sequential rules are functionally relevant.

Load-bearing premise

The whole argument for long-range structure depends on the artificial 6th-order Markov corpus faithfully removing all dependencies beyond six syllables while preserving every other statistical property of the natural songs, since any artifact in that generation procedure could produce the transformer's accuracy drop without genuine long-range dependencies.

Editorial extensions

If this is right

  • Next-syllable accuracy and cross-entropy on song corpora can rank model architectures, with attention-based models capturing structure that finite-order Markov and recurrent models miss.
  • Restricting attention span provides a quantitative readout of the distance over which sequential dependencies operate; for Bengalese finches, the steepest degradation appears between 3 and 10 syllables.
  • HVC ablation removes long-range syllabic order while leaving syllable acoustics intact, so the same model can detect loss of syntactic structure from song text alone.
  • A transformer trained on pre-ablation songs can classify whether a song was produced before or after HVC ablation, with accuracy falling as attention span shrinks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to apply the same attention-span-restriction protocol to recordings from other species; the distance at which cross-entropy rises would give a comparable context length for each species' vocal sequences.
  • A direct comparison with biology would record HVC neurons during singing and ask whether their sequence selectivity matches FinchGPT's effective receptive field, turning the model's attention maps into hypotheses about neural coding.
  • The synthetic Markov null is one possible baseline; a more demanding null would be a hierarchical grammar with variable motif repetition, which could reveal whether the transformer's advantage is detecting genuine long-range syntax rather than statistical regularities in chunk boundaries.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces FinchGPT, a GPT-2-style transformer trained from scratch on texturized Bengalese finch songs from three individual birds. It reports that this model achieves higher next-syllable prediction accuracy and lower cross-entropy than Markov, RNN, and LSTM baselines (Fig. 1C–D), and that its accuracy drops more than the Markov baseline when both are evaluated on an artificial corpus generated from a 6th-order Markov model (Fig. 1E). Attention visualization is used to argue that deeper layers attend to longer spans (Fig. 2A–C), and restricting the attention span degrades performance (Fig. 2D). Finally, the authors show that HVC ablation changes song statistics and reduces FinchGPT's performance and classification accuracy, which they interpret as a loss of long-range dependencies (Fig. 4).

Significance. The paper asks a timely and important question: whether transformer language models can reveal non-adjacent sequential structure in animal vocalizations. Its strengths include a multi-bird corpus, a causal attention-span manipulation, and the use of a Markov null corpus as a reference point. If the central claim is established, the approach could provide a broadly applicable tool for comparative studies of vocal sequence structure. However, the current evidence is not yet sufficient: the Markov null corpus is not shown to match the natural corpus in incidental statistics, and the main accuracy and scaling comparisons lack reported inferential statistics. These issues are fixable with additional analyses.

major comments (4)
  1. [§3, 'Procedures for training Markov, RNN, LSTM, and Transformer models'; Fig. 1E] The artificial 6th-order Markov corpus is not demonstrated to match the natural corpus on song-length distribution and boundary statistics. The Methods state only that transition probabilities including start and end tokens were estimated and that songs were generated by iterating until an end token was emitted. This sampling scheme imposes the length distribution implied by the estimated transition probabilities, which need not equal the empirical length distribution; next-token accuracy is sensitive to sequence length and boundary placement. Consequently, the accuracy drop in Fig. 1E is confounded: absence of dependencies beyond six syllables is not the only difference between the natural and artificial test sets. Please report side-by-side diagnostics (song-length distribution, start/end token frequencies, per-position accuracy, unigram and bigram frequencies) and, if mismatches appear, re-run the comparison on a length-matched artificial corpus or include a scrambled-order control.
  2. [§4.1, Fig. 1C–D] The claim that FinchGPT outperforms Markov, RNN, and LSTM models at next-syllable prediction is central but lacks inferential statistics. The Statistics section lists paired t-test and Wilcoxon tests, yet the Results report p-values only for Fig. 1E (p = 0.033 and p = 0.31). With n = 3 corpora, the accuracy and cross-entropy differences in Fig. 1C and 1D must be accompanied by p-values or confidence intervals for each pairwise comparison. The same applies to the 'significantly worse' claim for the 1L/1A model in Fig. 1G and to the data-size comparisons in Fig. 1H.
  3. [§4.3, Fig. 2D] The restricted-attention experiment is a strong causal design, but the functional-importance conclusion needs statistical support. No p-value or confidence interval is reported for the increase in cross-entropy as the attention span is restricted, despite the caption giving n = 235,256 predictions across 2,660 songs. Because masking changes the model's effective context and its training dynamics, please also report a control that separates context-length effects from capacity or optimization effects, such as a full-attention model with comparable effective capacity or a fixed-context Markov baseline evaluated under the same masking.
  4. [§4.5, Fig. 4D–F] The before/after HVC-ablation comparison is potentially confounded by distribution shift. The after-ablation corpus has 1,180 songs versus 752 before ablation, and 59,111 versus 38,599 predictions; the paper reports that pitch is unchanged but does not report whether syllable frequencies or song lengths changed after ablation. The increased cross-entropy of the before-trained model on after-ablation songs could reflect marginal distribution shift rather than a specific loss of long-range dependencies. Please provide these diagnostics, report the p-values for the Wilcoxon tests in Fig. 4D, and add error bars or statistics for the attention-span comparisons in Fig. 4E–F.
minor comments (6)
  1. [Fig. 2C] The attention-weight threshold of >0.5 used to define 'attention span length' is arbitrary; please justify it or provide a sensitivity analysis over thresholds.
  2. [Methods, 'Model Evaluation Indicators'] Excluding the token immediately following the start token from the cross-entropy calculation should be justified, since this choice changes the metric and affects comparability across models and corpora.
  3. [Throughout] The manuscript does not include a data or code availability statement, which is important for reproducing the corpus, the artificial-corpus generation, and the model training.
  4. [Editorial] There are several typographical and labeling issues: 'attenuation heads' should read 'attention heads' (Methods); 'aniterior' in Fig. 4A should be 'anterior'; and 'Rate of change' in the Fig. 4F caption should read 'ROC'.
  5. [§4.2, Fig. 1G] The sentence 'The model engineered one attention head layer (1L, 1A) resulted in significantly worse cross-entropy' is awkward and should be rewritten for clarity.
  6. [Fig. 4F] The caption states that the upper bound is the accuracy achieved when testing on the training dataset; this is an unusual definition of an upper bound and should be explained more clearly in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: model comparisons, the artificial 6th-order Markov corpus, and attention-span restriction are external, intervention-based tests.

full rationale

The claimed derivation chain is: (1) train Markov, RNN, LSTM, and Transformer models on identical natural Bengalese finch corpora and compare held-out next-token accuracy; (2) construct an artificial corpus by sampling from a 6th-order Markov model estimated from the same natural songs and show that the Transformer's accuracy drops relative to the Markov baseline; (3) restrict the Transformer's attention span and observe degraded cross-entropy and classification performance; (4) use HVC ablation as a biological manipulation and show that a model trained on before-ablation songs loses predictive accuracy on after-ablation songs. None of these steps defines its conclusion into its inputs. The artificial corpus is a generated test set, not a fitted parameter relabeled as a prediction; the accuracy drop on that corpus is an empirical generalization result. A potential confound, such as unstated differences in song-length distribution between natural and artificial corpora, is a validity threat rather than a circular reduction, and the paper does not exhibit any equation or construction that makes the conclusion true by definition. The attention-span restriction is an architectural intervention: performance is measured under the intervention, so it does not rely solely on observing attention weights to claim long-range importance. The only overlapping-author citation, SAIBS [13], is used as an annotation method and an ablation paradigm; it is methodological support rather than an unverified premise invoked to force a conclusion, and the main comparative claims are evaluated against held-out natural data and against an explicitly constructed null corpus. Therefore, no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several domain assumptions about what next-token prediction and attention weights reveal about animal vocal structure, and on the fidelity of the SAIBS annotator and the Markov null corpus. The paper introduces no new physical or semantic entities; the only hand-picked numeric parameters are the attention weight threshold, the Markov order, and the manually selected model sizes.

free parameters (4)
  • Attention weight threshold = 0.5
    Used to define attended token pairs for computing average attention span length; no justification for the threshold, and changing it would change the reported span lengths.
  • Markov order for baseline and control corpus = 6
    The comparison Markov model and the artificial null corpus use 6 preceding syllables; the choice of 6 is not derived from data and higher orders were not tested.
  • RNN and LSTM hyperparameters = not specified (TPE-selected)
    Tuned on evaluation data via Tree-structured Parzen Estimator; the Transformer was not tuned the same way, a source of comparison bias.
  • Model sizes (layers, heads, hidden dimensions) = small 1L-1A-192H; medium 6L-6A-384H; large 12L-12A-768H
    Selected by manual grid in Fig 1G; these architecture choices affect results but are not fitted to a theory.
assumptions (5)
  • domain assumption The next-token prediction objective is a valid probe for syntactic structure in animal vocal sequences.
    Section 4.1 uses next-syllable prediction accuracy and cross-entropy to compare models, assuming better prediction reflects better capture of sequential rules.
  • domain assumption Self-attention weights can be interpreted as dependency strengths between syllables, and Chu-Liu-Edmonds trees from them reflect syntactic structure.
    Section 4.3 and Fig 2B use maximum spanning trees over attention weights as evidence for long-range dependencies.
  • domain assumption The artificial 6th-order Markov corpus is a faithful null model that removes all dependencies beyond six syllables while preserving all other properties of natural songs.
    Section 4.1 uses this corpus to argue that the Transformer's performance drop is due specifically to absence of long-range dependencies; generation artifacts could confound.
  • domain assumption SAIBS syllable annotation is accurate enough that errors do not materially affect the sequential statistics.
    The entire corpus is produced by SAIBS; no validation of annotation accuracy is reported in this paper.
  • domain assumption HVC ablation disrupts syllable sequencing without affecting syllable phonology.
    Section 4.5 relies on prior work and the pitch comparison to assume the ablation specifically affects syntax; if phonology also changed, the model's cross-entropy increase could be confounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinchGPT: a Transformer based language model for birdsong analysis." pith.science (2026). https://pith.science/paper/PRH4G6HB

@misc{pith2026250200344,
  author       = {Pith},
  title        = {Pith review of: FinchGPT: a Transformer based language model for birdsong analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PRH4G6HB}},
  note         = {Machine review of arXiv:2502.00344}
}
read the original abstract

The long-range dependencies among the tokens, which originate from hierarchical structures, are a defining hallmark of human language. However, whether similar dependencies exist within the sequential vocalization of non-human animals remains a topic of investigation. Transformer architectures, known for their ability to model long-range dependencies among tokens, provide a powerful tool for investigating this phenomenon. In this study, we employed the Transformer architecture to analyze the songs of Bengalese finch (Lonchura striata domestica), which are characterized by their highly variable and complex syllable sequences. To this end, we developed FinchGPT, a Transformer-based model trained on a textualized corpus of birdsongs, which outperformed other architecture models in this domain. Attention weight analysis revealed that FinchGPT effectively captures long-range dependencies within syllables sequences. Furthermore, reverse engineering approaches demonstrated the impact of computational and biological manipulations on its performance: restricting FinchGPT's attention span and disrupting birdsong syntax through the ablation of specific brain nuclei markedly influenced the model's outputs. Our study highlights the transformative potential of large language models (LLMs) in deciphering the complexities of animal vocalizations, offering a novel framework for exploring the structural properties of non-human communication systems while shedding light on the computational distinctions between biological brains and artificial neural networks.

Figures

Figures reproduced from arXiv: 2502.00344 by the authors.

Figure 1
Figure 1. Generation and evaluation of GPT trained with birdsong. (A) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Attention visualization reveals the long-range dependencies within a song. (A) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Token embedding analysis reveals the contextual usage of syllables. (A) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation of brain nucleus reduces the performance of FinchGPT. (A) [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Steps towards an Ecology for the Internet

    cs.NI 2025-06 conditional novelty 4.0 of 10

    The Internet should borrow ecological principles such as local immunity, software mutation, and mutualism to avoid collapse as it scales toward trillions of nodes.

Reference graph

Works this paper leans on

55 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vaswani, et al., Attention is All you Need in Advances in Neural Information Processing Systems, (Curran Associates, Inc., 2017)

    A. Vaswani, et al., Attention is All you Need in Advances in Neural Information Processing Systems, (Curran Associates, Inc., 2017)

    work page 2017

  2. [2]

    Radford, et al., Language Models are Unsupervised Multitask Learners

    A. Radford, et al., Language Models are Unsupervised Multitask Learners. OpenAI (2019)

    work page 2019

  3. [3]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, T. Solorio, Eds. (Associat...

    work page 2019

  4. [4]

    Raffel, et al., Exploring the limits of transfer learning with a unified text-to-text transformer J

    C. Raffel, et al., Exploring the limits of transfer learning with a unified text-to-text transformer J. Mach. Learn. Res. 21, 140:5485-140:5551 (2020)

    work page 2020

  5. [5]

    Brown, et al., Language Models are Few-Shot Learners in Advances in Neural Information Processing Systems, (Curran Associates, Inc., 2020), pp

    T. Brown, et al., Language Models are Few-Shot Learners in Advances in Neural Information Processing Systems, (Curran Associates, Inc., 2020), pp. 1877–1901

    work page 2020

  6. [6]

    M. D. Hauser, N. Chomsky, W. T. Fitch, The Faculty of Language: What Is It, Who Has It, and How Did It Evolve? Science 298, 1569–1579 (2002)

    work page 2002

  7. [7]

    Chomsky, Syntactic Structures (Walter de Gruyter, 1957)

    N. Chomsky, Syntactic Structures (Walter de Gruyter, 1957)

    work page 1957

  8. [8]

    Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution (Oxford University Press, 2003)

    R. Jackendoff, Foundations of Language: Brain, Meaning, Grammar, Evolution (Oxford University Press, 2003)

    work page 2003

Show all 55 references
  1. [9]

    Berwick, G

    R. Berwick, G. Beckers, K. Okanoya, J. Bolhuis, A Bird’s Eye View of Human Language Evolution. Front. Evol. Neurosci, 4 (2012)

    2012

  2. [10]

    Kershenbaum, et al., Animal vocal sequences: not the Markov chains we thought they were

    A. Kershenbaum, et al., Animal vocal sequences: not the Markov chains we thought they were. Proceedings of the Royal Society B: Biological Sciences, 281, 20141370 (2014)

    2014

  3. [11]

    J. E. Markowitz, E. Ivie, L. Kligler, T. J. Gardner, Long-range Order in Canary Song. PLOS Computational Biology 9, e1003052 (2013)

    2013

  4. [12]

    Okanoya, The Bengalese Finch: A Window on the Behavioral Neurobiology of Birdsong Syntax

    K. Okanoya, The Bengalese Finch: A Window on the Behavioral Neurobiology of Birdsong Syntax. Annals of the New York Academy of Sciences 1016, 724–735 (2004)

    2004

  5. [13]

    Kawaji, M

    T. Kawaji, M. Fujibayashi, K. Abe, Goal-directed and flexible modulation of syllable sequence within birdsong. Nat Commun 15, 3419 (2024)

    2024

  6. [14]

    Sainburg, M

    T. Sainburg, M. Thielk, T. Q. Gentner, Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires. PLOS Computational Biology 16, e1008228 (2020)

    2020

  7. [15]

    Morfi, R

    V . Morfi, R. F. Lachlan, D. Stowell, Deep perceptual embeddings for unlabelled animal sound eventsa). The Journal of the Acoustical Society of America 150, 2–11 (2021)

    2021

  8. [16]

    S. Kahl, C. M. Wood, M. Eibl, H. Klinck, BirdNET: A deep learning solution for avian diversity monitoring. Ecological Informatics 61, 101236 (2021)

    2021

  9. [17]

    Hsu, et al., HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

    W.-N. Hsu, et al., HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Trans. Audio, Speech and Lang. Proc.29, 3451–3460 (2021)

    2021

  10. [18]

    Baevski, H

    A. Baevski, H. Zhou, A. Mohamed, M. Auli, wav2vec 2.0: a framework for self-supervised learning of speech representations in Proceedings of the 34th International Conference on Neural Information Processing Systems 1NIPS ’20., (Curran Associates Inc., 2020), pp. 12449–12460. 1...

    2020

  11. [19]

    Hagiwara, A VES: Animal V ocalization Encoder based on Self-Supervision

    M. Hagiwara, A VES: Animal V ocalization Encoder based on Self-Supervision. [Preprint] (2022). Available at: http://arxiv.org/abs/2210.14493 [Accessed 4 January 2025]

    2022 arXiv

  12. [20]

    Kuang, S

    Z. Kuang, S. Jiang, H. Huang, Y . Liu, X. Li, Research on end-to-end animal behavior speech classification based on Wav2Vec2.0 in 3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023) (SPIE, 2023), pp. 915–922

    2023

  13. [21]

    Cauzinille, et al., Investigating self-supervised speech models’ ability to classify animal vocalizations: The case of gibbon’s vocal signatures in (2024), pp

    J. Cauzinille, et al., Investigating self-supervised speech models’ ability to classify animal vocalizations: The case of gibbon’s vocal signatures in (2024), pp. 132–136

    2024

  14. [22]

    Robinson, M

    D. Robinson, M. Miron, M. Hagiwara, O. Pietquin, NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics. [Preprint] (2024) Available at: http://arxiv.org/abs/2411.07186 [Accessed 4 January 2025]

    2024 arXiv

  15. [23]

    Tchernichovski, F

    O. Tchernichovski, F. Nottebohm, C. E. Ho, B. Pesaran, P. P. Mitra, A procedure for an automated measurement of song similarity. Animal Behaviour 59, 1167–1176 (2000)

    2000

  16. [24]

    Yoeng-jin, L

    C. Yoeng-jin, L. Tseng-hong, On the shortest arborescence of a directed graph. Scientia Sinica 14, 1396–1400 (1965)

    1965

  17. [25]

    Raganato, J

    A. Raganato, J. Tiedemann, An Analysis of Encoder Representations in Transformer-Based Machine Translation in Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP T. Linzen, G. Chrupała, A. Alishahi, Eds. (Association for Compu...

    2018

  18. [26]

    K. Abe, D. Watanabe, Songbirds possess the spontaneous ability to discriminate syntactic rules. Nat Neurosci 14, 1067–1074 (2011)

    2011

  19. [27]

    Sainburg, B

    T. Sainburg, B. Theilman, M. Thielk, T. Q. Gentner, Parallels in the sequential organization of birdsong and human speech. Nature Communications 10, 3636 (2019)

    2019

  20. [28]

    Cohen, et al., Hidden neural states underlie canary song syntax

    Y . Cohen, et al., Hidden neural states underlie canary song syntax. Nature 582, 539–544 (2020)

    2020

  21. [29]

    Kaplan, et al., Scaling Laws for Neural Language Models

    J. Kaplan, et al., Scaling Laws for Neural Language Models. [Preprint] (2020). Available at: http://arxiv.org/abs/2001.08361 [Accessed 23 October 2024]

    2020 arXiv

  22. [30]

    Jawahar, B

    G. Jawahar, B. Sagot, D. Seddah, What Does BERT Learn about the Structure of Language? in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics A. Korhonen, D. Traum, L. Màrquez, Eds. (Association for Computational Linguistics, 2019), pp. 3651–3657

    2019

  23. [31]

    Tenney, D

    I. Tenney, D. Das, E. Pavlick, BERT Rediscovers the Classical NLP Pipeline in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics A. Korhonen, D. Traum, L. Màrquez, Eds. (Association for Computational Linguistics, 2019), pp. 4593–4601

    2019

  24. [32]

    R. Suge, K. Okanoya, Perceptual chunking in the self-produced songs of Bengalese finches (Lonchura striata var. domestica). Anim Cogn 13, 515–523 (2010)

    2010

  25. [33]

    M. E. Peters, et al., Deep Contextualized Word Representations in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) M. Walker, H. Ji, A. Stent, Eds. (Associatio...

    2018

  26. [34]

    Coenen, et al., Visualizing and Measuring the Geometry of BERT

    A. Coenen, et al., Visualizing and Measuring the Geometry of BERT. [Preprint] (2019). Available at: http://arxiv.org/abs/1906.02715 [Accessed 16 November 2024]

    2019 arXiv

  27. [35]

    M. Garcia, Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol...

    2021

  28. [36]

    M. S. Fee, J. H. Goldberg, A hypothesis for basal ganglia-dependent reinforcement learning in the songbird. Neuroscience 198, 152–170 (2011)

    2011

  29. [37]

    Chakraborty, E

    M. Chakraborty, E. D. Jarvis, Brain evolution by brain pathway duplication. Philosophical Transactions of the Royal Society B: Biological Sciences 370, 20150056 (2015)

    2015

  30. [38]

    Nottebohm, T

    F. Nottebohm, T. M. Stokes, C. M. Leonard, Central control of song in the canary, Serinus canarius. Journal of Comparative Neurology 165, 457–486 (1976)

    1976

  31. [39]

    E. T. Vu, M. E. Mazurek, Y . C. Kuo, Identification of a forebrain motor programming network for the learned song of zebra finches. J. Neurosci. 14, 6924–6934 (1994)

    1994

  32. [40]

    R. H. R. Hahnloser, A. A. Kozhevnikov, M. S. Fee, An ultra-sparse code underliesthe generation of neural sequences in a songbird. Nature 419, 65–70 (2002). 11 FinchGPT: A Transformer-based Language Model for Birdsong Analysis A PREPRINT

    2002

  33. [41]

    Sánchez-Valpuesta, et al., Corticobasal ganglia projecting neurons are required for juvenile vocal learning but not for adult vocal plasticity in songbirds

    M. Sánchez-Valpuesta, et al., Corticobasal ganglia projecting neurons are required for juvenile vocal learning but not for adult vocal plasticity in songbirds. Proceedings of the National Academy of Sciences 116, 22833–22843 (2019)

    2019

  34. [42]

    G. R. Isola, A. V ochin, J. T. Sakata, Manipulations of inhibition in cortical circuitry differentially affect spectral and temporal features of Bengalese finch song. Journal of Neurophysiology 123, 815–830 (2020)

    2020

  35. [43]

    Chomsky, Knowledge of Language: Its Nature, Origin, and Use (Bloomsbury Academic, 1986)

    N. Chomsky, Knowledge of Language: Its Nature, Origin, and Use (Bloomsbury Academic, 1986)

    1986

  36. [44]

    Chomsky, I

    N. Chomsky, I. Roberts, J. Watumull, Opinion | Noam Chomsky: The False Promise of ChatGPT. The New York Times (2023)

    2023

  37. [45]

    Y . Lin, Y . C. Tan, R. Frank, Open Sesame: Getting inside BERT’s Linguistic Knowledge inProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP ,T. Linzen, G. Chrupała, Y . Belinkov, D. Hupkes, Eds. (Association for Computational L...

    2019

  38. [46]

    R. S. Payne, S. McVay, Songs of Humpback Whales. Science 173, 585–597 (1971)

    1971

  39. [47]

    Sharma, et al., Contextual and combinatorial structure in sperm whale vocalisations

    P. Sharma, et al., Contextual and combinatorial structure in sperm whale vocalisations. Nat Commun 15, 3617 (2024)

    2024

  40. [48]

    I. M. Pepperberg, The Comparative Psychology of Intelligence: Some Thirty Years Later. Front. Psychol. 11 (2020)

    2020

  41. [49]

    Hedwig, A

    D. Hedwig, A. Kohlberg, Call combination in African forest elephants Loxodonta cyclotis. PLOS ONE 19, e0299656 (2024)

    2024

  42. [50]

    Leroux, et al., Call combinations and compositional processing in wild chimpanzees

    M. Leroux, et al., Call combinations and compositional processing in wild chimpanzees. Nat Commun 14, 2225 (2023)

    2023

  43. [51]

    A. R. Lameira, M. E. Hardus, A. Ravignani, T. Raimondi, M. Gamba, Recursive self-embedded vocal motifs in wild orangutans. eLife 12, RP88348 (2024)

    2024

  44. [52]

    Zador, et al., Catalyzing next-generation Artificial Intelligence through NeuroAI

    A. Zador, et al., Catalyzing next-generation Artificial Intelligence through NeuroAI. Nat Commun 14, 1597 (2023)

    2023

  45. [53]

    Hassabis, D

    D. Hassabis, D. Kumaran, C. Summerfield, M. Botvinick, Neuroscience-Inspired Artificial Intelligence. Neuron 95, 245–258 (2017)

    2017

  46. [54]

    Fenno, O

    L. Fenno, O. Yizhar, K. Deisseroth, The development and application of optogenetics. Annu Rev Neurosci 34, 389–412 (2011)

    2011

  47. [55]

    B. L. Roth, DREADDs for Neuroscientists. Neuron 89, 683–694 (2016). 12

    2016

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.