Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Designing and Contextualising Probes for African Languages

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that language models adapted for African languages encode more linguistic information about those languages than massively multilingual models do, and that this encoded knowledge—not probe memorisation—drives their strong…

desk verdict First systematic probe of African-language PLMs; plausible and useful, but the selectivity confound and weak baselines need fixing before the central claim is accepted. read the letter →

arxiv 2505.10081 v2 pith:ZMPIZE7K submitted 2025-05-15 cs.CL

Wisdom Aduah , Francois Meyer This is my paper · ORCID
classification cs.CL
keywords probingAfricanlanguagesmultilinguallanguagemodelscontroltasksselectivitypart-of-speechtaggingnamedentityrecognitioncross-lingualtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is the first systematic probing study of African-language pretrained models, covering six typologically diverse languages and three tasks. It claims that models adapted for African languages store more linguistic information about their target languages than massively multilingual models such as XLM-R, and that this is why adapted models perform well when transferred to new tasks. The authors design a control task for the MasakhaPOS dataset and use random baselines for named-entity recognition and topic classification to argue that the probe results reflect knowledge inside the model rather than memorisation by the probe. They also find syntactic information concentrated in middle-to-late layers while sentence-level semantic information is spread across all layers.

What carries the argument

The load-bearing tools are probes and their baselines. Each probe is a one-hidden-layer neural network with 50 hidden units, trained on hidden representations to predict parts of speech, named-entity labels, or news topics. To distinguish model knowledge from probe learning, the paper builds a control task for MasakhaPOS by randomly mapping each word type to a part-of-speech tag sampled from the empirical tag distribution, then re-annotating the corpus; selectivity is the difference between original-task accuracy and control-task accuracy. For named-entity recognition, the baseline is the same probe trained on a randomly re-initialised copy of each pretrained model, giving a per-layer F1 gain; for news topic classification, the baseline is an untrained sequence model that supplies contextual but not linguistic information.

What would settle it

Permute the word identities used to build hidden representations by shuffling the token embedding table before feeding text through the model, then retrain the probe; if the measured selectivity does not drop sharply, the probe is recovering surface word-type statistics rather than the linguistic knowledge the paper attributes to the model.

Watch

Extended reading notes

Core claim

The paper's central claim is that including a language in pretraining or multilingual adaptation changes the hidden representations themselves, not just the output layer: adapted models (AfroXLMR, Nguni-XLMR, AfriBERTa, AfroLM) show higher probe selectivity and higher probe gains than XLM-R on the same languages. The authors identify cross-lingual representation learning, especially among related Bantu languages, as the mechanism behind previously reported zero-shot performance on MasakhaNER. They further argue that this transfer is limited by language-family representation: Luganda benefits from Bantu-family transfer even when absent from pretraining, while Igbo, of the under-represented Volta-Niger family, shows the weakest encoding. The layer-wise picture—syntax peaking in middle-to-last layers and semantics spread across layers—matches earlier probing results in other languages and is presented as evidence that these models organise African-language information in familiar ways.

Load-bearing premise

The comparisons assume that probe scores, after subtracting control and random baselines, reflect genuine differences in what models know about a language rather than differences in how easily the probe can extract that knowledge from each model's representations.

Editorial extensions

If this is right

  • Fine-tuning or probing middle-to-last layers should be the most sample-efficient route for syntactic tasks like part-of-speech tagging and named-entity recognition in African-language models.
  • Multilingual adaptation should be treated as a representational intervention: it improves downstream performance by enhancing language-specific knowledge inside hidden layers, not merely by adding a task head.
  • Zero-shot named-entity recognition should be expected when the target language shares a family with languages in the model's pretraining data, but not when the entire family is absent or under-represented.
  • Probe selectivity curves offer a practical diagnostic for choosing which layers to freeze, prune, or early-exit in African-language models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of AfroLM's high part-of-speech selectivity would be to train an identical model on randomly selected training samples rather than self-active samples; if selectivity drops, active learning is indeed selecting syntactically informative data.
  • The named-entity recognition random-reinitialisation baseline could be replaced or supplemented by a span-level control task that randomly reassigns entity types to multi-word spans, giving named-entity recognition the same interpretability as part-of-speech tagging.
  • Luganda's high news-topic variance suggests that cross-lingual semantic transfer is weaker than syntactic transfer; this predicts that sentence-level probing on distant language pairs will show larger drops than word-level probing at matched data sizes.
  • If selectivity curves are stable across languages, early-exit and layer-dropping strategies for African-language models should preserve most syntactic competence; this is testable by rerunning MasakhaPOS probes on truncated models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a probing study of seven multilingual PLMs across six African languages, using POS tagging, NER, and news topic classification. The authors design a control task for the MasakhaPOS dataset following Hewitt and Liang (2019), and use random baselines for NER and NTC. The central claim is that PLMs adapted for African languages encode more linguistic information about their target languages than massively multilingual PLMs, and that cross-lingual representation learning explains previously reported zero-shot NER performance. The paper reports layer-wise probe performance, best-layer results in Table 3, and qualitative trends about the layer distributions of syntactic and semantic information.

Significance. If the central claim holds, the paper would be a valuable contribution to interpretability research for African NLP, offering the first systematic probing analysis for these languages and introducing a reusable control task for MasakhaPOS. The broad coverage of languages, models, and tasks is a strength, and the layer-wise analysis connects to established findings in BERTology. However, the main comparative claim (adapted models encode more) rests on a metric that may be confounded by representation context-sensitivity, and the causal claim about zero-shot NER is not directly supported. The paper is promising but requires substantial additional analysis to substantiate its headline conclusion.

major comments (3)
  1. [§3.2, Fig. 3, Table 3] The headline claim that adapted PLMs encode more linguistic information is based primarily on POS selectivity, but control-task accuracies are never reported. In the Hewitt-Liang control task, each word type is assigned a fixed random label, so a probe must map all contextual occurrences of a word to the same arbitrary label. Models with more context-sensitive representations—which adapted models are likely to have—will find this harder, lowering control accuracy and mechanically raising selectivity even if the amount of task-relevant linguistic knowledge is unchanged. The paper should report control accuracies separately, and should verify the adapted-versus-multilingual advantage using a method that is not sensitive to this confound, for example by comparing raw accuracies conditioned on control accuracy or by using an alternative control that preserves contextual difficulty.
  2. [§3.3 and §5.2] The NER gain over randomly re-initialised PLMs is used to support both the encoding claim and the causal statement that cross-lingual representation learning is 'the mechanism behind' previously reported zero-shot MasakhaNER performance. A randomly re-initialised transformer is not a matched baseline for contextualisation: adapted models may show higher gains simply because their representations are more context-sensitive, not because they encode more NER-specific information. Moreover, probe gains are correlational evidence about representation content, not causal evidence about downstream task success. The causal statement in §5.2 should be removed or substantially softened.
  3. [§4 and Table 3] All results appear to come from a single run, with no error bars, multiple seeds, or significance tests. Many of the differences in Table 3 (e.g., POS selectivity 16.39 vs 19.09 for XLM-R-base vs XLM-R-large on Swahili) may be within run-to-run variability for MLP probes. The authors should provide variance estimates, at least for the best-layer comparisons that drive the central claim, or explicitly state that the reported differences are not statistically tested.
minor comments (5)
  1. [Fig. 4] The y-axis label says 'F1-score' but the caption and text describe the plotted quantity as the gain over a random baseline; the label should read 'F1 gain' or similar.
  2. [§4.2] The phrase 'The six language in our study' should be 'The six languages in our study'.
  3. [§5.1] There is a typo: 'synactic' should be 'syntactic'.
  4. [§5.2] The sentence 'both exhibit high probe performance gains for Luganda than AfroLM' is ungrammatical; it should be 'both exhibit higher probe performance gains for Luganda than AfroLM'.
  5. [Fig. 3] For Igbo, several models show negative selectivity in some layers; this is mentioned in the text but not visualised clearly in the caption. Consider adding a note to the caption that negative values indicate control accuracy exceeding task accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the probing comparisons are empirical and self-contained, with only a non-load-bearing self-citation.

full rationale

The paper's central claims are empirical observations: probes are trained on frozen representations from seven PLMs and evaluated on MasakhaPOS, MasakhaNER, and MasakhaNEWS, with performance contextualized by a control task and random baselines. The conclusion that adapted models encode more linguistic information is a measured ranking of selectivity and gain, not an identity derived from definitions. The control task follows Hewitt and Liang (2019) applied to a new dataset; the NER/NTC baselines are standard architecture-matched random re-initializations. No parameter is fitted to a subset of the target results and then reported as a prediction. The only self-citation, Nguni-XLMR (Meyer et al., 2024, a co-author's prior model), is the object of study rather than an unverified premise used to force the conclusion. Concerns about control-accuracy confounds or the strength of random baselines are validity risks already acknowledged in the Limitations section, not circularity: no equation in the paper reduces 'adapted models encode more information' to the construction of the probe or the baseline.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard probing methodology. The main free parameter is the probe architecture (50 hidden units). The key assumptions are that probe performance and control/baseline subtraction isolate model-encoded knowledge, which is the central debated premise in the probing literature and is only partially mitigated here.

free parameters (1)
  • probe hidden size = 50
    MLP hidden units chosen by hand as a balance between complexity and simplicity (Section 3.1).
assumptions (3)
  • domain assumption A single-layer MLP probe with 50 hidden units can extract linguistic features from PLM representations if those features are present.
    Probe design in Section 3.1; the paper explicitly notes the debate between linear and nonlinear probes.
  • domain assumption The MasakhaPOS control task, which assigns random POS tags to word types sampled from the empirical tag distribution, provides a valid baseline for isolating model-encoded knowledge.
    Control task design in Section 3.2, following Hewitt and Liang (2019).
  • domain assumption Re-initialising a PLM architecture gives a valid random baseline with no linguistic knowledge for NER.
    NER baseline in Section 3.3; the paper acknowledges this baseline's known limitations in the Limitations section.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Designing and Contextualising Probes for African Languages." pith.science (2026). https://pith.science/paper/ZMPIZE7K

@misc{pith2026250510081,
  author       = {Pith},
  title        = {Pith review of: Designing and Contextualising Probes for African Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZMPIZE7K}},
  note         = {Machine review of arXiv:2505.10081}
}
read the original abstract

Pretrained language models (PLMs) for African languages are continually improving, but the reasons behind these advances remain unclear. This paper presents the first systematic investigation into probing PLMs for linguistic knowledge about African languages. We train layer-wise probes for six typologically diverse African languages to analyse how linguistic features are distributed. We also design control tasks, a way to interpret probe performance, for the MasakhaPOS dataset. We find PLMs adapted for African languages to encode more linguistic information about target languages than massively multilingual PLMs. Our results reaffirm previous findings that token-level syntactic information concentrates in middle-to-last layers, while sentence-level semantic information is distributed across all layers. Through control tasks and probing baselines, we confirm that performance reflects the internal knowledge of PLMs rather than probe memorisation. Our study applies established interpretability techniques to African-language PLMs. In doing so, we highlight the internal mechanisms underlying the success of strategies like active learning and multilingual adaptation.

Figures

Figures reproduced from arXiv: 2505.10081 by the authors.

Figure 1
Figure 1. POS probe performance (selectivity), aver [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. NER probe gains (over random baselines) across layers, averaged over 6 African languages. of-speech (POS) categories or semantic concepts, are encoded in hidden layers. Probing provides insights into the internal mech￾anisms of PLMs by revealing how models acquire, store, and leverage linguistic information in hidden layers. This allows NLP practitioners to better un￾derstand the mechanisms by which PLMs succeed in … view at source ↗
Figure 3
Figure 3. Probe selectivity for POS tagging (the difference between MasakhaPOS accuracy and control task [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Probe performance gains for NER tagging (F1 improvements over randomly re-initialised PLM architec [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Probe accuracy for news topic classification (visualised in comparison to a random contextual baseline) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Raw accuracies for POS tagging [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Raw F1-scores for Named Entity Recognition. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Judit \'Acs, \'Akos K\'ad\'ar, and Andr\'as Kornai. 2021. Subword pooling makes a difference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2021 . Association for Computational Linguistics

    work page 2021

  4. [4]

    Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F

    David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, ...

    work page 2022

  5. [5]

    David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunn...

    2023

  6. [6]

    Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow

    Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. https://aclanthology.org/2022.coling-1.382 Adapting pre-trained language models to A frican languages via multilingual adaptive fine-tuning . In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336--4349, Gyeongju, Republic of Korea. ...

    2022

  7. [7]

    Guillaume Alain and Yoshua Bengio. 2017. https://openreview.net/forum?id=ryF7rTqgl Understanding intermediate layers using linear classifier probes

    work page 2017

  8. [8]

    Arnav Arora, Lucie-aim \'e e Kaffee, and Isabelle Augenstein. 2023. https://doi.org/10.18653/v1/2023.c3nlp-1.12 Probing pre-trained language models for cross-cultural differences in values . In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114--130, Dubrovnik, Croatia. Association for Computational Linguistics

    doi:10.18653/v1/2023.c3nlp-1.12 2023

Show all 31 references
  1. [9]

    David Arps, Laura Kallmeyer, Younes Samih, and Hassan Sajjad. 2024. https://doi.org/10.18653/v1/2024.naacl-long.433 Multilingual nonce dependency treebanks: Understanding how language models represent and process syntactic structure . In Proceedings of the 2024 Conference of t...

    2024 doi

  2. [10]

    Yonatan Belinkov. 2022. https://doi.org/10.1162/coli_a_00422 Probing Classifiers: Promises, Shortcomings, and Advances . Computational Linguistics, 48(1):207--219

    2022 doi

  3. [11]

    Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. https://doi.org/10.18653/v1/P17-1080 What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics...

    2017 doi

  4. [12]

    Grzegorz Chrupa a, Bertrand Higy, and Afra Alishahi. 2020. https://doi.org/10.18653/v1/2020.acl-main.381 Analyzing analytical methods: The case of phonology in neural models of spoken language . In Proceedings of the 58th Annual Meeting of the Association for Computational Lin...

    2020 doi

  5. [13]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  6. [14]

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. https://doi.org/10.18653/v1/P18-1198 What you can cram into a single \ & ! \# * vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th Annual Meet...

    2018 doi

  7. [15]

    Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. ...

    2023

  8. [16]

    Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, and Chris Emezue. 2022. https://aclanthology.org/2022.sustainlp-1.11 A fro LM : A self-active learning-based multilingual pretrained lang...

    2022

  9. [17]

    Muhammad Usman Hadi, Qasem Al Tashi, Abbas Shah, Rizwan Qureshi, Amgad Muneer, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, et al. 2024. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospect...

    2024

  10. [18]

    John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019

  11. [19]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langu...

    2019 doi

  12. [20]

    Jue Hou, Anisia Katinskaia, Lari Kotilainen, Sathianpong Trangcasanchai, Anh-Duc Vu, and Roman Yangarber. 2024. https://aclanthology.org/2024.lrec-main.1518 What do transformers know about government? In Proceedings of the 2024 Joint International Conference on Computational L...

    2024

  13. [21]

    Anisia Katinskaia and Roman Yangarber. 2024. Probing the category of verbal aspect in transformer language models. In Findings of the Association for Computational Linguistics, pages 3347--3366, United States. The Association for Computational Linguistics. Findings of the Asso...

    2024

  14. [22]

    Daoyang Li, Mingyu Jin, Qingcheng Zeng, Haiyan Zhao, and Mengnan Du. 2024. https://arxiv.org/abs/2409.14459 Exploring multilingual probing in large language models: A cross-language analysis . Preprint, arXiv:2409.14459

    2024 arXiv

  15. [23]

    Haoyan Luo and Lucia Specia. 2024. https://arxiv.org/abs/2401.12874 From understanding to utilization: A survey on explainability for large language models . Preprint, arXiv:2401.12874

    2024 arXiv

  16. [24]

    Francois Meyer, Haiyue Song, Abhisek Chakrabarty, Jan Buys, Raj Dabre, and Hideki Tanaka. 2024. https://aclanthology.org/2024.lrec-main.1071 NGLUE ni: Benchmarking and adapting pretrained language models for nguni languages . In Proceedings of the 2024 Joint International Conf...

    2024

  17. [25]

    Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. https://aclanthology.org/2021.mrl-1.11 Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages . In Proceedings of the 1st Workshop on Multilingual Representation Lea...

    2021

  18. [26]

    Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. https://doi.org/10.18653/v1/2020.acl-main.420 Information-theoretic probing for linguistic structure . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  19. [27]

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. https://doi.org/10.1162/tacl_a_00349 A primer in BERT ology: What we know about how BERT works . Transactions of the Association for Computational Linguistics, 8:842--866

    2020 doi

  20. [28]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019 a . https://doi.org/10.18653/v1/P19-1452 BERT rediscovers the classical NLP pipeline . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593--4601, Florence, Italy. Association f...

    2019 doi

  21. [29]

    Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019 b . https://openreview.net/forum?id=SJzSgnRcKX What do you learn from context? probing for sentence structure in con...

    2019

  22. [30]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  23. [31]

    Kelly Zhang and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5448 Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting ...

    2018 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.